安装

优质

小牛编辑

128浏览

2023-12-01

Installation

Install on kube-system

Install

kubectl create secret generic bigip-login --from-literal=username=admin --from-literal=password=admin -n kube-system
kubectl create serviceaccount bigip-ctlr -n kube-system
kubectl create -f rbac.yaml
kubectl create -f cis.yaml

cis.yaml
rbac.yaml

Uninstall

cd install-on-kube-system/
kubectl delete -f cis.yaml
kubectl delete -f rbac.yaml
kubectl delete serviceaccount bigip-ctlr -n kube-system
kubectl delete secret bigip-login -n kube-system

Install on bigip-ctlr

Install

kubectl create ns bigip-ctlr
kubectl create secret generic bigip-login --from-literal=username=admin --from-literal=password=admin -n bigip-ctlr
kubectl create serviceaccount bigip-ctlr -n bigip-ctlr
kubectl create -f rbac.yaml
kubectl create -f cis.yaml

cis.yaml
rbac.yaml
rbac-small.yaml

Uninstall

cd install-on-bigip-ctlr/
kubectl delete -f cis.yaml
kubectl delete -f rbac.yaml
kubectl delete serviceaccount bigip-ctlr -n bigip-ctlr
kubectl delete secret bigip-login -n bigip-ctlr
kubectl delete ns bigip-ctlr

livenessProbe & readinessProbe

This page provide an example of CIS Pod with livenessProbe & readinessProbe.

Full Deployment YAML - cis-probe.yaml

Key Config

          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /health
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 15
            periodSeconds: 15
            successThreshold: 1
            timeoutSeconds: 15
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /health
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 30
            periodSeconds: 30
            successThreshold: 1
            timeoutSeconds: 15

VE 健壮性检测(curl)

Full Deployment YAML - cis-rhel.yaml

          livenessProbe:
            failureThreshold: 3
            exec:
              command:
              - curl
              - -k
              - -s
              - -o
              - /dev/null
              - https://192.168.200.208
            initialDelaySeconds: 15
            periodSeconds: 15
            successThreshold: 1

Note	`curl` 命令只存在于 Red Hat Linux Base 镜像中。

VE 健壮性检测(sh + python)

Full Deployment YAML - cis-debian.yaml

          livenessProbe:
            exec:
              command:
              - sh
              - -c
              - echo 'import requests' > /tmp/live.py ; echo 'response = requests.get("https://192.168.200.208", verify=False, timeout=3)'  >> /tmp/live.py ; python -W ignore /tmp/live.py
            initialDelaySeconds: 15
            periodSeconds: 15
            successThreshold: 1
            failureThreshold: 3
            timeoutSeconds: 5

timeoutSeconds	timeout	VE 重启过程 CIS Restart 次数
1	1	7
3	2	6
5	3	6
15	15	5

Integrate CIS with flannel （etcd mode, docker0 network）

Refer to flannel-etcd-mode for detailed steps.

Install on bigip-ctlr flannel vxlan Control Hub Mode

What’s Control Hub Mode

Control Hub Mode is mainly for Ops Convenient, which use a Control Hub(one or multiple namespaces) to hold CIS configmap only.

HubMode expands on current ConfigMap implementation in CIS using the AS3 API. One of key strength of CIS is it can help multiple team collaborate better together. With microservices architecture we are seeing organizations create dedicate team that combine network and system personas. This allegement request that network engineers (NetOps) be added to these teams. F5 CIS ConfigMap is a perfect fit as it can accelerates the developer experience and workflow tooling.

Setup on F5 VE

1. 创建 partition

create auth partition kubernetes

kubernetes - Partition 的名称需要与 CIS Deployment 中--bigip-partition参数指定的partition一致。

2. 创建 VxLan 隧道

create net tunnels vxlan fl-vxlan port 8472 flooding-type none
create net tunnels tunnel flannel_vxlan key 1 profile fl-vxlan local-address 192.168.200.208
create net self 10.244.20.100 address 10.244.20.100/16 allow-service none vlan flannel_vxlan
save sys config
show net tunnels tunnel flannel_vxlan all-properties

192.168.200.208 – F5 VE 上VS业务VLAN的 Self IP或管理IP，K8S上创建 BIG-IP Node 时会使用到改地址，对应 flannel flannel.alpha.coreos.com/backend-data标注。
10.244.20.100/16 – K8S Flannel vxlan self IP, 必需是 16 位，这样才可以确保所有目的网络都走vtep
show net tunnels tunnel flannel_vxlan all-properties 会输出 tunnel vxlan interface MAC 地址，该地址会在K8S上创建 BIG-IP Node 时使用，对应 flannel flannel.alpha.coreos.com/public-ip 标注。

Setup on K8s

1. Install

kubectl create ns bigip-ctlr
kubectl create secret generic bigip-login --from-literal=username=admin --from-literal=password=admin -n bigip-ctlr
kubectl create serviceaccount bigip-ctlr -n bigip-ctlr
kubectl create -f rbac.yaml
kubectl create -f cis.yaml

rbac.yaml
cis.yaml

2. 创建 Big-IP Node

kubectl apply -f bigip-node.yaml

bigip-node.yaml

3. 查看日志

kubectl logs -f $(kubectl get pods -n bigip-ctlr --no-headers | awk '{print $1}') -n bigip-ctlr

部署测试应用

1. 部署测试应用

kubectl apply -f deploy.yaml

deploy.yaml

2. 创建 CIS Configmap

kubectl apply -f cm.yaml

cm.yaml

3. 查看日志输出

$ kubectl logs -f $(kubectl get pods -n bigip-ctlr --no-headers | awk '{print $1}') -n bigip-ctlr
...
2021/05/12 11:27:27 [INFO] [CCCL] Successfully Sent the FDB Records
2021/05/12 11:27:29 [INFO] [2021-05-12 11:27:29,320 f5_cccl.resource.resource INFO] Updating ApiFDBTunnel: /Common/flannel_vxlan
2021/05/12 11:27:45 [INFO] [2021-05-12 11:27:45,726 f5_cccl.resource.resource INFO] Creating ApiArp: /Common/k8s-10.244.1.47
2021/05/12 11:31:06 [INFO] [CCCL] Successfully Sent the FDB Records
2021/05/12 11:31:24 [INFO] [2021-05-12 11:31:24,519 f5_cccl.resource.resource INFO] Creating ApiArp: /Common/k8s-10.244.1.49

TroubleShoting

1. FDB

# show net fdb

---------------------------------------------------------------------
Net::FDB
Tunnel         Mac Address        Member                      Dynamic
---------------------------------------------------------------------
flannel_vxlan  16:73:97:7f:83:28  endpoint:192.168.200.210%0  no
flannel_vxlan  e2:85:68:42:be:b7  endpoint:192.168.200.211%0  no

2. ARP

# show net arp

-------------------------------------------------------------------------------------------------------
Net::Arp
Name                     Address          HWaddress          Vlan               Expire-in-sec  Status
-------------------------------------------------------------------------------------------------------
/Common/k8s-10.244.1.47  10.244.1.47      e2:85:68:42:be:b7  -                  -              static
/Common/k8s-10.244.1.49  10.244.1.49      e2:85:68:42:be:b7  -                  -              static

3. 抓包

tcpdump -nni flannel_vxlan icmp
tcpdump -nni  0.0:nnnp -s0 host ip and port  -w  /var/tmp/zhuabao.pcap  –vvv

访问服务

$ curl 192.168.200.34/coffee
Server address: 10.244.1.47:8080
Server name: app-1-7f4585dc79-l6sx4
Date: 12/May/2021:11:35:07 +0000
URI: /coffee
Request ID: 03f319a0fde3233c478e16e34115386e