kube-proxy ipvs模式

什么是IPVS?

IPVS (IP Virtual Server) 基于linux内核模块Netfilter实现的传输层负载均衡技术

LVS中负载均衡使用的技术就是IPVS, IPVS能够代理TCP,UDP协议，因此基于IPVS作为kubernetes Service负载均衡实现再好不过。

iptables最初设计用于创建主机防火墙，其规则采用线性遍历方式，因此随着Service数量的增大，效率会出现问题。

举个例子，一个5000个节点的集群，如果有2000个Service,每个Service有10个Pod. 那么每个工作节点上的iptables记录是2000*10=20000个。iptables的效率为O(N), 这会让内核会相当忙碌。一个合理的想法是使用Hash表将算法效率优化成O(1), IPVS技术就是使用Hash表存放规则。

IPVS模式的网络拓扑结构

创建一个ClusterIP类型的Service, kube-proxy会做三件事:

• 创建默认网卡kube-ipvs0
• 每个Service的ClusterIP绑定到网卡kube-ipvs0
• 为每个Service创建IPVS规则

# kubectl describe svc nginx-service
Name:			nginx-service
...
Type:			ClusterIP
IP:			    10.102.128.4
Port:			http	3080/TCP
Endpoints:		10.244.0.235:8080,10.244.1.237:8080
Session Affinity:	None

# ip addr
...
73: kube-ipvs0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 1a:ce:f5:5f:c1:4d brd ff:ff:ff:ff:ff:ff
    inet 10.102.128.4/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever

# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn     
TCP  10.102.128.4:3080 rr                                        # service ClusterIP 
  -> 10.244.0.235:8080            Masq    1      0          0    # pod ip     
  -> 10.244.1.237:8080            Masq    1      0          0    

怎样启用ipvs模式？

使用kubeadm配置文件，启用ipvs. 以下是我的模板:

# the following content generated by bash shell command: 
# kubeadm config print init-defaults
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.10.60  # kubernetes主节点IP
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: node
  taints: null
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers # 镜像仓库
kind: ClusterConfiguration
kubernetesVersion: 1.21.14 # kubernetes版本
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12 # service cluster ip
  podSubnet: 10.1.0.0/16  # pod ip
scheduler: {}
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs

使用kubeadm初始化集群时：

kubeadm init --config init-config.yaml

如何确认ipvs模式已经启用，可以查看kube-proxy Pod日志：

$ kubectl logs -f kube-proxy-bnhgs -n kube-system
I1227 02:52:23.675957       1 node.go:172] Successfully retrieved node IP: 192.168.10.60
I1227 02:52:23.676053       1 server_others.go:140] Detected node IP 192.168.10.60
I1227 02:52:23.751687       1 server_others.go:206] kube-proxy running in dual-stack mode, IPv4-primary
I1227 02:52:23.752282       1 server_others.go:274] Using ipvs Proxier. # ipvs模式启用
I1227 02:52:23.752297       1 server_others.go:276] creating dualStackProxier for ipvs.

Reference List

1. IPVS-Based In-Cluster Load Balancing Deep Dive
2. IPVS
3. kubeadm init
4. kubeadm Configuration