k8s 集群的POD内不能访问，设置集群网络代理为--proxy-mode=ipvs

丁淇

2023-12-01

k8s 集群的POD内不能访问clusterIP和service

设置集群网络代理为–proxy-mode=ipvs

k8s 集群能ping通的环境kube-proxy使用了–proxy-mode=ipvs ,不能ping通的环境使用了默认模式（iptables）。

能通过coredns正常的解析到IP，然后去ping了一下service，发现不能ping通，ping clusterIP也不能ping通。

kubeadm 部署方式修改kube-proxy为 ipvs模式。

默认情况下，我们部署的kube-proxy通过查看日志，能看到如下信息：Flag proxy-mode="" unknown，assuming iptables proxy

[root@k8s-master ~]# kubectl logs -n kube-system kube-proxy-ppdb6 
W1013 06:55:35.773739       1 proxier.go:513] Failed to load kernel module ip_vs with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W1013 06:55:35.868822       1 proxier.go:513] Failed to load kernel module ip_vs_rr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W1013 06:55:35.869786       1 proxier.go:513] Failed to load kernel module ip_vs_wrr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W1013 06:55:35.870800       1 proxier.go:513] Failed to load kernel module ip_vs_sh with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W1013 06:55:35.876832       1 server_others.go:249] Flag proxy-mode="" unknown, assuming iptables proxy
I1013 06:55:35.890892       1 server_others.go:143] Using iptables Proxier.
I1013 06:55:35.892136       1 server.go:534] Version: v1.15.0
I1013 06:55:35.909025       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072
I1013 06:55:35.909053       1 conntrack.go:52] Setting nf_conntrack_max to 131072
I1013 06:55:35.919298       1 conntrack.go:83] Setting conntrack hashsize to 32768
I1013 06:55:35.945969       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
I1013 06:55:35.946044       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
I1013 06:55:35.946623       1 config.go:96] Starting endpoints config controller
I1013 06:55:35.946660       1 controller_utils.go:1029] Waiting for caches to sync for endpoints config controller
I1013 06:55:35.946695       1 config.go:187] Starting service config controller
I1013 06:55:35.946713       1 controller_utils.go:1029] Waiting for caches to sync for service config controller
I1013 06:55:36.047121       1 controller_utils.go:1036] Caches are synced for endpoints config controller
I1013 06:55:36.047195       1 controller_utils.go:1036] Caches are synced for service config controller

修改kube-proxy的配置文件,添加mode 为ipvs。

[root@k8s-master ~]# kubectl edit cm kube-proxy -n kube-system
...
ipvs:
      excludeCIDRs: null
      minSyncPeriod: 0s
      scheduler: ""
      strictARP: false
      syncPeriod: 30s
    kind: KubeProxyConfiguration
    metricsBindAddress: 127.0.0.1:10249
    mode: "ipvs"
   ...

将ipvs模式设置为ip_vs相关模块：

cat > /etc/sysconfig/modules/ipvs.modules <<EOF
 #!/bin/bash 
 modprobe -- ip_vs 
 modprobe -- ip_vs_rr 
 modprobe -- ip_vs_wrr 
 modprobe -- ip_vs_sh 
 modprobe -- nf_conntrack_ipv4 
EOF

重启kube-proxy 的pod 最小单元

chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4

删除kube-proxy 的pod 最小单元

[root@k8s-master ~]# kubectl get pod -n kube-system | grep kube-proxy |awk '{system("kubectl delete pod "$1" -n kube-system")}'
pod "kube-proxy-62gvr" deleted
pod "kube-proxy-n2rml" deleted
pod "kube-proxy-ppdb6" deleted
pod "kube-proxy-rr9cg" deleted

查看日志，注意模式变化

I1013 07:34:38.685794 1 server_others.go:170] Using ipvs Proxier.

[root@k8s-master ~]# kubectl get pod -n kube-system |grep kube-proxy
kube-proxy-cbm8p                     1/1     Running   0          85s
kube-proxy-d97pn                     1/1     Running   0          83s
kube-proxy-gmq6s                     1/1     Running   0          76s
kube-proxy-x6tcg                     1/1     Running   0          81s
[root@k8s-master ~]# kubectl logs -n kube-system kube-proxy-cbm8p 
I1013 07:34:38.685794       1 server_others.go:170] Using ipvs Proxier.
W1013 07:34:38.686066       1 proxier.go:401] IPVS scheduler not specified, use rr by default
I1013 07:34:38.687224       1 server.go:534] Version: v1.15.0
I1013 07:34:38.692777       1 conntrack.go:52] Setting nf_conntrack_max to 131072
I1013 07:34:38.693378       1 config.go:187] Starting service config controller
I1013 07:34:38.693391       1 controller_utils.go:1029] Waiting for caches to sync for service config controller
I1013 07:34:38.693406       1 config.go:96] Starting endpoints config controller
I1013 07:34:38.693411       1 controller_utils.go:1029] Waiting for caches to sync for endpoints config controller
I1013 07:34:38.793684       1 controller_utils.go:1036] Caches are synced for endpoints config controller
I1013 07:34:38.793688       1 controller_utils.go:1036] Caches are synced for service config controller

创建集群内部可访问的Service

# 暴露Service
[root@master ~]# kubectl expose deploy nginx --name=svc-nginx1 --type=ClusterIP --port=80 --target-port=80 -n dev
service/svc-nginx1 exposed

# 查看service
[root@master ~]# kubectl get svc svc-nginx1 -n dev -o wide
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE     SELECTOR
svc-nginx1   ClusterIP   10.109.179.231   <none>        80/TCP    3m51s   run=nginx
# 测试ping service
[root@k8s-master ~]# kubectl exec -it dns-test sh
/ # ping nginx-service
PING nginx-service (10.1.58.65): 56 data bytes
64 bytes from 10.1.58.65: seq=0 ttl=64 time=0.033 ms
64 bytes from 10.1.58.65: seq=1 ttl=64 time=0.069 ms
64 bytes from 10.1.58.65: seq=2 ttl=64 time=0.094 ms
64 bytes from 10.1.58.65: seq=3 ttl=64 time=0.057 ms
^C
--- nginx-service ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0.033/0.063/0.094 ms

最后，配置了集群网络代理为–proxy-mode=ipvs

可能是因为网络不稳定的问题
将docker 和 k8s 都设置为开机启动

systemctl enable docker
systemctl enable kubelet
reboot # 重启服务