Kubernetes吊舱无法进行出站通信

Question

我已经在一组运行Centos7的VM上安装了Kubernetes V1.13.10。当我部署吊舱时，它们可以相互连接，但不能连接到集群之外的任何东西。CoreDNS吊舱在日志中有以下错误：

[ERROR] plugin/errors: 2 app.harness.io.xentaurs.com. A: unreachable backend: read udp 172.21.0.33:48105->10.20.10.52:53: i/o timeout
 [ERROR] plugin/errors: 2 app.harness.io.xentaurs.com. AAAA: unreachable backend: read udp 172.21.0.33:49098->10.20.10.51:53: i/o timeout
 [ERROR] plugin/errors: 2 app.harness.io.xentaurs.com. AAAA: unreachable backend: read udp 172.21.0.33:53113->10.20.10.51:53: i/o timeout
 [ERROR] plugin/errors: 2 app.harness.io.xentaurs.com. A: unreachable backend: read udp 172.21.0.33:39648->10.20.10.51:53: i/o timeout

IPs 10.20.10.51和10.20.10.52是内部DNS服务器，可以从节点访问。我从DNS服务器捕获了Wireshark数据，我看到流量来自CoreDNS pod IP地址172.21.0.33。DNS服务器将没有路由返回到该IP，因为它在Kubernetes集群之外是不可路由的。

[root@kube-aci-1 ~]# iptables -t nat -L POSTROUTING -v --line-number
Chain POSTROUTING (policy ACCEPT 23 packets, 2324 bytes)
num   pkts bytes target     prot opt in     out     source               destination         
1     1990  166K KUBE-POSTROUTING  all  --  any    any     anywhere             anywhere             /* kubernetes postrouting rules */
2        0     0 MASQUERADE  all  --  any    ens192.152  172.21.0.0/16        anywhere  

[root@kube-aci-1 ~]# kubectl logs kube-proxy-llq22 -n kube-system
W1117 16:31:59.225870       1 proxier.go:498] Failed to load kernel module ip_vs with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W1117 16:31:59.232006       1 proxier.go:498] Failed to load kernel module ip_vs_rr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W1117 16:31:59.233727       1 proxier.go:498] Failed to load kernel module ip_vs_wrr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W1117 16:31:59.235700       1 proxier.go:498] Failed to load kernel module ip_vs_sh with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W1117 16:31:59.255278       1 server_others.go:296] Flag proxy-mode="" unknown, assuming iptables proxy
I1117 16:31:59.289360       1 server_others.go:148] Using iptables Proxier.
I1117 16:31:59.296021       1 server_others.go:178] Tearing down inactive rules.
I1117 16:31:59.324352       1 server.go:484] Version: v1.13.10
I1117 16:31:59.335846       1 conntrack.go:52] Setting nf_conntrack_max to 131072
I1117 16:31:59.336443       1 config.go:102] Starting endpoints config controller
I1117 16:31:59.336466       1 controller_utils.go:1027] Waiting for caches to sync for endpoints config controller
I1117 16:31:59.336493       1 config.go:202] Starting service config controller
I1117 16:31:59.336499       1 controller_utils.go:1027] Waiting for caches to sync for service config controller
I1117 16:31:59.436617       1 controller_utils.go:1034] Caches are synced for service config controller
I1117 16:31:59.436739       1 controller_utils.go:1034] Caches are synced for endpoints config controller

我尝试刷新iptables nat表，并在所有节点上重新启动kube-proxy，但问题仍然存在。上面的输出中有什么线索吗，或者关于进一步故障排除的想法？

kubectl get节点的输出：

[root@kube-aci-1 ~]# kubectl get nodes -o wide
NAME         STATUS   ROLES    AGE   VERSION    INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION          CONTAINER-RUNTIME
kube-aci-1   Ready    master   85d   v1.13.10   10.10.52.217   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://1.13.1
kube-aci-2   Ready    <none>   85d   v1.13.10   10.10.52.218   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://1.13.1

Answer 1

结果表明，如果需要从吊舱进行出站通信，则必须使用CNI在网络上可路由的子网。我使子网可以在外部网络上进行路由，豆荚现在可以对外通信了。