kube-proxy Failed to retrieve node info: Unauthorized

简介

flannel 容器无法启动，看日志内容如下

I1102 02:32:56.069875       1 main.go:488] Using interface with name bond0.170 and address xx.xx.xx.xx
I1102 02:32:56.069940       1 main.go:505] Defaulting external address to interface address (xx.xx.xx.xx)
E1102 02:32:56.265305       1 main.go:232] Failed to create SubnetManager: error retrieving pod spec for 'kube-system/kube-flannel-ds-amd64-4rh69': Get https://10.96.0.1:443/api/v1/namespaces/kube-system/pods/kube-flannel-ds-amd64-40.96.0.1:443: getsockopt: network is unreachable

初步怀疑是iptables有防火墙规则，确认后不是此问题
查看iptables的转发表也没有 10.96.0.1,怀疑是kube-proxy 没有生成正确的转发规则

kube-proxy 日志如下

k logs -f -n kube-system kube-proxy-4zs2c
W1102 03:52:17.820455       1 server_others.go:559] Unknown proxy mode "", assuming iptables proxy
E1102 03:52:17.825488       1 node.go:125] Failed to retrieve node info: Unauthorized
E1102 03:52:18.827659       1 node.go:125] Failed to retrieve node info: Unauthorized
E1102 03:52:21.175085       1 node.go:125] Failed to retrieve node info: Unauthorized
E1102 03:52:25.966158       1 node.go:125] Failed to retrieve node info: Unauthorized
E1102 03:52:35.352455       1 node.go:125] Failed to retrieve node info: Unauthorized
E1102 03:52:52.327513       1 node.go:125] Failed to retrieve node info: Unauthorized
I1102 03:52:52.327542       1 server_others.go:178] can't determine this node's IP, assuming 127.0.0.1; if this is incorrect, please set the --bind-address flag
I1102 03:52:52.327553       1 server_others.go:186] Using iptables Proxier.
I1102 03:52:52.327778       1 server.go:583] Version: v1.18.20
I1102 03:52:52.328152       1 conntrack.go:52] Setting nf_conntrack_max to 2097152
I1102 03:52:52.328341       1 config.go:133] Starting endpoints config controller
I1102 03:52:52.328361       1 shared_informer.go:223] Waiting for caches to sync for endpoints config
I1102 03:52:52.328389       1 config.go:315] Starting service config controller
I1102 03:52:52.328422       1 shared_informer.go:223] Waiting for caches to sync for service config
E1102 03:52:52.330188       1 event.go:260] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"etcd1.16b39e53c745056e", GenerateName:"", Namespace:"default", SelfLink:"", UIrsion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:m(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"etcd1", UID:"etcd1", APIVersiersion:"", FieldPath:""}, Reason:"Starting", Message:"Starting kube-proxy.", Source:v1.EventSource{Component:"kube-proxy", Host:"etcd1"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xc0584b6513913d6e, ext:34567606865, loc:(*time.Loc)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xc0584b6513913d6e, ext:34567606865, loc:(*time.Location)(0x28998a0)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Seriees)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'Unauthorized' (will not retry!)
E1102 03:52:52.330813       1 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.Endpoints: Unauthorized
E1102 03:52:52.331160       1 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.Service: Unauthorized
E1102 03:52:53.667507       1 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.Endpoints: Unauthorized
E1102 03:52:53.860980       1 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.Service: Unauthorized
E1102 03:52:56.219253       1 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.Endpoints: Unauthorized

报错日志来看是证书验证失败，github上看到了有此问题的解决方法 ，需要删除kube-proxy 依赖的secret
k delete secret -n kube-system kube-proxy-token-hljcr

删除此secret后会自动生成新的，然后删除相关的kube-proxy容器，这个时候正常启动

总结

问题的根本原因是有同事误操作重新跑了kubeadm，导致集群里保存的证书和新生成的证书不一致

引用

https://github.com/kubernetes/kubernetes/issues/84244