kubeasz错误问题(failed to run Kubelet: misconfiguration: kubelet cgroup driver)

运行环境

• Ubuntu 20.04

• DOCKER 19.03.15

• KUBEASZ 3.1.0

• K8S_BIN 1.21.0

1 错误1

1.1 事件还原

由于使用 kubeasz（地址：https://github.com/easzlab/kubeasz）
部署二进制的 k8s ，为了便于操作，我提前将 k8s-master1，k8s-master2，k8s-master3，k8s-node1，k8s-node2，k8s-node3 都安装了 docker，这样在执行 ezctl setup k8s-01 03 的时候，发现每台机器都安装了 docker 就不会再执行安装 docker 的操作了，相应的也不会再修改 /etc/docker/daemon.json 文件，直接导致 docker 运行的命名空间和默认的命名空间不一致，所以无法启动 kubelet 服务

执行 [root@k8s-master1 kubeasz]# ./ezctl setup k8s-01 04

报错如下：

TASK [kube-node : 轮询等待kubelet启动] *******************************************************************************************************
changed: [172.18.60.202]
changed: [172.18.60.201]
FAILED - RETRYING: 轮询等待node达到Ready状态 (8 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (8 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (7 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (7 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (6 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (6 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (5 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (5 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (4 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (4 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (3 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (3 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (2 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (2 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (1 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (1 retries left).

TASK [kube-node : 轮询等待node达到Ready状态] ***************************************************************************************************
fatal: [172.18.60.201]: FAILED! => {"attempts": 8, "changed": true, "cmd": "/usr/bin/kubectl get node 172.18.60.201|awk 'NR>1{print $2}'", "delta": "0:00:00.263088", "end": "2021-09-01 12:26:16.691810", "rc": 0, "start": "2021-09-01 12:26:16.428722", "stderr": "Error from server (NotFound): nodes \"172.18.60.201\" not found", "stderr_lines": ["Error from server (NotFound): nodes \"172.18.60.201\" not found"], "stdout": "", "stdout_lines": []}
fatal: [172.18.60.202]: FAILED! => {"attempts": 8, "changed": true, "cmd": "/usr/bin/kubectl get node 172.18.60.202|awk 'NR>1{print $2}'", "delta": "0:00:00.283975", "end": "2021-09-01 12:26:17.056414", "rc": 0, "start": "2021-09-01 12:26:16.772439", "stderr": "Error from server (NotFound): nodes \"172.18.60.202\" not found", "stderr_lines": ["Error from server (NotFound): nodes \"172.18.60.202\" not found"], "stdout": "", "stdout_lines": []}

PLAY RECAP *****************************************************************************************************************************
172.18.60.201              : ok=53   changed=28   unreachable=0    failed=1    skipped=1    rescued=0    ignored=0
172.18.60.202              : ok=50   changed=26   unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

[root@k8s-master1 kubeasz]# 

使用 journalctl -u kubelet 查看日志

为了便于查看，我们可以将日志重定向到一个文件中

journalctl -u kubelet > /tmp/kubelet.log

发现错误

Sep 02 08:52:33 k8s-master1.example.com kubelet[612230]: E0902 08:52:33.526931  612230 server.go:292] "Failed to run kubelet" err="failed to run Kubelet: misconfiguration: kubelet cgroup driver: \"systemd\" is different from docker cgroup driver: \"cgroupfs\""
Sep 02 08:52:33 k8s-master1.example.com systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Sep 02 08:52:33 k8s-master1.example.com systemd[1]: kubelet.service: Failed with result 'exit-code'.

1.2 解决办法

[root@k8s-master1 ~]# cat /etc/docker/daemon.json
{
  "data-root": "/var/lib/docker",
  "exec-opts": ["native.cgroupdriver=systemd"],
  "registry-mirrors": [
    "https://docker.mirrors.ustc.edu.cn",
    "http://hub-mirror.c.163.com"
  ],
  "insecure-registries": ["127.0.0.1/8","172.18.8.214","172.18.8.215"],
  "max-concurrent-downloads": 10,
  "live-restore": true,
  "log-driver": "json-file",
  "log-level": "warn",
  "log-opts": {
    "max-size": "50m",
    "max-file": "1"
    },
  "storage-driver": "overlay2"
}
[root@k8s-master1 ~]#
[root@k8s-master1 ~]# docker info|grep "Cgroup Driver"
WARNING: No swap limit support
 Cgroup Driver: systemd
[root@k8s-master1 ~]#

注意：Ubuntu18 的 /etc/docker/daemon.json 文件如下

root@k8s-master-1:~# docker info|grep "Cgroup Driver"
 Cgroup Driver: cgroupfs
WARNING: No swap limit support
root@k8s-master-1:~# cat /etc/docker/daemon.json
{
  "data-root": "/var/lib/docker",
  "exec-opts": ["native.cgroupdriver=cgroupfs"],
  "insecure-registries": ["127.0.0.1/8","172.18.8.214"],
  "max-concurrent-downloads": 10,
  "live-restore": true,
  "log-driver": "json-file",
  "log-level": "warn",
  "log-opts": {
    "max-size": "50m",
    "max-file": "1"
    },
  "storage-driver": "overlay2"
}

2 错误2

Sep 02 16:31:17 k8s-master2.com kubelet[87623]: W0902 16:31:17.259389   87623 cni.go:204] Error validating CNI config list {"cniVersion":"0.3.1","name":"mynet","plugins":[{"bridge":"mynet0","cniVersion":"0.3.1","hairpinMode":true,"ipMasq":true,"ipam":{"subnet":"10.200.0.0/16","type":"host-local"},"isDefaultGateway":true,"name":"mynet","type":"bridge"}]}: [failed to find plugin "bridge" in path [/opt/cni/bin]]
Sep 02 16:31:17 k8s-master2.com kubelet[87623]: W0902 16:31:17.260166   87623 cni.go:204] Error validating CNI config list {
Sep 02 16:31:17 k8s-master2.com kubelet[87623]:   "name": "cbr0",
Sep 02 16:31:17 k8s-master2.com kubelet[87623]:   "cniVersion": "0.3.1",
Sep 02 16:31:17 k8s-master2.com kubelet[87623]:   "plugins": [
Sep 02 16:31:17 k8s-master2.com kubelet[87623]:     {
Sep 02 16:31:17 k8s-master2.com kubelet[87623]:       "type": "flannel",
Sep 02 16:31:17 k8s-master2.com kubelet[87623]:       "delegate": {
Sep 02 16:31:17 k8s-master2.com kubelet[87623]:         "hairpinMode": true,
Sep 02 16:31:17 k8s-master2.com kubelet[87623]:         "isDefaultGateway": true
Sep 02 16:31:17 k8s-master2.com kubelet[87623]:       }
Sep 02 16:31:17 k8s-master2.com kubelet[87623]:     },
Sep 02 16:31:17 k8s-master2.com kubelet[87623]:     {
Sep 02 16:31:17 k8s-master2.com kubelet[87623]:       "type": "portmap",
Sep 02 16:31:17 k8s-master2.com kubelet[87623]:       "capabilities": {
Sep 02 16:31:17 k8s-master2.com kubelet[87623]:         "portMappings": true
Sep 02 16:31:17 k8s-master2.com kubelet[87623]:       }
Sep 02 16:31:17 k8s-master2.com kubelet[87623]:     }
Sep 02 16:31:17 k8s-master2.com kubelet[87623]:   ]
Sep 02 16:31:17 k8s-master2.com kubelet[87623]: }
Sep 02 16:31:17 k8s-master2.com kubelet[87623]: : [failed to find plugin "flannel" in path [/opt/cni/bin] failed to find plugin "portmap" in path [/opt/cni/bin]]
Sep 02 16:31:17 k8s-master2.com kubelet[87623]: W0902 16:31:17.260197   87623 cni.go:239] Unable to update cni config: no valid networks found in /etc/cni/net.d
Sep 02 16:31:17 k8s-master2.com kubelet[87623]: E0902 16:31:17.328048   87623 kubelet.go:2243] node "k8s-master2.com" not found

2.1 原因及解决办法

此台机器使用过 kubeadm 安装过 k8s，kubeadm 默认采用 flannel 的网络模式，再次使用二进制安装 k8s 需要重新安装操作系统