Kubespray安装kubernetes集群

齐健柏

2023-12-01

Kubespray安装kubernetes集群

Kubespray是什么？

Kubespray 是Kubernetes incubator 中的项目，目标是提供Production Ready Kubernetes部署方案，该项目基础是通过Ansible Playbook 来定义系统与Kubernetes 集群部署的任务。

kubespray部署环境信息：

Role	hostname	IP
Master	sam-master-1	192.168.0.248
Master	sam-master-2	192.168.0.141
Master	sam-master-3	192.168.0.53
Worker	sam-node-1	192.168.0.222

1.Kubespray安装准备

1.1.准备Bootstrap node

Bootstrap node用于部署k8s集群，需要安装的软件可参考以下内容：

https://github.com/kubernetes-incubator/kubespray#requirements
https://github.com/kubernetes-incubator/kubespray/blob/master/requirements.txt

1.1.1. 安装ansible

yum install -y ansible

1.1.2.安装python-netadd

安装pip

yum -y install epel-release
yum -y install python-pip

安装python-netaddr

pip install netaddr

1.1.3.升级Jinja

pip install --upgrade jinja2

1.2.配置boot-strap节点

1.2.1.关闭防火墙和selinux

systemctl stop firewalld
systemctl disable firewalld
iptables -F

1.2.2.关闭交换分区

swapoff -a

2.Kubespray安装

2.1.下载kubespray源码

git clone https://github.com/kubernetes-incubator/kubespray.git

默认Bootstrap node上的ansible通过ssh key的方式连接节点，如果需要支持ssh密码方式连接，则需要安装python sshpass模块

curl -O -L http://downloads.sourceforge.net/project/sshpass/sshpass/1.05/sshpass-1.05.tar.gz
tar zxvf sshpass-1.05.tar.gz
cd sshpass-1.05
./configure
make
make install

2.2. 修改配置文件

2.2.1. inventory.ini主机清单文件

inventory.ini主要是节点及其信息的配置文件，路径：kubespray/inventory/sample/inventory.ini
#拷贝sample文件

[all] 分组配置机器的IP，用户名，密码，还有etcd节点的名称
[kube_control_plane] 该分组主要安装kubernetes master的组件
[etcd] 该分组主要安装etcd的组件
[kube-node] 该分组主要安装worker节点的组件

cp -rfp inventory/sample inventory/sam
vi inventory/sam/inventory.ini
[all]
sam-master-1 ansible_host=192.168.0.248 ansible_user=root etcd_member_name=etc1 ansible_ssh_pass=XXX
sam-master-2 ansible_host=192.168.0.141 ansible_user=root etcd_member_name=etc2 ansible_ssh_pass=XXX
sam-master-3 ansible_host=192.168.0.53 ansible_user=root etcd_member_name=etc3 ansible_ssh_pass=XXX
sam-node-1 ansible_host=192.168.0.222 ansible_user=root ansible_ssh_pass=XXX

# configure a bastion host if your nodes are not directly reachable
#[bastion]
#bastion ansible_host=192.168.0.227 ansible_user=root 

[kube_control_plane]
sam-master-1
sam-master-2
sam-master-3

[etcd]
sam-master-1
sam-master-2
sam-master-3

[kube-node]
sam-node-1

[k8s-cluster:children]
kube_control_plane
kube-node

2.2.2.k8s-cluster.yml集群安装参数文件

k8s-cluster.yml主要为k8s集群的配置文件，路径为：kubespray/inventory/k8s/group_vars/k8s-cluster.yml。该文件可以修改安装的k8s集群的版本，参数为：kube_version: v1.9.6。具体可参考：

https://github.com/kubernetes-incubator/kubespray/blob/master/inventory/sample/group_vars/k8s-cluster.yml#L22

2.2.3.通过kubespray部署kubernetes

部署k8s集群

cd kubespray
ansible-playbook -i inventory/sam/inventory.ini cluster.yml -b -vvv

2.2.4. 检查ansible的部署结果

PLAY RECAP ********************************************************************************************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
sam-master-1               : ok=561  changed=109  unreachable=0    failed=0    skipped=1120 rescued=0    ignored=1   
sam-master-2               : ok=493  changed=95   unreachable=0    failed=0    skipped=978  rescued=0    ignored=0   
sam-master-3               : ok=495  changed=96   unreachable=0    failed=0    skipped=976  rescued=0    ignored=0   
sam-node-1                 : ok=358  changed=62   unreachable=0    failed=0    skipped=612  rescued=0    ignored=0   

Friday 09 April 2021  12:08:36 +0800 (0:00:00.055)       0:23:17.098 ********** 
=============================================================================== 
download_file | Download item ---------------------------------------------------------------------------------------------------------------- 440.57s
/root/kubespray-master/roles/download/tasks/download_file.yml:51 -------------------------------------------------------------------------------------
download_file | Download item ---------------------------------------------------------------------------------------------------------------- 210.54s
/root/kubespray-master/roles/download/tasks/download_file.yml:51 -------------------------------------------------------------------------------------
download_file | Download item ---------------------------------------------------------------------------------------------------------------- 151.26s
/root/kubespray-master/roles/download/tasks/download_file.yml:51 -------------------------------------------------------------------------------------
download_file | Download item ----------------------------------------------------------------------------------------------------------------- 66.51s
/root/kubespray-master/roles/download/tasks/download_file.yml:51 -------------------------------------------------------------------------------------
download_file | Download item ----------------------------------------------------------------------------------------------------------------- 48.99s
/root/kubespray-master/roles/download/tasks/download_file.yml:51 -------------------------------------------------------------------------------------
container-engine/docker : ensure docker packages are installed -------------------------------------------------------------------------------- 43.75s
/root/kubespray-master/roles/container-engine/docker/tasks/main.yml:102 ------------------------------------------------------------------------------
kubernetes/kubeadm : Join to cluster ---------------------------------------------------------------------------------------------------------- 35.93s
/root/kubespray-master/roles/kubernetes/kubeadm/tasks/main.yml:72 ------------------------------------------------------------------------------------
kubernetes/control-plane : Joining control plane node to the cluster. ------------------------------------------------------------------------- 31.09s
/root/kubespray-master/roles/kubernetes/control-plane/tasks/kubeadm-secondary.yml:52 -----------------------------------------------------------------
download_container | Download image if required ----------------------------------------------------------------------------------------------- 17.41s
/root/kubespray-master/roles/download/tasks/download_container.yml:55 --------------------------------------------------------------------------------
download_file | Download item ----------------------------------------------------------------------------------------------------------------- 15.35s
/root/kubespray-master/roles/download/tasks/download_file.yml:51 -------------------------------------------------------------------------------------
download_container | Download image if required ----------------------------------------------------------------------------------------------- 14.27s
/root/kubespray-master/roles/download/tasks/download_container.yml:55 --------------------------------------------------------------------------------
kubernetes/control-plane : kubeadm | Initialize first master ---------------------------------------------------------------------------------- 12.07s
/root/kubespray-master/roles/kubernetes/control-plane/tasks/kubeadm-setup.yml:114 --------------------------------------------------------------------
download_container | Download image if required ----------------------------------------------------------------------------------------------- 11.00s
/root/kubespray-master/roles/download/tasks/download_container.yml:55 --------------------------------------------------------------------------------
reload etcd ----------------------------------------------------------------------------------------------------------------------------------- 10.45s
/root/kubespray-master/roles/etcd/handlers/main.yml:23 -----------------------------------------------------------------------------------------------
download_container | Download image if required ----------------------------------------------------------------------------------------------- 10.10s
/root/kubespray-master/roles/download/tasks/download_container.yml:55 --------------------------------------------------------------------------------
download_container | Download image if required ------------------------------------------------------------------------------------------------ 6.48s
/root/kubespray-master/roles/download/tasks/download_container.yml:55 --------------------------------------------------------------------------------
Configure | Check if etcd cluster is healthy --------------------------------------------------------------------------------------------------- 5.20s
/root/kubespray-master/roles/etcd/tasks/configure.yml:2 ----------------------------------------------------------------------------------------------
Gen_certs | Write etcd member and admin certs to other etcd nodes ------------------------------------------------------------------------------ 4.95s
/root/kubespray-master/roles/etcd/tasks/gen_certs_script.yml:92 --------------------------------------------------------------------------------------
wait for etcd up ------------------------------------------------------------------------------------------------------------------------------- 4.92s
/root/kubespray-master/roles/etcd/handlers/main.yml:35 -----------------------------------------------------------------------------------------------
Gen_certs | Write etcd member and admin certs to other etcd nodes ------------------------------------------------------------------------------ 4.91s
/root/kubespray-master/roles/etcd/tasks/gen_certs_script.yml:92 --------------------------------------------------------------------------------------

2.2.5. 检查k8s集群部署情况

获取kubernetes节点运行情况

kubectl get node
NAME           STATUS   ROLES    AGE     VERSION
sam-master-1   Ready    master   4m30s   v1.19.6
sam-master-2   Ready    master   4m7s    v1.19.6
sam-master-3   Ready    master   3m58s   v1.19.6
sam-node-1     Ready    <none>   3m6s    v1.19.6

获取kubernetes component运行状态

kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS      MESSAGE                                                                                       ERROR
scheduler            Unhealthy   Get "http://127.0.0.1:10251/healthz": dial tcp 127.0.0.1:10251: connect: connection refused   
controller-manager   Unhealthy   Get "http://127.0.0.1:10252/healthz": dial tcp 127.0.0.1:10252: connect: connection refused   
etcd-0               Healthy     {"health":"true"}                                                                             
etcd-2               Healthy     {"health":"true"}                                                                             
etcd-1               Healthy     {"health":"true"}

Unhealthy的原因是scheduler和controller-manager配置不正确导致的，scheduler和controller-manager因为在manifest文件中添加了–port=0所以显示不正常,去掉配置后，显示状态正常.

kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok                  
controller-manager   Healthy   ok                  
etcd-2               Healthy   {"health":"true"}   
etcd-1               Healthy   {"health":"true"}   
etcd-0               Healthy   {"health":"true"}

Kubespray安装kubernetes集群