基于Vagrant搭建异构Kubernetes集群

准备工作

为节约时间,相关文件可以提前下载好,当然也可以在master搭建好后在master上下载

网络规划

vip.kamputer.online=192.168.1.100
master.kamputer.online=192.168.1.101
ubuntu.kamputer.online=192.168.1.102
centos.kamputer.online=192.168.1.103
node.kamputer.online=192.168.1.111
service网段：10.96.0.0/12
pod网段：172.16.0.1/12

# 网卡的名字,配置keepalived时需要
NET_INTERFACE=eth1
# 根据上面的网络规划确定Service网段起始地址
SERVICE_ADDRESS=10.96.0.1
# 初始化服务器节点地址,存有各种预下载文件及证书
INITER_ADDRESS="master.kamputer.online"
# 确定ETCD节点地址
ETCD_NODES="master.kamputer.online,ubuntu.kamputer.online,centos.kamputer.onilne"
# kubernetes master地址
KUBERNETES_MASTERS="master.kamputer.online,ubuntu.kamputer.online,centos.kamputer.onilne"
# 根据上面的网络规范确定vip地址,确定ApiServer在vip中的端口,这个其实是HAProxy端口,然后代理到后面ApiServer的6443端口
VIP_HOSTNAME=vip.kamputer.online
VIP_IP=$(nslookup $VIP_HOSTNAME|tail -2|head -1|awk '{print $2}')
VIP_PORT=8443
#如果是单master模式,使用下面配置
# VIP_ADDRESS=master.kamputer.online
# VIP_PORT=6443

# 获取自己的对外IP
LOCAL_IP=$(ip a|grep $NET_INTERFACE|grep inet|awk '{print $2}'|awk -F '/' '{print $1}')

下载好软件

• repo
  centos使用国内源,需要repo文件

 wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo

• haproxy
  centos是自带haproxy的,但是版本是1.5,已经过期了.ubuntu需要使用apt下载,我使用的ubuntu2204,自带的是2.0,既然要安装,索性两个都安装最新LTS
  去官网找一个合适的版本,下载源代码.我选的是2.4,过期时间是2026-Q2 (LTS),使用下面指令下载.

wget https://www.haproxy.org/download/2.4/src/haproxy-2.4.16.tar.gz

• Kubernetes

wget https://storage.googleapis.com/kubernetes-release/release/v1.19.0/kubernetes-server-linux-amd64.tar.gz

• etcd
  etcd的版本需要和Kubernetes匹配

wget https://github.com/etcd-io/etcd/releases/download/v3.4.13/etcd-v3.4.13-linux-amd64.tar.gz

• cfssl
  在cfssl的github可以看到最新版本,我使用的是v1.6.1

wget "https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl_1.6.1_linux_amd64" -O /usr/local/bin/cfssl
wget "https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssljson_1.6.1_linux_amd64" -O /usr/local/bin/cfssljson

操作系统安装

下载box

安装vagrant
下载box文件到本地

	vmware	virtualbox
centos7	883cf9e5-6e65-437c-acfa-fd1e7b3d9b8f	10b58d3b-00ae-4146-86d4-5b06e8aaa9c2
ubuntu20.04	a3d12620-a14f-427f-b4c9-7e3d19ed47b4	fd8e3d47-f8d1-4db5-979e-d85294bb1563

导入box

进入box所在目录执行下面命令

vagrant box add --name generic/ubuntu2004 .\a3d12620-a14f-427f-b4c9-7e3d19ed47b4
vagrant box add --name generic/ubuntu2004 .\fd8e3d47-f8d1-4db5-979e-d85294bb1563
vagrant box add --name generic/centos7 .\10b58d3b-00ae-4146-86d4-5b06e8aaa9c2
vagrant box add --name generic/centos7 .\883cf9e5-6e65-437c-acfa-fd1e7b3d9b8f

启动虚拟机（单机版）

初始化

vagrant init generic/centos7
vagrant init generic/ubuntu2004

修改配置文件

# 使用静态ip
config.vm.network "public_network", ip: "192.168.1.101"
# 修改CPU和内存
config.vm.provider "virtualbox" do |vb|
# Customize the amount of memory on the VM:
  vb.memory = "4096"
  vb.cpus =  2
end

启动

vagrant up
# 需要选择一个合适的网卡

启动虚拟机（集群版）

Vagrantfile

# -*- mode: ruby -*-
# vi: set ft=ruby :

# All Vagrant configuration is done below. The "2" in Vagrant.configure
# configures the configuration version (we support older styles for
# backwards compatibility). Please don't change it unless you know what
# you're doing.
Vagrant.configure("2") do |config|
  config.vm.provision "shell",inline: "echo kubernetes"
  
  config.vm.define "master" do |master|
    master.vm.provision "shell",inline: "echo master"
    master.vm.box = "generic/centos7"
    master.vm.network "public_network", ip: "192.168.1.101"
    master.vm.provider "virtualbox" do |vb|
      vb.memory = "4096"
      vb.cpus =  2
    end
  end

  config.vm.define "ubuntu" do |ubuntu|
    ubuntu.vm.provision "shell",inline: "echo ubuntu"
    ubuntu.vm.box = "generic/ubuntu2004"
    ubuntu.vm.network "public_network", ip: "192.168.1.102"
    ubuntu.vm.provider "virtualbox" do |vb|
      vb.memory = "4096"
      vb.cpus =  2
    end
  end

  config.vm.define "centos" do |centos|
    centos.vm.provision "shell",inline: "echo centos"
    centos.vm.box = "generic/centos7"
    centos.vm.network "public_network", ip: "192.168.1.103"
    centos.vm.provider "virtualbox" do |vb|
      vb.memory = "4096"
      vb.cpus =  2
    end
  end  

  config.vm.define "node" do |node|
    node.vm.provision "shell",inline: "echo node"
    node.vm.box = "generic/ubuntu2004"
    node.vm.network "public_network", ip: "192.168.1.111"
    node.vm.provider "virtualbox" do |vb|
      vb.memory = "8196"
      vb.cpus =  2
    end
  end

end

使用vagrant up可以一起启动所有虚拟机（对于virtualbox而言无法并行）
后面使用vagrant ssh指令时需要加上虚拟机名字

配置操作系统

vagrant ssh

保存操作记录

echo "HISTFILESIZE=99999" >> ~/.bashrc 
echo "HISTSIZE=99999" >> ~/.bashrc 
echo 'HISTTIMEFORMAT="%F %T "'>> ~/.bashrc  
# 命令立刻写入而不是退出时写入
echo 'PROMPT_COMMAND="history -a"' >> ~/.bashrc
exit

允许远程登录

# 允许root账号登录
sed -i 's/#PermitRootLogin yes/PermitRootLogin yes/g' /etc/ssh/sshd_config
# 允许密码登录
sed -i 's/PasswordAuthentication no/PasswordAuthentication yes/g' /etc/ssh/sshd_config
# 让配置生效
systemctl restart sshd
# 修改密码
passwd

配置源

CentOS

wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo && \
yum clean all && \
yum makecache && \
yum update

Ubuntu

# 备份源文件
cp /etc/apt/sources.list{,.backup} && \
# 使用清华源
echo '
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ focal main restricted universe multiverse
deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ focal main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ focal-updates main restricted universe multiverse
deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ focal-updates main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ focal-backports main restricted universe multiverse
deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ focal-backports main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ focal-security main restricted universe multiverse
deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ focal-security main restricted universe multiverse multiverse
' > sources.list && \
cp ./sources.list /etc/apt/sources.list && \
apt update && \
apt upgrade -y

其他设置

设置时间

timedatectl set-timezone 'Asia/Shanghai'

更复杂的方式可以参见https://www.linuxprobe.com/linux-time.html

时间同步

CentOS

无需安装

Ubuntu

apt install chrony -y

解除资源限制

下面指令的具体含义可以参见ulimit和/etc/security/limits.conf详解

ulimit -SHn 65535 && \
echo '
* soft nofile 655360
* hard nofile 131072
* soft nproc 655350
* hard nproc 655350
* seft memlock unlimited
* hard memlock unlimited
'>>/etc/security/limits.conf

haproxy & keepalived

签发证书

安装工具

etcd证书

# 创建CA根证书
echo '
{
  "CN": "etcd",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Shanghai",
      "L":"Shanghai",
      "O":"etcd",
      "OU":"Etcd Security"
    }
  ],
  "ca": {
    "expiry": "8760h"
  }
}'>etcd-ca-csr.json && \
cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare /etc/etcd/ssl/etcd-ca && \

# 设置CA配置
echo '
 {
   "signing": {
     "default": {
       "expiry": "8760h"
     },
     "profiles": {
       "kubernetes": {
         "usages": [
             "signing",
             "key encipherment",
             "server auth",
             "client auth"
         ],
         "expiry": "8760h"
       }
     }
   }
 }
 '>ca-config.json && \

# 生成etcd证书的配置
echo '{
  "CN": "etcd",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Shanghai",
      "L": "Shanghai",
      "O": "etcd",
      "OU": "Etcd Security"
    }
  ]
}'>etcd-ca-csr.json && \

cfssl gencert \
   -ca=/etc/etcd/ssl/etcd-ca.pem \
   -ca-key=/etc/etcd/ssl/etcd-ca-key.pem \
   -config=ca-config.json \
   -hostname=127.0.0.1,$ETCD_NODES \
   -profile=kubernetes \
   etcd-csr.json | cfssljson -bare /etc/etcd/ssl/etcd && \

#其他节点复制证书
for FILE in etcd-ca-key.pem  etcd-ca.pem  etcd-key.pem  etcd.pem; do
   scp $INITER_ADDRESS:/etc/etcd/ssl/${FILE} /etc/etcd/ssl/${FILE}
done

验证

可以使用下面命令来验证ca.pem

 openssl x509 -noout -text -in /etc/etcd/ssl/etcd-ca.pem 

输出如下
其中Issuer和申请信息是一致的

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            02:76:b1:68:b9:66:37:57:46:16:d3:07:df:3e:9b:6d:f0:02:33:f1
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=CN, ST=Shanghai, L=Shanghai, O=etcd, OU=Etcd Security, CN=etcd
        Validity
            Not Before: May 14 07:14:00 2022 GMT
            Not After : May 14 07:14:00 2023 GMT
        Subject: C=CN, ST=Shanghai, L=Shanghai, O=etcd, OU=Etcd Security, CN=etcd
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:c4:8b:f9:d1:a8:68:ad:f3:ed:c3:57:db:0c:aa:
                    37:04:81:a0:18:6a:fb:f9:f4:8e:24:b9:1d:b4:f6:
                    50:56:47:c8:c6:3e:8c:be:16:9b:fa:bf:2a:29:c3:
                    14:ca:e5:0d:9b:23:28:08:ca:de:47:b7:67:d6:ab:
                    3e:56:5e:25:82:bc:02:87:13:55:47:c8:a8:53:23:
                    af:ff:82:a1:98:80:bc:e8:3f:0c:f1:83:c5:d6:ac:
                    27:a3:40:5d:d0:be:2f:71:cb:a8:e7:2b:ec:70:45:
                    9c:fa:c7:13:9d:7a:41:f2:5a:35:a5:3e:84:2a:73:
                    0e:8e:5c:3d:88:13:46:55:3f:dd:1b:1b:a9:97:68:
                    5d:d0:84:bf:d1:fd:8b:e5:c6:d9:a2:96:3f:7b:4f:
                    86:b8:b1:e2:7f:fd:f5:8f:03:8f:25:14:39:d1:51:
                    82:94:d9:4e:f7:f8:a9:2d:34:1d:91:90:1e:f9:2e:
                    77:14:da:0a:f4:55:d4:99:b1:a7:bd:9b:eb:fa:94:
                    55:56:d8:ec:b1:50:48:1c:fc:45:65:ce:28:17:69:
                    6e:bf:ca:c3:d9:69:35:da:ea:3a:50:e0:5e:8b:1b:
                    8f:d7:a5:97:93:25:b3:1d:20:55:44:da:b6:3f:91:
                    a9:6a:6f:31:3b:2c:7a:95:42:c6:24:98:6a:79:88:
                    9a:5b
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Subject Key Identifier: 
                B9:D9:F5:9C:95:AA:CB:F6:A1:E4:C3:61:DA:74:3D:36:A2:61:57:10
    Signature Algorithm: sha256WithRSAEncryption
         35:c0:57:37:48:23:78:92:e3:c5:22:07:3f:0b:02:33:03:29:
         e0:1a:8a:c4:b1:ac:69:da:70:39:09:9f:80:d0:c4:6f:78:36:
         96:c6:37:5f:a1:8c:8c:0e:0e:61:d5:ad:44:fb:33:84:98:1e:
         84:d3:db:9b:a0:28:28:78:64:9d:53:9a:04:a7:23:52:6f:dd:
         90:ab:fd:e0:5c:4b:56:1d:95:09:4d:af:f8:b7:fc:5b:75:ef:
         d5:ba:40:51:92:23:e5:df:6e:ae:fe:93:46:75:54:53:a5:b4:
         01:a7:55:cf:e5:3f:b6:84:b6:c9:14:41:21:fc:25:d4:8b:7f:
         04:d6:d1:74:04:d2:d6:b5:a3:c7:f8:e7:93:eb:1b:82:d1:8d:
         44:06:e6:9b:7b:20:63:36:8a:9d:03:41:c6:ff:37:a9:e0:ab:
         53:75:e6:ce:f0:91:54:e7:ae:90:fe:13:40:48:39:00:df:b9:
         e8:c8:0d:5a:0d:f7:b2:35:8f:3c:1f:a3:fc:00:e2:07:a1:4e:
         4f:e0:5f:0b:21:6e:15:3b:4f:aa:b4:0f:48:73:93:7b:69:b6:
         2e:fd:a4:75:1d:dc:97:47:9c:3b:94:a6:68:af:30:6a:73:90:
         f1:ae:b3:77:93:1d:d8:8b:42:90:2c:21:35:85:a0:db:e0:8b:
         25:15:e4:00

Kubernetes证书

# 生成CA证书
echo '{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Shanghai",
      "L": "Shanghai",
      "O": "Kubernetes",
      "OU": "Kubernetes-manual"
    }
  ],
  "ca": {
    "expiry": "8760h"
  }
}
'>ca-csr.json && \
cfssl gencert -initca ca-csr.json | cfssljson -bare /etc/kubernetes/pki/ca && \

# ca配置文件
echo '{
  "signing": {
    "default": {
      "expiry": "8760h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "8760h"
      }
    }
  }
}
'>ca-config.json && \

# 签发apiserver证书,这里直接使用管道符,就不用给cfssl指定参数了
echo '{
  "CN": "kube-apiserver",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Shanghai",
      "L": "Shanghai",
      "O": "Kubernetes",
      "OU": "Kubernetes-manual"
    }
  ]
}'|cfssl gencert  \
-ca=/etc/kubernetes/pki/ca.pem   \
-ca-key=/etc/kubernetes/pki/ca-key.pem   \
-config=ca-config.json   \
-hostname=$SERVICE_ADDRESS,$VIP_ADDRESS,127.0.0.1,$KUBERNETES_MASTERS,\
	kubernetes,kubernetes.default,kubernetes.default.svc,\
	kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local  \
-profile=kubernetes | cfssljson -bare /etc/kubernetes/pki/apiserver

# 签发admin证书
echo '{
  "CN": "admin",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Shanghai",
      "L": "Shanghai",
      "O": "system:masters",
      "OU": "Kubernetes-manual"
    }
  ]
}'|cfssl gencert \
   -ca=/etc/kubernetes/pki/ca.pem \
   -ca-key=/etc/kubernetes/pki/ca-key.pem \
   -config=ca-config.json \
   -profile=kubernetes | cfssljson -bare /etc/kubernetes/pki/admin

# 签发proxy证书
for module in 'kube-proxy' 'kube-controller-manager' 'apiserver'
do
echo "{
  'CN': 'system:$module',
  'key': {
    'algo': 'rsa',
    'size': 2048
  },
  'names': [
    {
      'C': 'CN',
      'ST': 'Shanghai',
      'L': 'Shanghai',
      'O': 'system:$module',
      'OU': 'Kubernetes-manual'
    }
  ]
}"|cfssl gencert \
   -ca=/etc/kubernetes/pki/ca.pem \
   -ca-key=/etc/kubernetes/pki/ca-key.pem \
   -config=ca-config.json \
   -profile=kubernetes | cfssljson -bare /etc/kubernetes/pki/$module
done

AllInOne

centos

在这里插入代码片