kubeedge部署(keadm)
萧晓博
一、预备环境
kubeedge是基于kubernetes运行的，所以首先需要搭建kubernetes集群作为云端部署。
准备条件
- 需要部署kubernetes主结点作为cloudcore部署的结点
- 不需要部署kube-flannel网络插件
- 不需要在边端部署kube-proxy
二、kubeedge安装
1.到官网下载keadm安装包
# 直接下载
$ wget https://github.com/kubeedge/kubeedge/releases/download/v1.7.1/keadm-v1.7.1-linux-amd64.tar.gz
# 解压缩
$ tar -zxvf keadm-v1.7.1-linux-amd64.tar.gz
# 添加环境变量
$ mv ./keadm /usr/local/bin/
2.初始化cloudcore结点
$ keadm init --advertise-address="THE-EXPOSED-IP"(only work since 1.3 release)
遇到github访问不了，查询域名对应的ip，添加到/etc/hosts
# GitHub Start
52.74.223.119 github.com
192.30.253.119 gist.github.com
54.169.195.247 api.github.com
185.199.111.153 assets-cdn.github.com
185.199.110.133 raw.githubusercontent.com
151.101.108.133 user-images.githubusercontent.com
185.199.110.133 gist.githubusercontent.com
185.199.110.133 cloud.githubusercontent.com
185.199.110.133 camo.githubusercontent.com
185.199.110.133 avatars0.githubusercontent.com
185.199.110.133 avatars1.githubusercontent.com
185.199.110.133 avatars2.githubusercontent.com
185.199.110.133 avatars3.githubusercontent.com
185.199.110.133 avatars4.githubusercontent.com
185.199.110.133 avatars5.githubusercontent.com
185.199.110.133 avatars6.githubusercontent.com
185.199.110.133 avatars7.githubusercontent.com
185.199.110.133 avatars8.githubusercontent.com
# GitHub End
3.添加edgecore结点
# 1.获取主结点token
$ keadm gettoken
# 2.加入边结点
$ keadm join --cloudcore-ipport=192.168.11.100:10000 --token=6ee3556fc2d8e6736326e701b17169a90663f5de4ee37ad9e795ea9f76f8dcbe.eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2MjY2ODM4MDd9.tAaT0GwHWMLQ330sx0dBteNiQS2IeJH_fVmdip_gHfk
# 指定证书目录
--certPath="/etc/kubeedge/certs"
4.安装metrics-server
启用 kubectl logs 功能
kubectl logs 必须在使用 metrics-server 之前部署，通过以下操作激活功能：
-
确保您可以找到 Kubernetes 的 ca.crt 和 ca.key 文件。如果您通过 kubeadm 安装Kubernetes 集群，这些文件将位于 /etc/kubernetes/pki/ 目录中。
ls /etc/kubernetes/pki/ -
设置 CLOUDCOREIPS 环境。环境变量设置为指定的 cloudcore 的IP地址，如果您具有高可用的集群，则可以指定VIP（即弹性IP/虚拟IP）。
export CLOUDCOREIPS="192.168.11.100"
（警告：建议使用同一 终端 来保持系统工作的持续，在必要时再次键入此命令。）使用以下命令检查环境变量：
echo $CLOUDCOREIPS -
在云端节点上为 CloudStream 生成证书，但是，生成的文件不在 /etc/kubeedge/ 中，我们需要从GitHub的存储库中拷贝一份。
将用户更改为root：
sudo su
从原始克隆的存储库中拷贝证书：
cp $GOPATH/src/github.com/kubeedge/kubeedge/build/tools/certgen.sh /etc/kubeedge/
将目录更改为kubeedge目录：
cd /etc/kubeedge/
从 **certgen.sh** 生成证书
/etc/kubeedge/certgen.sh stream
- 需要在主机上设置 iptables。（此命令应该在每个apiserver部署的节点上执行。）（在这种情况下，须在master节点上执行，并由root用户执行此命令。） 在运行每个apiserver的主机上运行以下命令：
注意: 您需要先设置CLOUDCOREIPS变量
$ iptables -t nat -A OUTPUT -p tcp --dport 10350 -j DNAT --to $CLOUDCOREIPS:10003
# 端口10003和10350是 CloudStream 和 Edgecore 的默认端口，如果已发生变更，请使用您自己设置的端口。
如果您不确定是否设置了iptables，并且希望清除所有这些表。（如果您错误地设置了iptables，它将阻止您使用 kubectl logs 功能） 可以使用以下命令清理iptables规则：
$ iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X
- /etc/kubeedge/config/cloudcore.yaml 和 /etc/kubeedge/config/edgecore.yaml 上 cloudcore 和 edgecore 都要 修改。将 cloudStream 和 edgeStream 设置为 enable: true 。将服务器IP更改为 cloudcore IP（与 $ CLOUDCOREIPS 相同）。
在 cloudcore 中打开 YAML 文件：
vim /etc/kubeedge/config/cloudcore.yaml
在以下文件中修改( enable: true )内容：
cloudStream:
enable: true
streamPort: 10003
tlsStreamCAFile: /etc/kubeedge/ca/streamCA.crt
tlsStreamCertFile: /etc/kubeedge/certs/stream.crt
tlsStreamPrivateKeyFile: /etc/kubeedge/certs/stream.key
tlsTunnelCAFile: /etc/kubeedge/ca/rootCA.crt
tlsTunnelCertFile: /etc/kubeedge/certs/server.crt
tlsTunnelPrivateKeyFile: /etc/kubeedge/certs/server.key
tunnelPort: 10004
在 edgecore 中打开 YAML 文件：
vim /etc/kubeedge/config/edgecore.yaml
修改以下部分中的文件 (enable: true), (server: 192.168.0.193:10004):
edgeStream:
enable: true
handshakeTimeout: 30
readDeadline: 15
server: 192.168.0.139:10004
tlsTunnelCAFile: /etc/kubeedge/ca/rootCA.crt
tlsTunnelCertFile: /etc/kubeedge/certs/server.crt
tlsTunnelPrivateKeyFile: /etc/kubeedge/certs/server.key
writeDeadline: 15
- 重新启动所有cloudcore和edgecore
sodu su
cloudCore:
pkill cloudcore
nohup cloudcore > cloudcore.log 2>&1 &
edgeCore
systemctl restart edgecore.service
如果您无法重启 edgecore，请检查是否是由于 kube-proxy 的缘故，同时杀死这个进程。 kubeedge 默认不纳入该进程，我们使用 edgemesh 来进行替代
注意： 可以考虑避免 kube-proxy 部署在edgenode上。有两种解决方法：
- 通过调用 kubectl edit daemonsets.apps -n kube-system kube-proxy 添加以下设置：
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-role.kubernetes.io/edge
operator: DoesNotExist
- 如果您仍然要运行 kube-proxy ，请通过在以下位置添加 edgecore.service 中的 env 变量来要求 edgecore 不进行检查edgecore.service：
sudo vi /etc/kubeedge/edgecore.service
- 将以下行添加到 edgecore.service 文件：
Environment="CHECK_EDGECORE_ENVIRONMENT=false"
- 最终文件应如下所示：
Description=edgecore.service
[Service]
Type=simple
ExecStart=/root/cmd/ke/edgecore --logtostderr=false --log-file=/root/cmd/ke/edgecore.log
Environment="CHECK_EDGECORE_ENVIRONMENT=false"
[Install]
WantedBy=multi-user.target
在云端支持 Metrics-server
-
实现该功能点的是重复使用了 cloudstream 和 edgestream 模块。因此，您还需要执行 启用 kubectl logs 功能 的所有步骤。
-
部署metrics-server(需要0.4.0及以上版本)
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
k8s-app: metrics-server
rbac.authorization.k8s.io/aggregate-to-admin: "true"
rbac.authorization.k8s.io/aggregate-to-edit: "true"
rbac.authorization.k8s.io/aggregate-to-view: "true"
name: system:aggregated-metrics-reader
rules:
- apiGroups:
- metrics.k8s.io
resources:
- pods
- nodes
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
k8s-app: metrics-server
name: system:metrics-server
rules:
- apiGroups:
- ""
resources:
- pods
- nodes
- nodes/stats
- namespaces
- configmaps
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
k8s-app: metrics-server
name: metrics-server-auth-reader
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
k8s-app: metrics-server
name: metrics-server:system:auth-delegator
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
k8s-app: metrics-server
name: system:metrics-server
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:metrics-server
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
spec:
ports:
- name: https
port: 443
protocol: TCP
targetPort: https
selector:
k8s-app: metrics-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
spec:
selector:
matchLabels:
k8s-app: metrics-server
strategy:
rollingUpdate:
maxUnavailable: 0
template:
metadata:
labels:
k8s-app: metrics-server
spec:
hostNetwork: true
containers:
- args:
- --cert-dir=/tmp
- --secure-port=4443
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
- --kubelet-use-node-status-port
- --metric-resolution=15s
- --kubelet-insecure-tls
- --kubelet-use-node-status-port
image: bitnami/metrics-server:0.4.0
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /livez
port: https
scheme: HTTPS
periodSeconds: 10
name: metrics-server
ports:
- containerPort: 4443
name: https
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /readyz
port: https
scheme: HTTPS
initialDelaySeconds: 20
periodSeconds: 10
resources:
requests:
cpu: 100m
memory: 200Mi
securityContext:
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
volumeMounts:
- mountPath: /tmp
name: tmp-dir
nodeSelector:
master: "true"
priorityClassName: system-cluster-critical
serviceAccountName: metrics-server
volumes:
- emptyDir: {}
name: tmp-dir
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
labels:
k8s-app: metrics-server
name: v1beta1.metrics.k8s.io
spec:
group: metrics.k8s.io
groupPriorityMinimum: 100
insecureSkipTLSVerify: true
service:
name: metrics-server
namespace: kube-system
version: v1beta1
versionPriority: 100
[root@m1 ~]# cat metrics-server-deployment.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
k8s-app: metrics-server
rbac.authorization.k8s.io/aggregate-to-admin: "true"
rbac.authorization.k8s.io/aggregate-to-edit: "true"
rbac.authorization.k8s.io/aggregate-to-view: "true"
name: system:aggregated-metrics-reader
rules:
- apiGroups:
- metrics.k8s.io
resources:
- pods
- nodes
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
k8s-app: metrics-server
name: system:metrics-server
rules:
- apiGroups:
- ""
resources:
- pods
- nodes
- nodes/stats
- namespaces
- configmaps
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
k8s-app: metrics-server
name: metrics-server-auth-reader
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
k8s-app: metrics-server
name: metrics-server:system:auth-delegator
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
k8s-app: metrics-server
name: system:metrics-server
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:metrics-server
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
spec:
ports:
- name: https
port: 443
protocol: TCP
targetPort: https
selector:
k8s-app: metrics-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
spec:
selector:
matchLabels:
k8s-app: metrics-server
strategy:
rollingUpdate:
maxUnavailable: 0
template:
metadata:
labels:
k8s-app: metrics-server
spec:
hostNetwork: true
containers:
- args:
- --cert-dir=/tmp
- --secure-port=4443
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
- --kubelet-use-node-status-port
- --metric-resolution=15s
- --kubelet-insecure-tls
- --kubelet-use-node-status-port
image: bitnami/metrics-server:0.4.0
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /livez
port: https
scheme: HTTPS
periodSeconds: 10
name: metrics-server
ports:
- containerPort: 4443
name: https
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /readyz
port: https
scheme: HTTPS
initialDelaySeconds: 20
periodSeconds: 10
resources:
requests:
cpu: 100m
memory: 200Mi
securityContext:
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
volumeMounts:
- mountPath: /tmp
name: tmp-dir
nodeSelector:
master: "true"
priorityClassName: system-cluster-critical
serviceAccountName: metrics-server
volumes:
- emptyDir: {}
name: tmp-dir
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
labels:
k8s-app: metrics-server
name: v1beta1.metrics.k8s.io
spec:
group: metrics.k8s.io
groupPriorityMinimum: 100
insecureSkipTLSVerify: true
service:
name: metrics-server
namespace: kube-system
version: v1beta1
versionPriority: 100
重要提示
- Metrics-server需要使用主机网络网络模式。
- 使用您自己编译的镜像，并将 imagePullPolicy 设置为Never。
- 为 Metrics 服务器启用 –kubelet-use-node-status-port 功能
5.删除边结点
# cloud
$ keadm reset # --kube-config=$HOME/.kube/config
# edge
$ keadm reset # 节点 keadm reset 将停止 edgecore ，并且不会卸载/删除任何先决条件。
# 逐个停止进程，删除相关文件依赖
rm -rf /var/lib/kubeedge /var/lib/edged /etc/kubeedge
rm -rf /etc/systemd/system/edgecore.service
rm -rf /usr/local/bin/edgecore
6.遇到的问题
1.edgecore与docker的CgroupDtriver不同
# 查看日志
journalctl -f -u edgecore
# 注意可能是docker和edge的cgroup角色不同，edge默认cgroupfs，修改docker的daemon.json
{
"exec-opts": ["native.cgroupdriver=cgroupfs"]
}
# 也可以修改edgecore的配制为systemd,/etc/kubeedge/edgecore.yaml
modules:
edged:
cgroupDriver: systemd
2.边端应用要用hostNetwork模式部署
kubeedge默认没有提供类似于flannel这样的CNI插件，所以pod网络无法跨主机通信，设置为hostNetwork后，直接用主机端口访问应用
类似资料：