【ClickHouse系列】使用clickhouse-operator搭建clickhouse集群
蓬野
clickhouse-operator特性
- 根据提供的自定义资源规范创建ClickHouse群集
- 自定义存储资源调配（VolumeClaim模板）
- 自定义pod模板
- 自定义service模板
- ClickHouse配置和设置（包括Zookeeper集成）
- 柔性模板
- ClickHouse集群自动扩展
- ClickHouse版本升级
- ClickHouse指标导出Prometheus
要求：Kubernetes 1.12.6+
搭建方法
0.自建一套Kubernetes集群
方法参考：CentOS7 Kubernetes 1.14.1安装、启动、验证及踩坑
1.下载源码
git clone https://github.com/Altinity/clickhouse-operator.git
2.安装clickhouse-operator
执行clickhouse-operator/deploy/operator/clickhouse-operator-install.sh这个脚本
[root@k8s-master operator]# pwd
/home/ch/clickhouse-operator/deploy/operator
[root@k8s-master operator]# ./clickhouse-operator-install.sh
Setup ClickHouse Operator into kube-system namespace
1. Build manifest
2. No need to create kube-system namespace
3. Install operator into kube-system namespace
customresourcedefinition.apiextensions.k8s.io/clickhouseinstallations.clickhouse.altinity.com created
customresourcedefinition.apiextensions.k8s.io/clickhouseinstallationtemplates.clickhouse.altinity.com created
customresourcedefinition.apiextensions.k8s.io/clickhouseoperatorconfigurations.clickhouse.altinity.com created
serviceaccount/clickhouse-operator created
clusterrolebinding.rbac.authorization.k8s.io/clickhouse-operator created
configmap/etc-clickhouse-operator-files created
configmap/etc-clickhouse-operator-confd-files created
configmap/etc-clickhouse-operator-configd-files created
configmap/etc-clickhouse-operator-templatesd-files created
configmap/etc-clickhouse-operator-usersd-files created
deployment.apps/clickhouse-operator created
service/clickhouse-operator-metrics created
查看状态，直到clickhouse-operator-75695fddb8-9rwbc为Running，这样operator即安装成功
[root@k8s-master chi-examples]# kubectl get pod -A -l app=clickhouse-operator
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system clickhouse-operator-75695fddb8-9rwbc 2/2 Running 2 73m
使用官方小例子验证是否operator可用，使用这个例子01-simple-layout-02-1shard-2repl.yaml
[root@k8s-master chi-examples]# pwd
/home/ch/clickhouse-operator/docs/chi-examples
[root@k8s-master chi-examples]# kubectl apply -f 01-simple-layout-02-1shard-2repl.yaml
clickhouseinstallation.clickhouse.altinity.com/simple-02 created
第一次创建会有点慢，因为会取拉取clickhouse-server官方镜像查询详情，会发现Pulling image "yandex/clickhouse-server:latest"
[root@k8s-master chi-examples]# kubectl describe pod chi-simple-02-shard1-repl1-0-0-0
Name: chi-simple-02-shard1-repl1-0-0-0
Namespace: default
Priority: 0
PriorityClassName: <none>
Node: k8s-node1/192.168.80.134
Start Time: Tue, 17 Mar 2020 14:57:05 +0800
Labels: clickhouse.altinity.com/app=chop
clickhouse.altinity.com/chi=simple-02
clickhouse.altinity.com/chiScopeCycleIndex=0
clickhouse.altinity.com/chiScopeCycleOffset=0
clickhouse.altinity.com/chiScopeCycleSize=0
clickhouse.altinity.com/chiScopeIndex=0
clickhouse.altinity.com/chop=0.9.2
clickhouse.altinity.com/cluster=shard1-repl1
clickhouse.altinity.com/clusterScopeCycleIndex=0
clickhouse.altinity.com/clusterScopeCycleOffset=0
clickhouse.altinity.com/clusterScopeCycleSize=0
clickhouse.altinity.com/clusterScopeIndex=0
clickhouse.altinity.com/namespace=default
clickhouse.altinity.com/replica=0
clickhouse.altinity.com/replicaScopeIndex=0
clickhouse.altinity.com/settings-version=73d42865206a76de2371ae93f33a21c234c7dca7
clickhouse.altinity.com/shard=0
clickhouse.altinity.com/shardScopeIndex=0
clickhouse.altinity.com/zookeeper-version=cea9bfc4a19073e068023bb6a2f873b66cd1cb9d
controller-revision-hash=chi-simple-02-shard1-repl1-0-0-6695d584b
statefulset.kubernetes.io/pod-name=chi-simple-02-shard1-repl1-0-0-0
Annotations: <none>
Status: Pending
IP:
Controlled By: StatefulSet/chi-simple-02-shard1-repl1-0-0
Containers:
clickhouse:
Container ID:
Image: yandex/clickhouse-server:latest
Image ID:
Ports: 8123/TCP, 9000/TCP, 9009/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Readiness: http-get http://:http/ping delay=10s timeout=1s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/etc/clickhouse-server/conf.d/ from chi-simple-02-deploy-confd-shard1-repl1-0-0 (rw)
/etc/clickhouse-server/config.d/ from chi-simple-02-common-configd (rw)
/etc/clickhouse-server/users.d/ from chi-simple-02-common-usersd (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-pcvlq (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
chi-simple-02-common-configd:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: chi-simple-02-common-configd
Optional: false
chi-simple-02-common-usersd:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: chi-simple-02-common-usersd
Optional: false
chi-simple-02-deploy-confd-shard1-repl1-0-0:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: chi-simple-02-deploy-confd-shard1-repl1-0-0
Optional: false
default-token-pcvlq:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-pcvlq
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 105s default-scheduler Successfully assigned default/chi-simple-02-shard1-repl1-0-0-0 to k8s-node1
Normal Pulling 104s kubelet, k8s-node1 Pulling image "yandex/clickhouse-server:latest"
直到两个pod实例都变为Running，即operator功能正常
[root@k8s-master chi-examples]# kubectl get pod
NAME READY STATUS RESTARTS AGE
chi-simple-02-shard1-repl1-0-0-0 1/1 Running 0 4m31s
chi-simple-02-shard1-repl1-0-1-0 1/1 Running 0 2m22s
clickhouse-operator yaml模板
Kind:ClickHouseInstallation：一个自定义的controller，集成了一些自定义模板（如podTemplates、serviceTemplates、dataVolumeClaimTemplate、logVolumeClaimTemplate等）及一些自定义参数（如clusters、zookeeper、users、profiles等），方便创建ch集群
yaml例子
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
# ch集群名称，创建的pod名称例如chi-demo-{clusterName}-0-0-0，{}中下面会说明
name: "demo"
spec:
defaults:
templates:
# service模板
serviceTemplate: service-template
# pod模板，实际上内部使用的是StatefulSet
podTemplate: pod-template
# 数据文件pvc模板
dataVolumeClaimTemplate: volume-claim
# 日志文件pvc模板
logVolumeClaimTemplate: volume-claim
configuration:
# zk主机名（域名）及端口信息，可配置多个
zookeeper:
nodes:
- host: zookeeper-0
port: 2181
clusters:
# ch集群的clusterName，名字会填充上面的{clusterName}部分
- name: "cluster-name"
layout:
# ch集群的分片数和每个分片的副本数
shardsCount: 2
replicasCount: 2
templates:
serviceTemplates:
- name: service-template
spec:
ports:
- name: http
port: 8123
- name: tcp
port: 9000
type: LoadBalancer
podTemplates:
- name: pod-template
spec:
containers:
- name: clickhouse
imagePullPolicy: Always
image: yandex/clickhouse-server:latest
volumeMounts:
# 挂载数据文件路径
- name: volume-claim
mountPath: /var/lib/clickhouse
# 挂载数据文件路径
- name: volume-claim
mountPath: /var/log/clickhouse-server
resources:
# 配置cpu和内存大小
limits:
memory: "64Mi"
cpu: "0.2"
requests:
memory: "64Mi"
cpu: "0.2"
volumeClaimTemplates:
- name: volume-claim
spec:
accessModes:
- ReadWriteOnce
resources:
# pv的存储大小
requests:
storage: 1Gi
这里没有配置StorageClass，因为operator的一些用法是需要aws支持的，aws的k8s环境一半都会提供default StorageClass，所以不用配置也可以创建成功，如果是自建的Kubernetes集群，是不能成功的，会发现报错
[root@k8s-master yaml]# kubectl describe pod chi-demo-cluster-name-0-0-0
Name: chi-demo-cluster-name-0-0-0
......
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 79s (x2 over 79s) default-scheduler pod has unbound immediate PersistentVolumeClaims (repeated 2 times)
[root@k8s-master yaml]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
volume-claim-chi-demo-cluster-name-0-0-0 Pending 2m35s
[root@k8s-master yaml]# kubectl describe pvc volume-claim-chi-demo-cluster-name-0-0-0
Name: volume-claim-chi-demo-cluster-name-0-0-0
......
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal FailedBinding 4s (x15 over 3m6s) persistentvolume-controller no persistent volumes available for this claim and no storage class is set
Mounted By: chi-demo-cluster-name-0-0-0
提示没有可用的持久化卷，其实这里可以自行创建pvc，创建pv来时间pod的挂载。
如何使用StorageClass
这里为了趋向于aws的方式，自行搭建基于nfs的StorageClass用于pod挂载，一般类似aws，都会有一个provisioner来提供pvc与pv的动态绑定，所以先要创建provisioner，这里用直接托管给k8s的方式来创建，但是前提是有一套可用的nfs，所以要先找一台机器搭建nfs
使用Deployment方式来创建，但是前提是有一套可用的nfs，所以要先找一台机器搭建nfs
搭建nfs
yum install -y nfs-utils
#保证相关网段的机器都可以挂载
vim /etc/exports
/data 10.244.0.0/16(rw,sync)
/data 172.17.0.0/16(rw,sync)
/data 192.168.80.0/16(rw,sync)
systemctl enable rpcbind.service
systemctl enable nfs-server.service
systemctl start rpcbind.service
systemctl start nfs-server.service
#使配置生效
exportfs -r
#查看配置
exportfs
/data 10.244.0.0/16
/data 172.17.0.0/16
/data 192.168.80.0/16
创建provisioner
provisioner.yaml文件如下，包含认证及创建
apiVersion: v1
kind: ServiceAccount
metadata:
name: provisioner
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: provisioner-runner
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["list", "watch", "create", "update", "patch"]
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: run-provisioner
subjects:
- kind: ServiceAccount
name: provisioner
namespace: default
roleRef:
kind: ClusterRole
name: provisioner-runner
apiGroup: rbac.authorization.k8s.io
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: provisioner
spec:
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: provisioner
spec:
serviceAccountName: provisioner
containers:
- name: provisioner
image: xiedeyantu/nfs-client-provisioner
volumeMounts:
- name: root
mountPath: /persistentvolumes
env:
- name: PROVISIONER_NAME
value: demo/nfs
- name: NFS_SERVER
value: 10.244.1.1
- name: NFS_PATH
value: /data
volumes:
- name: root
nfs:
# server ip是一台搭建了nfs的机器在10.244网段的ip
server: 10.244.1.1
path: /data
直接创建，直至provisioner-5749684665-62q2p为Running
[root@k8s-master yaml]# kubectl apply -f provisioner.yaml
serviceaccount/provisioner created
clusterrole.rbac.authorization.k8s.io/provisioner-runner created
clusterrolebinding.rbac.authorization.k8s.io/run-provisioner created
deployment.extensions/provisioner created
[root@k8s-master yaml]# kubectl get pod
NAME READY STATUS RESTARTS AGE
provisioner-5749684665-62q2p 1/1 Running 0 17m
最后就可以创建StorageClass了，yaml如下
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: sc-nfs
# 同上面的PROVISIONER_NAME
provisioner: demo/nfs
创建并设置默认sc
[root@k8s-master yaml]# kubectl apply -f sc.yaml
storageclass.storage.k8s.io/sc-nfs created
[root@k8s-master yaml]# kubectl patch storageclass sc-nfs -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
storageclass.storage.k8s.io/sc-nfs patched
[root@k8s-master yaml]# kubectl get sc
NAME PROVISIONER AGE
sc-nfs (default) demo/nfs 7s
一系列复杂过程结束后，就可以使用operator的那个yaml模板创建了
ch集群创建
创建集群
[root@k8s-master yaml]# kubectl apply -f demo.yaml
clickhouseinstallation.clickhouse.altinity.com/demo created
[root@k8s-master yaml]# kubectl get pod
NAME READY STATUS RESTARTS AGE
chi-demo-cluster-name-0-0-0 2/2 Running 0 4m12s
chi-demo-cluster-name-0-1-0 2/2 Running 0 3m7s
chi-demo-cluster-name-1-0-0 2/2 Running 0 3m2s
chi-demo-cluster-name-1-1-0 2/2 Running 0 62s
provisioner-5749684665-62q2p 1/1 Running 0 38m
查看service
root@ubuntu:/opt/code# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
chi-demo-cluster-name-0-0 ClusterIP None <none> 8123/TCP,9000/TCP,9009/TCP 12m
chi-demo-cluster-name-0-1 ClusterIP None <none> 8123/TCP,9000/TCP,9009/TCP 11m
chi-demo-cluster-name-1-0 ClusterIP None <none> 8123/TCP,9000/TCP,9009/TCP 9m
chi-demo-cluster-name-1-1 ClusterIP None <none> 8123/TCP,9000/TCP,9009/TCP 5m
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 4h52m
service-demo LoadBalancer 10.105.81.230 <pending> 8123:32268/TCP,9000:30966/TCP 2m56s
到此可以使用CLUSTER-IP：10.105.81.230登陆ch集群，本地搭建的不支持EXTERNAL-IP
类似资料：