k8s admission webhook初探

前言

之前一篇博客讨论的问题，就是如何杀掉container的主进程，并且让container不重启，最后有讲到说也许可以使用admission webhook，动态替换容器，做到给一个空容器，也就是deployment或者是statufulset的image标签，换成一个空镜像启动，然后在这里container中就可以java -jar启动自己的容器， 关闭自己的容器，完全当成是一个小型Linux来使用了，方便快速验证功能，不用每次都需要跑CI/CD。所以今天就来看看方案可行性，并且动手实践一把。

实践

废话不多说，让我们开始吧，首先需要环境准备：k8s 1.9版本以上一台

开启admission webhook功能

在k8s master节点上，编辑文件：

vim /etc/kubernetes/manifests/kube-apiserver.yaml

- --enable-admission-plugins=NodeRestriction

改为：

- --enable-admission-plugins=NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook

k8s会监听这里的文件变化，自动更新apiserver：

kubectl get pods -n kube-system

输出：

NAME                              READY   STATUS    RESTARTS   AGE
coredns-66bff467f8-jst9n          1/1     Running   19         111d
coredns-66bff467f8-ln6ct          1/1     Running   20         111d
etcd-master                       1/1     Running   23         111d
kube-apiserver-master             1/1     Running   3          12s
kube-controller-manager-master    1/1     Running   265        5h14m
kube-proxy-z9fhk                  1/1     Running   20         111d
kube-scheduler-master             1/1     Running   246        111d
metrics-server-6f9f86ddf9-77xbs   1/1     Running   116        100d
weave-net-wpgnh                   2/2     Running   61         111d

可以看见apiserver已经重启了，检查是否添加成功：

kubectl get pods kube-apiserver-master -n kube-system -o jsonpath='{.spec.containers[0].command[5]}'

输出：

--enable-admission-plugins=NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook

准备TLS用的密钥

新建文件csr.conf，输入以下内容：

[ req ]
default_bits = 2048
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn

[ dn ]
C = CN
ST = SiChuan
L = SZ
O = Wise2c
OU = Wise2c
CN = diyadmissionwebhook.test.svc #这里要注意填写服务名称是自定义服务名称.服务所在namespace名称.svc的形式

[ req_ext ]
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = diyadmissionwebhook.test.svc #这里要注意填写服务名称是自定义服务名称.服务所在namespace名称.svc的形式
 
[ v3_ext ]
authorityKeyIdentifier=keyid,issuer:always
basicConstraints=CA:FALSE
keyUsage=keyEncipherment,dataEncipherment
extendedKeyUsage=serverAuth,clientAuth
subjectAltName=@alt_names

然后新建脚本predeploy.sh，输入以下内容并执行：

#! /bin/bash

kubectl create namespace test
# 给namespace打上需要注入空容器的label，后边会用到
kubectl label namespace test inject-empty-container=enabled
openssl req -x509 -nodes -new -sha256 -days 3650 -newkey rsa:2048 -subj "/CN=diyadmissionwebhook.test.svc" \
  -keyout ca.key \
  -out ca.crt
# 注意上面的CN值
openssl genrsa -out server.key 2048
openssl req -new -key server.key -out server.csr -config csr.conf

openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key \
  -CAcreateserial -out server.crt -days 3650 \
  -extensions v3_ext -extfile csr.conf

kubectl create secret generic webhook-secret -n test \
  --from-file=./ca.crt \
  --from-file=./server.key \
  --from-file=./server.crt

rm ca.key ca.crt server.key server.crt

构建admission webhook程序

这里参考k8s官方demo，稍加修改，可以参考我写的例子，也是借鉴过来的。要注意使用TLS的密钥：

const (
	keyFile  = "/etc/webhook/certs/server.key"
	certFile = "/etc/webhook/certs/server.crt"
)

func main() {
	mux := http.NewServeMux()
	mux.HandleFunc("/mutating-service", serveMutateServices) // 提供两个接口出去
	mux.HandleFunc("/mutating-pods", serveMutatePods)
	log.Fatal(http.ListenAndServeTLS(":80", certFile, keyFile, mux))
}

修改容器为空容器的伪代码：

···
const (
	podsRemoveContainerPatch string = `[
		 {"op":"replace","path":"/spec/template/spec/containers/0/image","value":"empty-container:latest"}
	]`
)
reviewResponse := v1.AdmissionResponse{}
reviewResponse.Allowed = true
reviewResponse.Patch = []byte(podsRemoveContainerPatch)
pt := v1.PatchTypeJSONPatch
reviewResponse.PatchType = &pt
...可以添加更过的业务逻辑，这里只是简单的将deployment的容器，
...更换为空容器empty-container:latest

这里的空容器是指：

cat Dockerfile 

FROM alpine:3
ENTRYPOINT ["tail","-f","/dev/null"]

打上empty-container:latest的标签构建出来的，
将自己的webhook，配置为一个服务，贴上相应的webhook.yaml配置：

vim webhook.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: diyadmissionwebhook
  namespace: test
  labels:
    app: diyadmissionwebhook
spec:
  replicas: 1
  template:
    metadata:
      name: diyadmissionwebhook
      labels:
        app: diyadmissionwebhook
    spec:
      containers:
        - name: diyadmissionwebhook
          image: admissionwebhook:latest
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: webhook-certs
              mountPath: /etc/webhook/certs # 将secret内容的值，挂载到pod中使用，路径为/etc/webhook/certs/server.crt和/etc/webhook/certs/server.key
              readOnly: true
          ports:
            - containerPort: 80
              protocol: TCP
      volumes:
        - name: webhook-certs
          secret:
            secretName: webhook-secret
      restartPolicy: Always
  selector:
    matchLabels:
      app: diyadmissionwebhook
---
apiVersion: v1
kind: Service
metadata:
  name: diyadmissionwebhook
  namespace: test
spec:
  selector:
    app: diyadmissionwebhook
  ports:
    - port: 80
  type: NodePort
---
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: diyadmissionwebhook
  namespace: test
  labels:
    app: diyadmissionwebhook
webhooks: # 可配置多个webhook，指向同一个服务的不同接口
  - name: diyadmissionwebhook.test.svc
    clientConfig:
      service:
        name: diyadmissionwebhook
        namespace: test
        path: "/mutating-pods" #这里是服务暴露出来的接口地址
        port: 80
      caBundle: # 注意这里，需要填写生成的密钥ca.crt, base64后的值，不用写双引号
    rules:
      - apiGroups: [ "apps" ]
        apiVersions: [ "v1" ]
        operations: [ "CREATE","UPDATE" ]
        resources: [ "deployments" ]
        scope: "*"
    namespaceSelector:
      matchLabels:
        inject-empty-container: enabled
    admissionReviewVersions: [ "v1" ]
    sideEffects: "NoneOnDryRun"
  - name: diyadmissionwebhook.test.svc1
    admissionReviewVersions: [ "v1" ]
    clientConfig: # 如果下面的规则满足条件，则会调用这里定义的服务
      caBundle: # 注意这里，需要填写生成的密钥ca.crt, base64后的值，不用写双引号
      service:
        name: diyadmissionwebhook
        namespace: test
        path: "/mutating-service"
        port: 80
    rules: # 规则，监听这些变化，如果这写资源有定义的操作，则会通知自己的admission webhook服务
      - apiGroups: [ "apps" ]
        apiVersions: [ "v1" ]
        operations: [ "CREATE","UPDATE" ]
        resources: [ "services" ]
        scope: "*"
    sideEffects: "NoneOnDryRun"
    namespaceSelector: # 对于多个namespace生效，label匹配的namespace，都会监听
      matchLabels:
        inject-empty-container: enabled

注意：
caBundle标签的值，需要修改为下面命令执行的输出值

kubectl get secret webhook-secret -n test -o jsonpath='{.data.ca\.crt}'

输出：

LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5akNDQWJJQ0NRREhUeDl0V2ZXVlVqQU5CZ2txaGtpRzl3MEJBUXNGQURBbk1TVXdJd1lEVlFRRERCeGsKYVhsaFpHMXBjM05wYjI1M1pXSm9iMjlyTG5SbGMzUXVjM1pqTUI0WERUSXhNRE15TURFd01qUXpObG9YRFRNeApNRE14T0RFd01qUXpObG93SnpFbE1DTUdBMVVFQXd3Y1pHbDVZV1J0YVhOemFXOXVkMlZpYUc5dmF5NTBaWE4wCkxuTjJZekNDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFNQ2txb3gybUgyZE5sYVQKSUtzZ3pCZDErTzFnY0tBUDhyRlJoZVNhMlc3Z1FCc3lNeUp6Y3JsZG1QdTlkVzJFZFBkd1FGWndwY1JZNWZ6cQp2RGRkU2t6OG42MWphRmdBUnpPaCtJeFpkZHlBOHBIckVvZE4rVkllVm9IejJMRUpNN3E2bFBLcDlvc21FaXB1CnE0VXJ2TGxqOEFCWFY1dVdNL3JsZ01KaFlFSDlQa0h3bU1TN2U1QzVvMXNRbzJ6UE40NVpIbXR3cVBlR01LQUQKVFhDelNXc3BaNFZkbWpyQXZwTlZZNHNIRlpSTkhWM3Q2aUNBQmFDQkViVHVDakpvSURjaUJuSnMwd0J4Tml0SgozQm0rVW0xTTZvSjQ2a1NBcGVOSXRuOE1wZmtSYkhtSGpMNmd6Nzl0Y0JkaGRuUlBLd2tNUkdndEdadU53UHc0Cm96Nmw5TDhDQXdFQUFUQU5CZ2txaGtpRzl3MEJBUXNGQUFPQ0FRRUFuY2g3anZCMjdkWVhzV1JiOEpXR0kyN1YKdVBaQkVFRWZpZ1dPNTA1NU13SS9KSnJaU0pDTnd4cXcrenZOY1k0Ty80c3ZWQUNUS1FIZWdIV1dpK0I5MkU4SwpwUUNvMXZVa1FTTW5qTFNYTEUzQjlROENQcHQ4Z2dBTjNVUitPa0NVUlFYWGFwSHFGLzlvU2pkdS9DYlZ0dUdzCmZtK1VvQjkydVhrOCtXRjlCVWRYTFJQeThYQnd1eWV0a2ZEMTBleGEzYW95bC81bElwcGNVRmZPNlRlUENEdzIKUVNMek5Db24zcnA3ai9zU2phaVQvbkx1RkFtQXFQMHZlRUMvU1hKaDdaYzUwT0hBdW1Nd0g3anVKaEJJeTh1bgpwVzFnSFJkZEMzVjJNWCtISmZCdlhLRXVpb3ZjVm83eTNQQUdNN2dYRG9NRGFRWkltME5ST1paMnVBYTNnZz09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K

由于不支持secret直接引用，会稍微复杂些。

kubectl apply -f webhook.yaml

kubectl get pods -n test

NAME                                   READY   STATUS    RESTARTS   AGE
diyadmissionwebhook-6445fc5cbb-vn9hb   1/1     Running   0          74s

可以看见自定义的admission webhook服务已经起来了。

以上就已经基本完成了自定义的admission webhook了

验证

验证yaml：

cat test.yaml

输出：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test
  namespace: test
spec:
  selector:
    matchLabels:
      test: test
  replicas: 1
  template:
    metadata:
      labels:
        test: test
    spec:
      containers:
        - image: knative:latest # 这里随便使用一个镜像，观察这里会不会被替换为empty-contaienr:latest
          imagePullPolicy: IfNotPresent
          name: test
          ports:
            - containerPort: 80

执行：

kubectl apply -f test.yaml

输出

deployment.apps/test created

查看deployment的image标签

kubectl get deployment test -n test -o jsonpath='{.spec.template.spec.containers[0].image}'

输出

empty-container:latest

可以看见，已经成功替换为空镜像了。

后记

简单说一下admission webhook，这里是到apiserver之前的一次拦截，类似与SpringMVC中的filter一样，这里会串行执行。也就是如果rule标签满足条件的话，无论是自定义的webhook还是k8s自带的webhook，都会按照顺序执行。然后在这个webhook中做自己的业务逻辑判断，比如k8s自带的资源大小限制，istio中注入envoy的sidecar都是使用这样的方式做的。具体更多的信息，还是参见官方文档

可能遇到的问题

• 使用官方demo，版本需要将v1beta1.AdmissionReview改为v1.AdmissionReview，否则可能会报错：

no kind "AdmissionReview" is registered for version "admission.k8s.io/v1" in scheme "pkg/runtime/scheme.go:101"

• 使用官方demo，版本需要手动加上kind和apiversion：

responseAdmissionReview := v1.AdmissionReview{
		TypeMeta: metav1.TypeMeta{
			Kind:       "AdmissionReview",
			APIVersion: "admission.k8s.io/v1",
		},
	}

否则可能会报错：

expected webhook response of admission.k8s.io/v1, Kind=AdmissionReview, got /, Kind=