库伯内特斯：负载均衡器与就绪健康检查

Question

我正在Kubernetes（GKE）中运行WebService后端应用程序。它仅由我们的前端Web应用程序使用。通常，来自同一用户（ClientIP）的请求有数十个序列。我的应用程序设置为运行至少2个实例（“minReplicas:2”）。

问题：从日志中我可以看到一个pod过载（接收许多请求）而另一个闲置的情况。两个吊舱都处于就绪状态。

我试图修复它：我试图添加一个自定义就绪状态检查，当有太多打开的连接时，该检查将返回“不健康”状态。但即使在运行状况检查返回“不正常”后，负载平衡器也会在第二个（正常）pod空闲时向同一个pod发送进一步的请求。

这是服务的摘录。yaml：

kind: Service
metadata:
  annotations:
    networking.gke.io/load-balancer-type: "Internal"
spec:
  type: LoadBalancer
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080

未指定sessionAffinity，因此我希望它为“None”

我的问题：我做错了什么？就绪状态检查对负载平衡器有影响吗？如何控制请求分发？

补充信息：

集群创建：

gcloud container --project %PROJECT% clusters create %CLUSTER% 
  --zone "us-east1-b" --release-channel "stable" --machine-type "n1-standard-2" 
  --disk-type "pd-ssd" --disk-size "20" --metadata disable-legacy-endpoints=true 
  --scopes "storage-rw" --num-nodes "1" --enable-stackdriver-kubernetes 
  --enable-ip-alias --network "xxx" --subnetwork "xxx" 
  --cluster-secondary-range-name "xxx" --services-secondary-range-name "xxx" 
  --no-enable-master-authorized-networks 

节点池：

gcloud container node-pools create XXX --project %PROJECT% --zone="us-east1-b" 
  --cluster=%CLUSTER% --machine-type=c2-standard-4 --max-pods-per-node=16 
  --num-nodes=1 --disk-type="pd-ssd" --disk-size="10" --scopes="storage-full" 
  --enable-autoscaling --min-nodes=1 --max-nodes=30

服务：

apiVersion: v1
kind: Service
metadata:
  name: XXX
  annotations:
    networking.gke.io/load-balancer-type: "Internal"
  labels:
    app: XXX
    version: v0.1
spec:
  selector:
    app: XXX
    version: v0.1
  type: LoadBalancer
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080

HPA：

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: XXX
spec:
  scaleTargetRef:
    apiVersion: "apps/v1"
    kind:       Deployment
    name:       XXX
  minReplicas: 2
  maxReplicas: 30
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 40
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 70

部署：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: XXX
  labels:
    app: XXX
    version: v0.1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: XXX
      version: v0.1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0

  template:
    metadata:
      labels:
        app: XXX
        version: v0.1
    spec:
      containers:
      - image: XXX
        name: XXX
        imagePullPolicy: Always        
        resources:
          requests:
            memory: "10Gi"
            cpu: "3200m"
          limits:
            memory: "10Gi"
            cpu: "3600m"
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 8080
          initialDelaySeconds: 3
          periodSeconds: 8
          failureThreshold: 3                        
        livenessProbe:
          httpGet:
            path: /health/live
            port: 8080
          initialDelaySeconds: 120
          periodSeconds: 30   
      nodeSelector:
        cloud.google.com/gke-nodepool: XXX

Answer 1

发布此社区wiki答案以扩展我对复制步骤的评论。

我已经复制了您的设置，但我无法复制您遇到的问题。请求被平均分配。至于映像，我使用了纯nginx，所有测试都显示使用率/平衡在~50%（来自容器的日志，它们的cpu使用率）。请检查您的设置中nginx映像是否会发生同样的情况？

我遵循的复制步骤：

• 运行以下脚本，该脚本将创建网络、子网、群集并添加节点池：

project_id="INSERT_PROJECT_ID_HERE"
zone="us-east1-b"
region="us-east1"

gcloud compute networks create vpc-network --project=$project_id --subnet-mode=auto --mtu=1460 --bgp-routing-mode=regional
gcloud compute firewall-rules create vpc-network-allow-icmp --project=$project_id --network=projects/$project_id/global/networks/vpc-network --description=Allows\ ICMP\ connections\ from\ any\ source\ to\ any\ instance\ on\ the\ network. --direction=INGRESS --priority=65534 --source-ranges=0.0.0.0/0 --action=ALLOW --rules=icmp
gcloud compute firewall-rules create vpc-network-allow-internal --project=$project_id --network=projects/$project_id/global/networks/vpc-network --description=Allows\ connections\ from\ any\ source\ in\ the\ network\ IP\ range\ to\ any\ instance\ on\ the\ network\ using\ all\ protocols. --direction=INGRESS --priority=65534 --source-ranges=10.128.0.0/9 --action=ALLOW --rules=all
gcloud compute firewall-rules create vpc-network-allow-rdp --project=$project_id --network=projects/$project_id/global/networks/vpc-network --description=Allows\ RDP\ connections\ from\ any\ source\ to\ any\ instance\ on\ the\ network\ using\ port\ 3389. --direction=INGRESS --priority=65534 --source-ranges=0.0.0.0/0 --action=ALLOW --rules=tcp:3389
gcloud compute firewall-rules create vpc-network-allow-ssh --project=$project_id --network=projects/$project_id/global/networks/vpc-network --description=Allows\ TCP\ connections\ from\ any\ source\ to\ any\ instance\ on\ the\ network\ using\ port\ 22. --direction=INGRESS --priority=65534 --source-ranges=0.0.0.0/0 --action=ALLOW --rules=tcp:22
gcloud compute networks subnets update vpc-network --region=$region --add-secondary-ranges=service-range=10.1.0.0/16,pods-range=10.2.0.0/16
gcloud container --project $project_id clusters create cluster --zone $zone --release-channel "stable" --machine-type "n1-standard-2" --disk-type "pd-ssd" --disk-size "20" --metadata disable-legacy-endpoints=true --scopes "storage-rw" --num-nodes "1" --enable-stackdriver-kubernetes --enable-ip-alias --network "vpc-network" --subnetwork "vpc-network" --cluster-secondary-range-name "pods-range" --services-secondary-range-name "service-range" --no-enable-master-authorized-networks 
gcloud container node-pools create second-pool --project $project_id --zone=$zone --cluster=cluster --machine-type=n1-standard-4 --max-pods-per-node=16 --num-nodes=1 --disk-type="pd-ssd" --disk-size="10" --scopes="storage-full" --enable-autoscaling --min-nodes=1 --max-nodes=5
gcloud container clusters get-credentials cluster --zone=$zone --project=$project_id
# n1-standard-4 used rather than c2-standard-4

• 使用以下清单计划群集上的工作负载：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx
        imagePullPolicy: Always        
        resources:
          requests:
            memory: "10Gi"
            cpu: "3200m"
          limits:
            memory: "10Gi"
            cpu: "3200m"
      nodeSelector:
        cloud.google.com/gke-nodepool: second-pool
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  annotations:
    networking.gke.io/load-balancer-type: "Internal"
  labels:
    app: nginx
spec:
  selector:
    app: nginx
  type: LoadBalancer
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80

• $kubectl获取节点

NAME                                     STATUS   ROLES    AGE     VERSION
gke-cluster-default-pool-XYZ             Ready    <none>   3h25m   v1.18.17-gke.1901
gke-cluster-second-pool-one              Ready    <none>   83m     v1.18.17-gke.1901
gke-cluster-second-pool-two              Ready    <none>   83m     v1.18.17-gke.1901
gke-cluster-second-pool-three            Ready    <none>   167m    v1.18.17-gke.1901

• <代码>$kubectl获取吊舱-o宽

NAME                     READY   STATUS    RESTARTS   AGE   IP          NODE                                    NOMINATED NODE   READINESS GATES
nginx-7db7cf7c77-4ttqb   1/1     Running   0          85m   10.2.1.6    gke-cluster-second-pool-three          <none>           <none>
nginx-7db7cf7c77-dtwc8   1/1     Running   0          85m   10.2.1.34   gke-cluster-second-pool-two            <none>           <none>
nginx-7db7cf7c77-r6wv2   1/1     Running   0          85m   10.2.1.66   gke-cluster-second-pool-one            <none>           <none>

测试是在可以访问内部负载平衡器的同一区域中使用VM进行的。

使用的工具/命令：

• <代码>$ab-100000http://INTERNAL_LB_IP_ADDRESS/

日志相应地显示了每个pod的请求：