库伯内特斯CPU节流，CPU使用率远低于请求/限制

Question

我对我的pod的所有容器设置了CPU和内存请求=限制，以使其符合有保证的服务质量等级。现在，查看过去6小时内同一Pod的CPU使用率和CPU节流图。

这看起来是不是很正常，也在意料之中？

CPU使用率甚至一次都没有达到设定限制的50%，但有时仍被限制在58%。

还有一个附带的问题，节流图中25%的红线表示什么？

我对这个主题做了一些研究，发现内核中有一个错误Linux可能导致这个问题，它在内核的4.18版本中得到了修复。参考：这个和这个

我们在GKE上运行谷歌的容器优化操作系统。我检查了我们节点上的linux内核版本，它们在4.19.112上，所以我想我们已经有了那个补丁？这种节流模式的原因还有什么？

P、 这个pod（实际上是一个具有自动缩放功能的部署）部署在一个单独的节点池上，该节点池上没有运行我们的其他工作负载。因此，除了此部署之外，在该节点池中的节点上运行的唯一pod是一些度量、日志代理和导出程序。以下是在调度上述pod的同一节点上运行的pod的完整列表。确实有一些吊舱没有设置任何CPU限制。我是否也需要对这些设置CPU限制？

我们的GKE版本是1.16.9-GKE。2.

下面是包含部署、服务和自动缩放器定义的清单文件。

apiVersion: apps/v1
kind: Deployment
metadata:
  name: endpoints
  labels:
    app: endpoints
spec:
  replicas: 2
  selector:
    matchLabels:
      run: endpoints
  strategy:
    rollingUpdate:
      maxSurge: 2
      maxUnavailable: 0
  template:
    metadata:
      labels:
        run: endpoints
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: endpoints
          image: gcr.io/<PROJECT_ID>/endpoints:<RELEASE_VERSION_PLACEHOLDER>
          livenessProbe:
            httpGet:
              path: /probes/live
              port: 8080
            initialDelaySeconds: 20
            timeoutSeconds: 5
          readinessProbe:
            httpGet:
              path: /probes/ready
              port: 8080
            initialDelaySeconds: 20
            timeoutSeconds: 5
          ports:
            - containerPort: 8080
              protocol: TCP
          env:
            - name: GOOGLE_APPLICATION_CREDENTIALS
              value: "/path/to/secret/gke-endpoints-deployments-access.json"
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: DEPLOYMENT_NAME
              value: "endpoints"
          resources:
            requests:
              memory: "5Gi"
              cpu: 2
            limits:
              memory: "5Gi"
              cpu: 2
          volumeMounts:
            - name: endpoints-gcp-access
              mountPath: /path/to/secret
              readOnly: true
          lifecycle:
            preStop:
              exec:
                # SIGTERM triggers a quick exit; gracefully terminate instead
                command: ["/bin/sh","-c","sleep 3; /usr/sbin/nginx -s quit; sleep 57"]
        # [START proxy_container]
        - name: cloudsql-proxy
          image: gcr.io/cloudsql-docker/gce-proxy:1.16
          command: ["/cloud_sql_proxy",
                    "-instances=<PROJECT_ID>:<ZONE>:prod-db=tcp:3306,<PROJECT_ID>:<ZONE>:prod-db-read-replica=tcp:3307",
                    "-credential_file=/path/to/secret/gke-endpoints-deployments-access.json"]
          # [START cloudsql_security_context]
          securityContext:
            runAsUser: 2  # non-root user
            allowPrivilegeEscalation: false
          # [END cloudsql_security_context]
          resources:
            requests:
              memory: "50Mi"
              cpu: 0.1
            limits:
              memory: "50Mi"
              cpu: 0.1
          volumeMounts:
            - name: endpoints-gcp-access
              mountPath: /path/to/secret
              readOnly: true
        # [END proxy_container]
        # [START nginx-prometheus-exporter container]
        - name: nginx-prometheus-exporter
          image: nginx/nginx-prometheus-exporter:0.7.0
          ports:
            - containerPort: 9113
              protocol: TCP
          env:
            - name: CONST_LABELS
              value: "app=endpoints"
          resources:
            requests:
              memory: "50Mi"
              cpu: 0.1
            limits:
              memory: "50Mi"
              cpu: 0.1
        # [END nginx-prometheus-exporter container]
      tolerations:
        - key: "qosclass"
          operator: "Equal"
          value: "guaranteed"
          effect: "NoSchedule"
      nodeSelector:
        qosclass: guaranteed
      # [START volumes]
      volumes:
        - name: endpoints-gcp-access
          secret:
            secretName: endpoints-gcp-access
      # [END volumes]
---
apiVersion: cloud.google.com/v1beta1
kind: BackendConfig
metadata:
  name: endpoints-backendconfig
spec:
  timeoutSec: 60
  connectionDraining:
    drainingTimeoutSec: 60
---
apiVersion: v1
kind: Service
metadata:
  name: endpoints
  labels:
    app: endpoints
  annotations:
    cloud.google.com/neg: '{"ingress": true}' # Creates a NEG after an Ingress is created
    beta.cloud.google.com/backend-config: '{"ports": {"80":"endpoints-backendconfig"}}'
spec:
  type: NodePort
  selector:
    run: endpoints
  ports:
    - name: endpoints-nginx
      port: 80
      protocol: TCP
      targetPort: 8080
    - name: endpoints-metrics
      port: 81
      protocol: TCP
      targetPort: 9113
---
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: endpoints-autoscaler
spec:
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      targetAverageUtilization: 40
  - type: External
    external:
      metricName: external.googleapis.com|prometheus|nginx_http_requests_total
      metricSelector:
        matchLabels:
          metric.labels.app: endpoints
      targetAverageValue: "5"
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: endpoints
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: endpoints-nginx-monitor
  namespace: monitoring
  labels:
    app: endpoints-nginx-monitor
    chart: prometheus-operator-8.13.7
    release: prom-operator
    heritage: Tiller
spec:
  selector:
    matchLabels:
      app: endpoints
  namespaceSelector:
    any: true
  endpoints:
  - port: endpoints-metrics
    path: "/metrics"

这是部署中使用的唯一自定义容器映像的dockerfile：

# Dockerfile extending the generic PHP image with application files for a
# single application.
FROM gcr.io/google-appengine/php:latest

# The Docker image will configure the document root according to this
# environment variable.
ENV DOCUMENT_ROOT /app

RUN /bin/bash /stackdriver-files/enable_stackdriver_integration.sh

Answer 1

你们通常会根据你们设定的阈值看到这样的线条，我相信你们可以在设定阈值时选择线条的颜色

Answer 2

我不知道那条红线是什么，所以我将跳过那条红线。不过，很高兴知道您期望CPU节流情况会发生什么。

因此，关于您的CPU使用率和节流，没有迹象表明会出现任何问题。当有大量CPU可用时，任何现代系统都会发生CPU节流。因此，它会减慢时钟，并开始运行较慢（例如，2.3GHz机器切换到2.0GHz）。这就是您不能根据百分比设置CPU限制的原因。

所以，从你的图表中，我推测看到的是CPU时钟下降，自然百分比上升；正如预期的那样。没什么奇怪的。