如何使用Istio的Prometheus配置kubernetes HPA？

Question

我们有一个Istio集群，我们正在尝试为Kubernetes配置水平pod自动Scale。我们希望使用请求计数作为HPA的自定义度量。我们如何利用伊斯蒂奥的普罗米修斯来达到同样的目的？

Answer 1

这个问题比我预想的要复杂得多，但我终于找到了答案。

>

首先，您需要配置应用程序以提供自定义度量。它位于开发应用程序端。这里有一个例子，如何用Go语言制作：用普罗米修斯观察度量

其次，您需要定义并部署应用程序的部署（或者Pod，或者您想要的任何东西）到Kubernetes，例如：

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: podinfo
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: podinfo
      annotations:
        prometheus.io/scrape: 'true'
    spec:
      containers:
      - name: podinfod
        image: stefanprodan/podinfo:0.0.1
        imagePullPolicy: Always
        command:
          - ./podinfo
          - -port=9898
          - -logtostderr=true
          - -v=2
        volumeMounts:
          - name: metadata
            mountPath: /etc/podinfod/metadata
            readOnly: true
        ports:
        - containerPort: 9898
          protocol: TCP
        readinessProbe:
          httpGet:
            path: /readyz
            port: 9898
          initialDelaySeconds: 1
          periodSeconds: 2
          failureThreshold: 1
        livenessProbe:
          httpGet:
            path: /healthz
            port: 9898
          initialDelaySeconds: 1
          periodSeconds: 3
          failureThreshold: 2
        resources:
          requests:
            memory: "32Mi"
            cpu: "1m"
          limits:
            memory: "256Mi"
            cpu: "100m"
      volumes:
        - name: metadata
          downwardAPI:
            items:
              - path: "labels"
                fieldRef:
                  fieldPath: metadata.labels
              - path: "annotations"
                fieldRef:
                  fieldPath: metadata.annotations
---
apiVersion: v1
kind: Service
metadata:
  name: podinfo
  labels:
    app: podinfo
spec:
  type: NodePort
  ports:
    - port: 9898
      targetPort: 9898
      nodePort: 31198
      protocol: TCP
  selector:
    app: podinfo

null

kubectl edit configmap -n istio-system prometheus

# pod's declared ports (default is a port-free target if none are declared).
- job_name: 'kubernetes-pods'
  # if you want to use metrics on jobs, set the below field to
  # true to prevent Prometheus from setting the `job` label
  # automatically.
  honor_labels: false
  kubernetes_sd_configs:
  - role: pod
  # skip verification so you can do HTTPS to pods
  tls_config:
    insecure_skip_verify: true
  # make sure your labels are in order
  relabel_configs:
  # these labels tell Prometheus to automatically attach source
  # pod and namespace information to each collected sample, so
  # that they'll be exposed in the custom metrics API automatically.
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: namespace
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: pod
  # these labels tell Prometheus to look for
  # prometheus.io/{scrape,path,port} annotations to configure
  # how to scrape
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
    target_label: __address__
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]
    action: replace
    target_label: __scheme__

之后，自定义度量出现在普罗米修斯中。但是，在更改普罗米修斯配置时要小心，因为Istio所需的一些度量可能会消失，请仔细检查所有内容。

现在是安装普罗米修斯自定义度量适配器的时候了。

• 下载此存储库
• 在 /deploy/manifests/custom-metrics-apiserver-deployment.yaml 文件中更改Prometheus服务器的地址。示例，--prometheus-url=http://prometheus.istio-system:9090/
• 运行commandkubectl apply-f /deploy/manifests 一段时间后，custom.metrics.k8s.io/v1beta1应该出现在命令'kubectl api-vesions'的输出中。

{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/http_requests"
  },
  "items": [
    {
      "describedObject": {
        "kind": "Pod",
        "namespace": "default",
        "name": "podinfo-6b86c8ccc9-kv5g9",
        "apiVersion": "/__internal"
          },
          "metricName": "http_requests",
          "timestamp": "2018-01-10T16:49:07Z",
          "value": "901m"    },
        {
      "describedObject": {
        "kind": "Pod",
        "namespace": "default",
        "name": "podinfo-6b86c8ccc9-nm7bl",
        "apiVersion": "/__internal"
      },
      "metricName": "http_requests",
      "timestamp": "2018-01-10T16:49:07Z",
      "value": "898m"
    }
  ]
}

如果是，您可以进入下一步。如果没有，请查看CustomMetricsKubectl get中pod可用的API--raw“/APIs/custom.metrics.k8s.io/v1beta1”jq。grep“pods/”和对于http_requestsKubectl get--raw“/apis/custom.metrics.k8s.io/v1beta1”jq。grep“http”。MetricNames是根据Prometheus从豆荚中收集的度量生成的，如果它们是空的，您需要朝那个方向看。

最后一步是配置HPA并对其进行测试。因此，在我的例子中，我为podinfo应用程序创建了HPA，前面定义了：

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: podinfo
spec:
  scaleTargetRef:
    apiVersion: extensions/v1beta1
    kind: Deployment
    name: podinfo
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Pods
    pods:
      metricName: http_requests
      targetAverageValue: 10

并使用简单的Go应用程序测试负载：

#install hey
go get -u github.com/rakyll/hey
#do 10K requests rate limited at 25 QPS
hey -n 10000 -q 5 -c 5 http://<K8S-IP>:31198/healthz

• 使用Prometheus观察度量-向应用程序添加度量的示例
• k8s-prom-hpa-为Prometheus创建自定义度量的示例（与上面的文章相同）
• Prometheus的Kubernetes自定义度量适配器
• 设置自定义度量适配器和示例应用程序