如何在普罗米修斯中从Kube状态度量中提取度量值时获得一个豆荚的标签

Question

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    APP: AppABC
    TEAM: TeamABC
...

scrape_configs:
  - job_name: 'pod monitoring'
    honor_labels: true
    kubernetes_sd_configs:
    - role: pod
    relabel_configs:
    - action: labelmap
      regex: __meta_kubernetes_pod_label_(.+)
...

kube_pod_container_status_restarts_total{
    app="kube-state-metrics",
    container="appabccontainer",
    job="pod monitoring",
    namespace="test-namespace",
    pod="appabc-766cbcb68d-29smr"
}

我有没有办法从kube-state-metrics中刮取这些指标，但在不覆盖它们的情况下将应用程序和团队标签放在一起？

 

编辑-我想通了

expr: kube_pod_status_phase{phase="Failed"} > 0

expr: kube_pod_status_phase{phase="Failed"} * on (pod,namespace) group_right kube_pod_labels > 0

- name: Pod_Failed
  rules:
  - alert: pod_failed
    expr: kube_pod_status_phase{phase="Failed"} * on (pod,namespace) group_right kube_pod_labels > 0
    labels:
      appname: '{{ $labels.label_APP }}' # This is what I wanted to capture
      teamname: '{{ $labels.label_TEAM }}' # This is what I wanted to capture
    annotations:
      summary: 'Pod: {{ $labels.pod }} is down'
      description: 'Pod: {{ $labels.pod }} is down in {{ $labels.namespace }} namespace.'

Answer 1

解决方案：使用PromQL，您可以进行分组。所以在我的普罗米修斯规则中，我改变了这个：

expr: kube_pod_status_phase{phase="Failed"} > 0

对此：

expr: kube_pod_status_phase{phase="Failed"} * on (pod,namespace) group_right kube_pod_labels > 0

因此，我的新警报规则如下所示：

- name: Pod_Failed
  rules:
  - alert: pod_failed
    expr: kube_pod_status_phase{phase="Failed"} * on (pod,namespace) group_right kube_pod_labels > 0
    labels:
      appname: '{{ $labels.label_APP }}' # This is what I wanted to capture
      teamname: '{{ $labels.label_TEAM }}' # This is what I wanted to capture
    annotations:
      summary: 'Pod: {{ $labels.pod }} is down'
      description: 'Pod: {{ $labels.pod }} is down in {{ $labels.namespace }} namespace.'