CPU使用率低于阈值且存在多个节点-普罗米修斯

Question

我试图在prometheus中创建一个警报规则，以便当标签为agentpool=“worker”的所有节点在过去3分钟内的平均CPU使用率低于30%时，它会发出警报。

- alert: NodeCPUUtilizationLow
    expr: instance:node_cpu:rate:sum * 100 < 30
    labels:
        severity: none
    annotations:
        description: CPU utilization has been lower than 30% for last 3 minutes (current value is {{$value}})

{instance="10.240.0.187:9100"}  26.466666666668715

kube_node_labels{endpoint="kube-state-metrics",instance="10.240.0.187:8080",job="kube-prometheus-exporter-kube-state",label_agentpool="worker",label_beta_kubernetes_io_arch="amd64",label_beta_kubernetes_io_instance_type="Standard_B2s",label_beta_kubernetes_io_os="linux",label_failure_domain_beta_kubernetes_io_region="southcentralus",label_failure_domain_beta_kubernetes_io_zone="0",label_kubernetes_azure_com_cluster="dev-kube-cluster",label_kubernetes_io_hostname="k8s-worker-22588695-1",label_kubernetes_io_role="agent",namespace="monitoring",node="k8s-worker-22588695-1",pod="kube-prometheus-exporter-kube-state-854f846569-8lnk2",service="kube-prometheus-exporter-kube-state"}   1
kube_node_labels{endpoint="kube-state-metrics",instance="10.240.0.187:8080",job="kube-prometheus-exporter-kube-state",label_agentpool="worker",label_beta_kubernetes_io_arch="amd64",label_beta_kubernetes_io_instance_type="Standard_B2s",label_beta_kubernetes_io_os="linux",label_failure_domain_beta_kubernetes_io_region="southcentralus",label_failure_domain_beta_kubernetes_io_zone="1",label_kubernetes_azure_com_cluster="dev-kube-cluster",label_kubernetes_io_hostname="k8s-worker-22588695-0",label_kubernetes_io_role="agent",namespace="monitoring",node="k8s-worker-22588695-0",pod="kube-prometheus-exporter-kube-state-854f846569-8lnk2",service="kube-prometheus-exporter-kube-state"}   1

Answer 1

实例:node_cpu:rate:sum可能是https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/

因此，您可以向Prometheus添加一个新的记录规则来求和工作节点的CPU，或者直接使用相应的查询。