Google GKE工作负载突然显示为脱机状态，错误为：ErrImagePull

Question

我有一个GKE集群，直到最近一直运行良好。现在我看到一大堆库伯内特斯工作负载显示为离线，以下错误msg：

 Type     Reason          Age                    From                                                          Message
  ----     ------          ----                   ----                                                          -------
  Normal   Scheduled       6m23s                  default-scheduler                                               Warning  Failed          5m39s (x3 over 6m22s)  kubelet, gke-platsol-bots-staging-default-pool-f489f2f3-rjrq  Error: ErrImagePull
  Normal   BackOff         5m2s (x7 over 6m21s)   kubelet, gke-platsol-bots-staging-default-pool-f489f2f3-rjrq  Back-off pulling image "us.gcr.io/project/poc-app-bot@sha256:b99b5fb1b77407ade49d9bf42a94919e90422fee26c1a46ec6247370bd96c4d8"
  Normal   Pulling         4m49s (x4 over 6m22s)  kubelet, gke-platsol-bots-staging-default-pool-f489f2f3-rjrq  pulling image "us.gcr.io/project/poc-app-bot@sha256:b99b5fb1b77407ade49d9bf42a94919e90422fee26c1a46ec6247370bd96c4d8"
  Warning  Failed          81s (x22 over 6m21s)   kubelet, gke-platsol-bots-staging-default-pool-f489f2f3-rjrq  Error: ImagePullBackOff

不确定是什么变化导致了这个问题。

这是库贝特的输出

Name:               project-5dddbd66b5-vpw8q
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               gke-platsol-bots-staging-default-pool-f489f2f3-rjrq/10.x.x.x
Start Time:         Wed, 18 Sep 2019 16:48:23 +0100
Labels:             app=bot
                    pod-template-hash=5dddbd66b5
Annotations:        kubernetes.io/limit-ranger: LimitRanger plugin set: cpu request for container project
Status:             Pending
IP:                 10.20.1.9
Controlled By:      ReplicaSet/bot-5dddbd66b5
Containers:
  project:
    Container ID:
    Image:          us.gcr.io/project/project@sha256:b99b5fb1b77407ade49d9bf42a94919e90422fee26c1a46ec6247370bd96c4d8
    Image ID:
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:      100m
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  default-token-99cns:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-99cns
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason  Age                     From                                                          Message
  ----     ------  ----                    ----                                                          -------
  Warning  Failed  4m38s (x793 over 3h4m)  kubelet, gke-platsol-bots-staging-default-pool-f489f2f3-rjrq  Error: ImagePullBackOff

下面是我对部署的YAML定义。我还没有定义一个秘密，因为从Google容器注册表中提取图像不需要秘密，

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      annotations:
        deployment.kubernetes.io/revision: "3"
        kubectl.kubernetes.io/last-applied-configuration: |
<redacted annotations>
      creationTimestamp: 2019-06-06T08:37:01Z
      generation: 3
      labels:
        app: project
      name: bot
      namespace: default
      resourceVersion: "68945490"
      selfLink: /apis/apps/v1/namespaces/default/deployments/bot
      uid: 412ce711-8836-11e9-905f-42010a8e016c
     image: us.gcr.io/project/app-bot@sha256:b99b5fb1b77407ade49d9bf42a94919e90422fee26c1a46ec6247370bd96c4d8
        imagePullPolicy: IfNotPresent

好的，所以当我从GCR中提取图像时，我按照这个指南用一个“秘密”来修补服务帐户https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/

我SSH到一个节点上，可以成功地为一个应用程序拉取图像，

vinay@cloudshell:~ (project-id)$ docker pull us.gcr.io/project-id/project2-bot@sha256:9817462c743a93bb9206e4b8685
5322f731a768dca18e26b8bfc39b0cc886d31
sha256:9817462c743a93bb9206e4b86855322f731a768dca18e26b8bfc39b0cc886d31: Pulling from project-id/project2-bot
092586df9206: Pull complete
ef599477fae0: Pull complete
4530c6472b5d: Pull complete
d34d61487075: Pull complete
272f46008219: Pull complete
12ff6ccfe7a6: Pull complete
f26b99e1adb1: Pull complete
bb50901cd579: Pull complete
64a286652062: Pull complete
283785ced197: Pull complete
ed5a2062edd6: Pull complete
Digest: sha256:9817462c743a93bb9206e4b86855322f731a768dca18e26b8bfc39b0cc886d31
Status: Downloaded newer image for us.gcr.io/project-id/project2-bot@sha256:9817462c743a93bb9206e4b86855322f731a768dca18e26b8
bfc39b0cc886d31
us.gcr.io/project-id/project2-bot@sha256:9817462c743a93bb9206e4b86855322f731a768dca18e26b8bfc39b0cc886d31

但这个应用程序似乎抛出了一个错误，

vinay@cloudshell:~ (project-id)$ docker pull us.gcr.io/project-id/project1-plug@sha256:c53ac1c536a1187ce940f9221730cc0eae3103f4313033659e2162a70bc66c59
    sha256:c53ac1c536a1187ce940f9221730cc0eae3103f4313033659e2162a70bc66c59: Pulling from project-id/project1-plug
    a4d8138d0f6b: Pulling fs layer
    dbdc36973392: Pulling fs layer
    f59d6d019dd5: Pulling fs layer
    aaef3e026258: Waiting
    5e86b04a4500: Waiting
    1a6643a2873a: Waiting
    2ad1e30fc17c: Waiting
    ddb5baaf3393: Waiting
    0a7edc889b3c: Waiting
    31a1f16c256b: Waiting
    172a500f7b4d: Waiting
    error pulling image configuration: unknown blob

Answer 1

ErrImagePull很可能是最常见的，幸运的是，它易于调试和诊断。出现这种情况时，您将看到ErrImagePull作为状态消息，指示Kubernetes无法检索清单中指定的映像（该映像可能已从寄存器中删除）。

您可以使用kubectl descripe[pod]命令立即获得有关此错误发生原因的更详细信息。这并不完全是一种错误情况，因为从技术上讲，库伯内特斯正处于等待状态，希望图像能够被获得