필사 모드: Kubernetes ImagePullBackOff and ErrImagePull Fully Dissected — Ending It With One Cause String

Introduction — The Image Name Is Clearly Correct but the Pod Will Not Start

After a deploy, the pod gets stuck in this state.

kubectl get pod -n analytics

NAME                        READY   STATUS             RESTARTS   AGE
ingest-6b8c94f7d5-4tzqr     0/1     ImagePullBackOff   0          3m12s
ingest-6b8c94f7d5-9wdhm     0/1     ErrImagePull       0          3m12s

These are pods of the same Deployment, yet one is ImagePullBackOff and the other is ErrImagePull. You have re-read the image name several times and there is no typo. At this point most people open the registry UI and start verifying the tag by eye, but that is unnecessary. The kubelet already left the reason for the failure in an event, as a literal string.

ErrImagePull and ImagePullBackOff Are Two Stages of the Same Event

The relationship between the two states is simple.

• The kubelet attempts to download the image
• If it fails, the Reason in the container status becomes ErrImagePull, and a Failed event carrying the reason is recorded
• The kubelet retries a moment later. During that waiting window the Reason changes to ImagePullBackOff
• If the retry fails again, the wait doubles. It starts at 10 seconds and is capped at 5 minutes

So ErrImagePull is the moment of failure and ImagePullBackOff is the wait in between. That is also why pods of the same Deployment appear in different states. The only difference is where in each pod retry cycle the moment of your query happened to land.

A practically important conclusion follows from this. After fixing the registry side, deleting the pod is faster than waiting up to 5 minutes. A new pod has no backoff history and attempts its first pull immediately.

kubectl rollout restart deployment/ingest -n analytics

The Cause Is Written Verbatim in the Last Line of Events

Diagnosis starts with a single describe, and in practice it ends there.

kubectl describe pod ingest-6b8c94f7d5-4tzqr -n analytics | tail -12

Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  3m24s                 default-scheduler  Successfully assigned analytics/ingest-6b8c94f7d5-4tzqr to ip-10-0-2-77
  Normal   Pulling    2m1s (x4 over 3m23s)  kubelet            Pulling image "ghcr.io/example/ingest:2.7.0"
  Warning  Failed     2m1s (x4 over 3m22s)  kubelet            Failed to pull image "ghcr.io/example/ingest:2.7.0": rpc error: code = NotFound desc = failed to pull and unpack image "ghcr.io/example/ingest:2.7.0": failed to resolve reference "ghcr.io/example/ingest:2.7.0": ghcr.io/example/ingest:2.7.0: not found
  Warning  Failed     2m1s (x4 over 3m22s)  kubelet            Error: ErrImagePull
  Normal   BackOff    97s (x6 over 3m22s)   kubelet            Back-off pulling image "ghcr.io/example/ingest:2.7.0"
  Warning  Failed     97s (x6 over 3m22s)   kubelet            Error: ImagePullBackOff

The Message on the Warning Failed line is everything. Read that one string precisely and the cause is settled.

When sweeping several namespaces at once, filtering the events directly is faster.

kubectl get events -A --field-selector reason=Failed --sort-by=.lastTimestamp | tail -5

NAMESPACE   LAST SEEN   TYPE      REASON   OBJECT                          MESSAGE
analytics   41s         Warning   Failed   pod/ingest-6b8c94f7d5-4tzqr     Failed to pull image "ghcr.io/example/ingest:2.7.0": ... not found
payments    2m8s        Warning   Failed   pod/checkout-5d7f8b9c4-x2klm    Failed to pull image "registry.example.com/checkout:1.14.2": ... 401 Unauthorized
media       6m11s       Warning   Failed   pod/transcode-79c5d6f8b-vn4pq   Failed to pull image "redis:7.2": ... toomanyrequests: You have reached your pull rate limit.

A common wrong answer: re-checking the image name first without reading the Events. A name typo is only one of the eight branches, and the other seven stay invisible no matter how long you stare at the name.

Cause Branches and Diagnosis

A Tag That Does Not Exist

The simplest branch, and the one that shows up most often. Either CI applied the manifest before pushing the image, or the tag convention changed, or the tag was deleted by a cleanup policy. You can check it before ever entering the cluster.

crane ls ghcr.io/example/ingest | tail -5

2.6.4
2.6.5
2.7.0-rc1
2.7.1

crane manifest ghcr.io/example/ingest:2.7.0
Error: fetching manifest ghcr.io/example/ingest:2.7.0: GET https://ghcr.io/v2/example/ingest/manifests/2.7.0: MANIFEST_UNKNOWN

There is no 2.7.0, but there is 2.7.0-rc1 and 2.7.1. The release pipeline failed to strip the rc suffix.

Node Disk Exhaustion

A branch that is missed surprisingly often. When the node disk fills up the kubelet image GC runs, but if you then try to fetch an image larger than what it freed, the pull fails.

kubectl describe node ip-10-0-2-77 | grep -A8 "Conditions:"

Conditions:
  Type             Status  LastTransitionTime                Reason                       Message
  ----             ------  ------------------                ------                       -------
  MemoryPressure   False   Sat, 25 Jul 2026 22:10:44 +0900   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     True    Sun, 26 Jul 2026 08:52:03 +0900   KubeletHasDiskPressure       kubelet has disk pressure
  PIDPressure      False   Sat, 25 Jul 2026 22:10:44 +0900   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Sat, 25 Jul 2026 22:10:54 +0900   KubeletReady                 kubelet is posting ready status

A node whose DiskPressure is True also refuses to schedule new pods. The image GC thresholds are tuned with imageGCHighThresholdPercent and imageGCLowThresholdPercent in the kubelet configuration. The defaults are 85 and 80 respectively.

Private Registry Authentication — Confirming It Attached Is Harder Than Creating It

If a 401 came up, check the following in order.

First create the Secret. The trap is that the --docker-server value must match the registry host of the image reference exactly.

kubectl create secret docker-registry regcred \
  --docker-server=registry.example.com \
  --docker-username=deploy-bot \
  --docker-password="$(cat ~/.registry-token)" \
  --namespace=payments

Docker Hub is the exception. Even when the image reference is as short as nginx:1.25, the real host is normalized to docker.io, and the authentication server value must be https://index.docker.io/v1/. Put docker.io there and the Secret is created but the authentication never attaches.

Once you have created it, decode the contents and confirm them by eye.

kubectl get secret regcred -n payments \
  -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq .

{
  "auths": {
    "registry.example.com": {
      "username": "deploy-bot",
      "password": "glpat-xxxxxxxxxxxx",
      "auth": "ZGVwbG95LWJvdDpnbHBhdC14eHh4eHh4eHh4eHg="
    }
  }
}

Here comes the real trap. Even with the Secret created, nothing happens unless the pod is instructed to use it. There is the approach of putting it directly in the pod spec and the approach of attaching it to the ServiceAccount, and since Helm charts often do not expose an imagePullSecrets value, the latter is the useful one in practice.

kubectl patch serviceaccount default -n payments \
  -p '{"imagePullSecrets": [{"name": "regcred"}]}'

serviceaccount/default patched

This is where most people get stuck. The imagePullSecrets on a ServiceAccount is copied into the pod spec at the moment the pod is created. It is not applied retroactively to pods that were already running. After the patch you have to recreate the pods.

kubectl rollout restart deployment/checkout -n payments

Then confirm that it actually made it into the pod spec. This one line resolves most of the "I configured it, so why does it not work" cases.

kubectl get pod -n payments -l app=checkout \
  -o jsonpath='{.items[0].spec.imagePullSecrets}'

[{"name":"regcred"}]

Organized as a checklist, there are four items.

• Is the Secret in the same namespace as the pod. Secrets do not cross namespaces
• Does the --docker-server value match the host of the image reference as a string
• Is imagePullSecrets actually attached to the pod spec or to the ServiceAccount the pod uses
• Is the pod actually using that ServiceAccount. The default is default, but the chart may have created a dedicated ServiceAccount

kubectl get pod -n payments -l app=checkout \
  -o jsonpath='{.items[0].spec.serviceAccountName}'

checkout-sa

If you patched only default, this is where it goes wrong.

Finally, you have to remember token expiry on cloud registries. The AWS ECR authentication token expires after 12 hours, so a static Secret created with kubectl create secret will inevitably produce a 401 at some point. Node IAM role based authentication or a refresh controller is the right answer, and a static Secret should be used only for temporary diagnosis.

A common wrong answer: SSH into the node and run docker login. Most clusters use containerd as the runtime, so the Docker credentials are never referenced at all, and even when it does work it vanishes the moment the node is replaced.

Two Things That Work Locally but Fail in the Cluster

Architecture Mismatch

Run docker build on an Apple Silicon Mac and the default artifact is linux/arm64. Push that onto a cluster made of amd64 nodes and two different symptoms appear. Telling the two apart matters.

If the image is a manifest list but has no entry matching the node platform, it fails at the pull stage.

kubectl describe pod ingest-6b8c94f7d5-4tzqr -n analytics | grep "Failed to pull"

  Warning  Failed  8s  kubelet  Failed to pull image "ghcr.io/example/ingest:2.7.1": no match for platform in manifest: not found

By contrast, if a single-architecture image slips past the platform check, the pull succeeds and it dies at the execution stage. In that case it shows up as CrashLoopBackOff rather than ImagePullBackOff.

kubectl logs ingest-6b8c94f7d5-4tzqr -n analytics --previous

exec /usr/local/bin/ingest: exec format error

exec format error is effectively the fingerprint of an architecture mismatch. Digging through application code after seeing this message is a waste of time.

Verification goes in both directions, the image side and the node side.

docker buildx imagetools inspect ghcr.io/example/ingest:2.7.1

Name:      ghcr.io/example/ingest:2.7.1
MediaType: application/vnd.oci.image.index.v1+json
Digest:    sha256:2b1e7f43c9d8a0b6e2f14c7d9a3b58e0c1f26d4a8b7e930f5c2a1d6b4e83f97c

Manifests:
  Name:      ghcr.io/example/ingest:2.7.1@sha256:8f3c1d...
  MediaType: application/vnd.oci.image.manifest.v1+json
  Platform:  linux/arm64

kubectl get nodes -o custom-columns=NAME:.metadata.name,ARCH:.status.nodeInfo.architecture

NAME              ARCH
ip-10-0-1-14      amd64
ip-10-0-2-77      amd64
ip-10-0-3-91      amd64

The image is arm64 only and the nodes are all amd64. The fix is a multi-architecture build.

docker buildx create --use --name multiarch
docker buildx build \
  --platform linux/amd64,linux/arm64 \
  -t ghcr.io/example/ingest:2.7.1 \
  --push .

In a mixed-architecture cluster, another option is to constrain the pod so that it only goes to matching nodes.

spec:
  nodeSelector:
    kubernetes.io/arch: amd64

Docker Hub Rate Limit

If you use a public image and it fails only during certain time windows, it is the anonymous pull limit. The limit is counted per IP, so in a cluster sharing a single NAT gateway it drains faster as the node count grows.

kubectl describe pod transcode-79c5d6f8b-vn4pq -n media | grep "Failed to pull"

  Warning  Failed  15s  kubelet  Failed to pull image "redis:7.2": failed to pull and unpack image "docker.io/library/redis:7.2": failed to resolve reference "docker.io/library/redis:7.2": unexpected status from HEAD request to https://registry-1.docker.io/v2/library/redis/manifests/7.2: 429 Too Many Requests

You can query the currently remaining limit directly.

TOKEN=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq -r .token)
curl -s --head -H "Authorization: Bearer $TOKEN" \
  https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest | grep -i ratelimit

ratelimit-limit: 100;w=21600
ratelimit-remaining: 7;w=21600

100 pulls per 6-hour window, with 7 remaining. There are three remedies, and you review them in order. Attaching imagePullSecrets so that pulls run under an authenticated account is the fastest, standing up an in-house registry mirror is the most fundamental, and replicating frequently used base images into the in-house registry is the safest.

The containerd mirror configuration is set on the node like this.

# /etc/containerd/certs.d/docker.io/hosts.toml
server = "https://registry-1.docker.io"

[host."https://registry-mirror.example.com/v2"]
  capabilities = ["pull", "resolve"]
  skip_verify = false

Preventing Recurrence — Sorting Out imagePullPolicy and Digest Pinning

The Trap Created by the imagePullPolicy Default

If you do not state it explicitly, Kubernetes decides by looking at the tag.

• If the tag is latest or there is no tag at all, Always
• Every other tag gets IfNotPresent
• Specifying by digest gives IfNotPresent

The second rule is what causes incidents. In a pipeline that overwrites a mutable tag, a node that has already cached that tag will not fetch the new image. As a result, pods of the same Deployment end up running different code, and it becomes a bug that does not reproduce. Comparing the actual image IDs per node exposes it.

kubectl get pods -n analytics -l app=ingest \
  -o custom-columns=POD:.metadata.name,NODE:.spec.nodeName,IMAGEID:.status.containerStatuses[0].imageID

POD                       NODE           IMAGEID
ingest-6b8c94f7d5-4tzqr   ip-10-0-1-14   ghcr.io/example/ingest@sha256:8f3c1d...
ingest-6b8c94f7d5-9wdhm   ip-10-0-2-77   ghcr.io/example/ingest@sha256:2b1e7f...

Same tag, different digests.

A common wrong answer: resolving this problem by switching imagePullPolicy to Always. The symptom disappears, but the node queries the registry for the manifest every time it starts a pod, which leaves you fully exposed to rate limits and registry outages. The fundamental fix is to make the tag immutable and pin by digest.

spec:
  containers:
    - name: ingest
      image: ghcr.io/example/ingest@sha256:8f3c1d5b2e7a94c0f13d68b5a2e9c47f0d81b3a65e2c9f7048d1b6a35c8e29f4
      imagePullPolicy: IfNotPresent

With digest pinning, every node runs identical bytes even if the tag is overwritten. Rollbacks become exact too.

Verify That the Image Exists Before Deploying

Add a step to CI that checks, before applying the manifest, that the image really exists and that authentication passes, and half of your ImagePullBackOff cases disappear.

skopeo inspect \
  --creds "deploy-bot:${REGISTRY_TOKEN}" \
  docker://registry.example.com/checkout:1.14.2 \
  --format '{{.Digest}} {{.Architecture}}'

sha256:9c1f0f4d2a8b73e5c16f9d20a4b8e7c35f1d69a20c8b4e73f5a1d92c6b8e40f7 amd64

Catch Pull Failures With Alerts

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: image-pull-alerts
  namespace: monitoring
spec:
  groups:
    - name: image-pull
      rules:
        - alert: ImagePullFailing
          expr: kube_pod_container_status_waiting_reason{reason=~"ImagePullBackOff|ErrImagePull"} == 1
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "{{ $labels.namespace }}/{{ $labels.pod }} cannot pull its image"

Since the pod never comes up at all, application metrics will never catch this. The rule absolutely has to be placed on the kube-state-metrics side.

Conclusion — Read the Cause String and Guessing Becomes Unnecessary

ImagePullBackOff is what you see most often in the status column, but it is also the problem you can finish fastest. The Warning Failed line of describe spells out whether it is not found, 401, toomanyrequests, no match for platform, or no space left on device, and these five strings cover most of the cause branches.

The one sentence to remember is this. Diagnosing ImagePullBackOff is reading comprehension rather than inference, and the sentence you need to read is already in the events.