필사 모드: Diagnosing Kubernetes CrashLoopBackOff by Cause — What to Look At When the Logs Are Empty

Introduction — the RESTARTS number keeps climbing and the logs are empty

Right after a deploy, the Pod list looks like this.

kubectl get pod -n payments

NAME                        READY   STATUS             RESTARTS        AGE
checkout-7d9f6c5b4d-2xk9p   0/1     CrashLoopBackOff   6 (2m41s ago)   11m
checkout-7d9f6c5b4d-lm4vz   0/1     CrashLoopBackOff   6 (2m38s ago)   11m
checkout-7d9f6c5b4d-q8w2n   0/1     CrashLoopBackOff   6 (2m44s ago)   11m

Try to read the logs and nothing comes back.

kubectl logs checkout-7d9f6c5b4d-2xk9p -n payments

Error from server (BadRequest): container "checkout" in pod "checkout-7d9f6c5b4d-2xk9p" is waiting to start: CrashLoopBackOff

At this point many people conclude the app does not log anything and start digging through the logging configuration. That is the wrong direction. The logs are there. The container you are trying to query is simply the next-generation container that has not started yet.

CrashLoopBackOff is not a cause, it is a restart delay

Read the name carefully. CrashLoopBackOff tells you nothing about why the container died. It only means the kubelet has declared that this container keeps dying, so it will hold off on restarting it for a while.

The kubelet restart delay follows a fixed rule.

• Wait 10 seconds after the first failure
• The wait doubles with every failure — 10s, 20s, 40s, 80s, 160s
• The ceiling is 300 seconds, that is 5 minutes
• If the container runs healthily for long enough (10 minutes by default), the delay timer resets and starts over at 10 seconds

Two practical conclusions follow from that rule.

First, a Pod that has been left alone for a while has a restart interval stretched out to 5 minutes. There is no need to push a fixed image and then sit there wondering why it still has not come up. Delete the Pod and the new one starts immediately with no back-off history.

kubectl delete pod checkout-7d9f6c5b4d-2xk9p -n payments

Second, a Pod whose RESTARTS count creeps up slowly may never show up as CrashLoopBackOff. An app that survives more than 10 minutes before dying resets the back-off every time, so the status column reads Running while RESTARTS quietly increases. That pattern is actually more dangerous. Sweep for it periodically with the command below.

kubectl get pods -A --sort-by='.status.containerStatuses[0].restartCount' | tail -20

NAMESPACE   NAME                          READY   STATUS    RESTARTS         AGE
search      indexer-6c4d9f7b8-h2klp       1/1     Running   17 (43m ago)     6d
payments    ledger-5f8b7c6d9-nm3xt        1/1     Running   23 (12m ago)     9d

Where diagnosis starts — Last State, Exit Code, and the previous logs

The order is always the same. Confirm the reason for death with describe, then read the logs of the dead container.

kubectl describe pod checkout-7d9f6c5b4d-2xk9p -n payments

Containers:
  checkout:
    Container ID:   containerd://3f2a91c4e88b7d5641a0c2f9e37b1d80
    Image:          registry.example.com/checkout:1.14.2
    Image ID:       registry.example.com/checkout@sha256:9c1f...
    Port:           8080/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Sun, 26 Jul 2026 09:41:12 +0900
      Finished:     Sun, 26 Jul 2026 09:41:13 +0900
    Ready:          False
    Restart Count:  6
Events:
  Type     Reason     Age                    From     Message
  ----     ------     ----                   ----     -------
  Normal   Pulled     3m12s (x5 over 11m)    kubelet  Container image "registry.example.com/checkout:1.14.2" already present on machine
  Normal   Created    3m12s (x5 over 11m)    kubelet  Created container checkout
  Normal   Started    3m11s (x5 over 11m)    kubelet  Started container checkout
  Warning  BackOff    2m41s (x24 over 10m)   kubelet  Back-off restarting failed container checkout

There are three values to read.

• The Reason under Last State — is it Error, OOMKilled, or Completed
• Exit Code — cross-reference it against the exit code dictionary below
• The gap between Started and Finished — a 1-second gap means it died during startup, 30 seconds or more means it came up and died later

In the example above, Started and Finished are 1 second apart. The process died the moment it came up, so suspect the startup-time configuration rather than the runtime logic.

Now read the logs of the dead container. The key is the --previous flag. Because the kubelet creates a new container each time it restarts, kubectl logs without the flag points at a container that has not started yet or was just born. The logs of the dead container live in the previous generation.

kubectl logs checkout-7d9f6c5b4d-2xk9p -n payments --previous

2026-07-26T00:41:12.881Z INFO  starting checkout 1.14.2
2026-07-26T00:41:13.104Z ERROR config: required key PAYMENT_SIGNING_KEY is not set
2026-07-26T00:41:13.105Z FATAL exiting with status 1

A common wrong answer: concluding that the application does not log because the logs are empty. In the overwhelming majority of cases you simply did not type --previous. If --previous is empty too, then and only then does it mean the process died without writing a single line, and in that case the cause is almost always the image entrypoint or a volume mount.

For a Pod with multiple containers, first pin down which container is dying.

kubectl get pod checkout-7d9f6c5b4d-2xk9p -n payments \
  -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.restartCount}{"\t"}{.lastState.terminated.reason}{"\t"}{.lastState.terminated.exitCode}{"\n"}{end}'

checkout    6    Error    1
istio-proxy 0    <none>   <none>

The exit code dictionary — 0, 1, 127, 137, 139, 143

Exit codes cut the diagnosis in half. Any value above 128 is 128 plus a signal number, so you can work backward to the signal that killed it.

• 0 — clean exit. The process finished its work and ended. A Deployment restartPolicy is always Always, so even a clean exit gets restarted by the kubelet and eventually becomes CrashLoopBackOff. Reason shows Completed, not Error. The cause is usually that the process was not run in the foreground. Running nginx in daemon mode, or a shell script that throws its last command into the background and exits, are the classic cases.
• 1 — a generic application error. Exceptions the framework did not catch, configuration validation failures, and missing required environment variables all land here. The cause is definitely in the logs.
• 2 — shell usage error. The shell returns this when it is passed bad arguments.
• 126 — the file exists but is not executable. The entrypoint script never got its execute bit.
• 127 — command not found. Either the command has a typo, or you invoked a shell that does not exist in a distroless or scratch image.
• 137 — 128 plus 9, that is SIGKILL. The kernel OOM killer or the kubelet force-killed it. If Reason is OOMKilled the memory limit was exceeded; if Reason is Error, it failed to exit within the grace period after SIGTERM and was force-killed.
• 139 — 128 plus 11, that is SIGSEGV. A segmentation fault in native code. It shows up with native extension modules, JNI, CGO, or an image built for the wrong architecture.
• 143 — 128 plus 15, that is SIGTERM. Someone asked for a graceful shutdown and the process complied. That is normal during a rolling update or a node drain, but if 143 repeats for no reason, a liveness probe is very likely killing the container.

A common wrong answer: raising the memory limit the moment you see 137. All 137 tells you is SIGKILL. Whether memory was the cause has to be judged by whether the Reason field says OOMKilled. Giving more memory to a container that was force-killed for refusing to exit after a liveness failure changes nothing. If it is confirmed as a memory issue, it is faster to move on to the OOMKilled 137 diagnosis article.

Cause branches and diagnostic commands

There are roughly seven paths that converge on CrashLoopBackOff. Rule them out one at a time.

Missing config and secrets

This is the most common one. Start by checking whether the key the Pod spec references actually exists.

kubectl get secret payment-keys -n payments -o jsonpath='{.data}' | tr ',' '\n'

{"PAYMENT_API_URL":"aHR0cHM6...","PAYMENT_WEBHOOK_SECRET":"czNjcjN0"}

The spec requires PAYMENT_SIGNING_KEY but the secret does not have that key. A typo in the key name is the cause.

There is an important design point here. If you reference it under env with secretKeyRef and set optional: true, the Pod comes up even when the key is missing and the application dies at runtime. Conversely, if you do not specify optional, the kubelet does not even start the container and stops at CreateContainerConfigError. The latter is far easier to diagnose.

env:
  - name: PAYMENT_SIGNING_KEY
    valueFrom:
      secretKeyRef:
        name: payment-keys
        key: PAYMENT_SIGNING_KEY
        # omitting optional defaults to false — if the key is missing, the container is not started

In that case the Pod status shows CreateContainerConfigError rather than CrashLoopBackOff, and describe spells out the exact reason.

kubectl describe pod checkout-7d9f6c5b4d-2xk9p -n payments | grep -A3 "Warning  Failed"

  Warning  Failed     9s (x3 over 25s)  kubelet  Error: couldn't find key PAYMENT_SIGNING_KEY in Secret payments/payment-keys

When a liveness probe is doing the killing

If Unhealthy and Killing appear together in Events, the probe is the culprit.

kubectl get events -n payments --field-selector involvedObject.name=checkout-7d9f6c5b4d-2xk9p --sort-by=.lastTimestamp

LAST SEEN   TYPE      REASON      OBJECT                          MESSAGE
4m12s       Warning   Unhealthy   pod/checkout-7d9f6c5b4d-2xk9p   Liveness probe failed: Get "http://10.42.3.17:8080/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
4m12s       Normal    Killing     pod/checkout-7d9f6c5b4d-2xk9p   Container checkout failed liveness probe, will be restarted

Attach a liveness probe with initialDelaySeconds 10 to an app that takes 40 seconds to start and it will never come up. Raising initialDelaySeconds here is a stopgap; the right answer is a startupProbe. The detailed arithmetic is laid out in the three-probe design article.

Wrong command and entrypoint

The Message spells out, word for word, that the executable could not be found.

kubectl describe pod migrate-runner-0 -n payments | grep -A2 "Last State"

    Last State:     Terminated
      Reason:       StartError
      Message:      failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/app/entrypoint.sh": permission denied

A permission problem gives 126; a missing file gives 127. Using command: ["sh", "-c", ...] in a distroless image yields 127 because there is no shell.

Volume mount failures and init containers

If the container never even started, look at Events rather than the logs.

kubectl describe pod ledger-0 -n payments | tail -8

Events:
  Type     Reason       Age                 From               Message
  ----     ------       ----                ----               -------
  Warning  FailedMount  2m15s (x9 over 12m) kubelet            MountVolume.SetUp failed for volume "ledger-data" : rpc error: code = Internal desc = volume attachment is being deleted
  Warning  FailedMount  38s                 kubelet            Unable to attach or mount volumes: unmounted volumes=[ledger-data], unattached volumes=[ledger-data kube-api-access-x9k2m]: timed out waiting for the condition

When an init container fails, the STATUS column gets an Init prefix and the main container never runs at all. You have to name the container explicitly when reading logs.

kubectl get pod ledger-0 -n payments

NAME       READY   STATUS                  RESTARTS      AGE
ledger-0   0/1     Init:CrashLoopBackOff   4 (48s ago)   3m

kubectl logs ledger-0 -n payments -c wait-for-db --previous

waiting for postgres.payments.svc.cluster.local:5432 ...
timeout after 30s

Getting inside the container while keeping it alive

When the logs alone do not crack it, you have to look inside the dying container yourself. There are two ways, and they serve different purposes.

First, override the command so the container sleeps instead of running the process. You can inspect the filesystem, environment variables and DNS under exactly the same conditions as the original.

apiVersion: v1
kind: Pod
metadata:
  name: checkout-shell
  namespace: payments
spec:
  restartPolicy: Never
  containers:
    - name: checkout
      image: registry.example.com/checkout:1.14.2
      command: ['sh', '-c', 'sleep infinity']
      envFrom:
        - secretRef:
            name: payment-keys
      volumeMounts:
        - name: config
          mountPath: /etc/checkout
  volumes:
    - name: config
      configMap:
        name: checkout-config

kubectl apply -f checkout-shell.yaml
kubectl exec -it checkout-shell -n payments -- sh

/ # env | grep PAYMENT
PAYMENT_API_URL=https://api.example.com
PAYMENT_WEBHOOK_SECRET=s3cr3t
/ # /app/checkout
ERROR config: required key PAYMENT_SIGNING_KEY is not set

To get the same effect without touching the existing Deployment, make a copy.

kubectl debug checkout-7d9f6c5b4d-2xk9p -n payments \
  --copy-to=checkout-debug \
  --container=checkout \
  -- sleep infinity

kubectl exec -it checkout-debug -n payments -- sh

Second, ephemeral containers. They are especially useful in distroless environments where the original image has no shell. Because they share a process namespace with the target container, you can observe even a dying process.

kubectl debug -it checkout-7d9f6c5b4d-2xk9p -n payments \
  --image=busybox:1.36 \
  --target=checkout \
  -- sh

Defaulting debug container name to debugger-7v2mp.
/ # ls /proc
1  14  self ...
/ # cat /proc/1/cmdline
/app/checkout
/ # wget -qO- http://localhost:8080/healthz
wget: can't connect to remote host: Connection refused

An ephemeral container does not modify the Pod spec, so it does not trigger a restart. Volumes are not shared by default, though, so if you need to see a mounted file, reach the filesystem of the container named by --target through the /proc/1/root path.

/ # ls /proc/1/root/etc/checkout
application.yaml  logging.yaml

Preventing recurrence — make crashes surface in the deploy pipeline

To avoid repeating the same incident, install guardrails in three places.

First, make the application validate all of its configuration at startup and die with a clear message. Code that only reads a required key on the first runtime request surfaces as a 500 error instead of CrashLoopBackOff, and gets found far later.

Second, make the deploy command wait for failure before it returns. kubectl apply submits the resource and returns success immediately, which creates the state where CI is green while production is dead.

kubectl apply -f deploy/checkout.yaml
kubectl rollout status deployment/checkout -n payments --timeout=180s

Waiting for deployment "checkout" rollout to finish: 0 of 3 updated replicas are available...
error: deployment "checkout" exceeded its progress deadline

Specify progressDeadlineSeconds on the Deployment and this verdict is attached automatically.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
  namespace: payments
spec:
  replicas: 3
  progressDeadlineSeconds: 180
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1

maxUnavailable: 0 keeps the existing Pods alive until the new ones are Ready. Deploy a crashing new version and the service stays up.

Third, catch rising restart counts with an alert. CrashLoopBackOff is conspicuous, but the quiet restart every 10 minutes mentioned earlier will sit ignored for weeks unless you are watching a dashboard.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: pod-restart-alerts
  namespace: monitoring
spec:
  groups:
    - name: pod-health
      rules:
        - alert: PodRestartingRepeatedly
          expr: increase(kube_pod_container_status_restarts_total[1h]) > 3
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "{{ $labels.namespace }}/{{ $labels.pod }} restarted more than 3 times in an hour"
        - alert: PodInCrashLoop
          expr: kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"} == 1
          for: 5m
          labels:
            severity: critical

Closing — when the logs are empty, type previous first

Diagnosing CrashLoopBackOff is not talent, it is order. Check the Reason and Exit Code under Last State with describe, hear the dead container out with kubectl logs --previous, and if that still gives you nothing, override the command to keep the container alive and go inside. Those three steps end most incidents within 10 minutes.

The one sentence to remember is this. CrashLoopBackOff is not a diagnosis, it is the name of the waiting room, and the real diagnosis is always written in the Exit Code and the logs of the previous container.