Using the Three Kubernetes Probes Properly — The Exact Boundaries of liveness, readiness, and startup

Introduction — A Perfectly Fine Pod Restarts Every Few Minutes

The application works fine, but the RESTARTS column quietly climbs.

kubectl get pod -n payments

NAME                        READY   STATUS    RESTARTS       AGE
checkout-7d9f6c5b4d-2xk9p   1/1     Running   11 (4m2s ago)  3h18m
checkout-7d9f6c5b4d-lm4vz   1/1     Running   9 (12m ago)    3h18m

The logs contain nothing but records of normal request handling right up to the moment of termination. The answer is in the events.

kubectl describe pod checkout-7d9f6c5b4d-2xk9p -n payments | tail -5

Events:
  Type     Reason     Age                    From     Message
  ----     ------     ----                   ----     -------
  Warning  Unhealthy  4m12s (x33 over 3h17m) kubelet  Liveness probe failed: Get "http://10.42.3.17:8080/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
  Normal   Killing    4m2s (x11 over 3h17m)  kubelet  Container checkout failed liveness probe, will be restarted

Kubernetes is killing this container. More precisely, we are the ones who told it to.

The Three Probes Answer Different Questions

The syntax of the three probes is identical, which is what makes them confusing, but what they do on failure is completely different.

• startupProbe — "Has startup finished". It completely halts liveness and readiness until it succeeds. Once it succeeds, it never runs again for the lifetime of that container. If it burns through failureThreshold, it kills the container
• readinessProbe — "May this take traffic right now". On failure, the pod address is dropped from the service endpoint. It does not restart the container. When it succeeds, the address comes back in
• livenessProbe — "Does this process need to be restarted". On failure, the kubelet kills the container and brings it back up according to restartPolicy

Let us clear up the most common misunderstanding first. A readiness failure never triggers a restart. People often panic that a pod stays at 0/1 and never restarts, but that is the design working as intended. Restarting is the exclusive authority of liveness and startup.

You confirm a readiness failure at the endpoint.

kubectl get endpointslice -n payments -l kubernetes.io/service-name=checkout

NAME             ADDRESSTYPE   PORTS   ENDPOINTS               AGE
checkout-x7k2m   IPv4          8080    10.42.3.17,10.42.1.9    3h20m

Only two of the three pods are in there. The remaining one is the pod that failed readiness.

kubectl get pod -n payments -l app=checkout \
  -o custom-columns='POD:.metadata.name,IP:.status.podIP,READY:.status.containerStatuses[*].ready'

POD                         IP           READY
checkout-7d9f6c5b4d-2xk9p   10.42.3.17   true
checkout-7d9f6c5b4d-lm4vz   10.42.1.9    true
checkout-7d9f6c5b4d-q8w2n   10.42.2.31   false

Three Incidents That Come from Using Them Wrong

Incident 1 — The Self-Destruction a Hypersensitive liveness Creates

This is the most dangerous combination. When load piles up, the application responds more slowly. The probe slows down along with it. When it times out, Kubernetes kills the container. More load piles onto the remaining pods. Those pods time out too. It becomes a structure where the higher the load the more things die, and the system tears itself down.

In this scenario liveness is not a stability device but an amplifier. The diagnosis reveals itself immediately when you overlay the restart timestamps on the traffic graph. If restarts cluster only at peak hours, it is confirmed.

Incident 2 — Deploying Without readiness

Without readiness, the pod is treated as Ready the moment the container process comes up, and it enters the endpoint right away. Requests arrive even while the application is still building its connection pool and filling its cache.

kubectl get events -n payments --field-selector reason=Started --sort-by=.lastTimestamp | tail -2

If 5xx spikes for only a few seconds right after a deploy and then returns to normal, it is almost always this problem. Because it repeats on every rolling update, it hardens into a permanent error rate for teams that deploy several times a day.

Incident 3 — When liveness Massacres a Slow Startup

If you attach a liveness probe with initialDelaySeconds 30 to an application that takes 90 seconds to start, the probe begins failing at the 30 second mark, the container dies, comes back up, and dies again. It never reaches Ready.

kubectl describe pod ledger-0 -n payments | grep -E "Started|Killing"

  Normal   Started    2m11s (x5 over 7m32s)  kubelet  Started container ledger
  Normal   Killing    91s (x4 over 6m52s)    kubelet  Container ledger failed liveness probe, will be restarted

The gap between Started and Killing holds steady at about 40 seconds. That is an unmistakable signal that it is being killed over and over before startup completes.

A common wrong answer: raising initialDelaySeconds to 200. Two costs follow. First, a container that is genuinely dead also goes undetected for the first 200 seconds. Second, that value has to be sized for the slowest startup, so on days when the node is busy it is still not enough. The right answer is startupProbe.

Parameter Math — How Many Seconds Until It Actually Dies

Let us pin down the defaults and the meaning of the five parameters first.

• initialDelaySeconds (default 0) — the wait from container start to the first probe
• periodSeconds (default 10) — how often the probe runs
• timeoutSeconds (default 1) — how long a single probe waits for a response
• failureThreshold (default 3) — how many consecutive failures count as a failure
• successThreshold (default 1) — how many consecutive successes count as a success. Only 1 is allowed for liveness and startup

The timeoutSeconds default of 1 second is the most common seed of an incident. A GC pause, a momentary thread pool saturation, disk latency — any of them can push past 1 second. No matter how light the health endpoint is, if the whole process stalls, the response is late.

Now we calculate. Assume that with the settings below the container comes to a complete stop at second t.

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 0
  periodSeconds: 10
  timeoutSeconds: 3
  failureThreshold: 3

• In the worst case the first failing probe starts not right after t but on the next cycle, that is at most t plus 10 seconds
• Because three consecutive failures are required, that means three probes at 10 second intervals, and the last failure lands near t plus 30 seconds
• Each probe burns its 3 second timeout, so the actual verdict comes a few seconds later than that

In other words, the worst case detection delay is roughly periodSeconds times failureThreshold plus timeoutSeconds. The settings above come to about 33 seconds. You have to line this value up with your SLO. For a service that must recover within 30 seconds, you need to drop periodSeconds to 5.

The budget for startupProbe is set by multiplication.

startupProbe:
  httpGet:
    path: /healthz
    port: 8080
  periodSeconds: 5
  failureThreshold: 60
  timeoutSeconds: 3

5 seconds times 60 attempts allows a startup of up to 300 seconds. During those 300 seconds liveness does not run at all. Once startup finishes, startup is permanently disabled and liveness takes over on a short period. This is the only way to get a generous startup and agile detection at the same time.

Here is the finished form that uses all three probes together.

containers:
  - name: checkout
    image: registry.example.com/checkout:1.14.2
    ports:
      - name: http
        containerPort: 8080
    startupProbe:
      httpGet:
        path: /healthz
        port: http
      periodSeconds: 5
      failureThreshold: 60
      timeoutSeconds: 3
    readinessProbe:
      httpGet:
        path: /readyz
        port: http
      periodSeconds: 5
      timeoutSeconds: 3
      failureThreshold: 2
      successThreshold: 1
    livenessProbe:
      httpGet:
        path: /healthz
        port: http
      periodSeconds: 10
      timeoutSeconds: 5
      failureThreshold: 6

Three design intentions are baked in. readiness got a small period and threshold so it drops out fast and comes back fast. liveness is the opposite, set generously so it restarts only when the container has definitively been dead for 60 seconds or more. The paths differ too. liveness uses /healthz and readiness uses /readyz.

Putting Dependencies in liveness Turns a Failure into a Total Outage

This is the implementation that looks most natural when you first build a health check.

GET /healthz
  → SELECT 1 against the database
  → PING to Redis
  → HEAD request to the payment gateway
  → 200 if everything succeeds

It looks honest, but in production it is a disaster. Let us walk through, in order, what happens when the database wobbles for 3 minutes.

• Every pod fails liveness at the same time. They are all looking at the same dependency, so it is all of them without exception
• The kubelet kills every container
• The pods restart simultaneously, build new connection pools, and every cache is emptied
• The instant the database recovers, every pod dumps connections and queries onto it at once
• The database goes over again
• It repeats

A 3 minute database failure escalates into a 30 minute service outage. On top of that, the restarts wipe out the in-flight requests, the in-memory queues, and the warmed caches, which makes recovery even slower.

There is exactly one question liveness has to answer: does restarting this process make things better. When the database is down, restarting the process improves nothing. So the database must not go into liveness.

The only things that belong in liveness are the states a restart actually fixes. Roughly: a stalled event loop, a deadlock, and unrecoverable corruption of internal state.

GET /healthz
  → check only whether the process can accept a request and produce a response
  → always 200, except that if the event loop is blocked the response never goes out at all

This simple handler is the best option in most cases. And a conclusion one step beyond it is also possible. If there is no failure mode that a restart would fix, it is safer to have no liveness probe at all. Without a probe, the container restarts only when the process dies, and that is usually the right behavior.

How Far Can You Go with readiness

Because readiness does not trigger a restart, there is room to put dependency checks in it. The criterion is this. Is there even a single request this pod can serve without that dependency.

• If every request needs the database, putting it in readiness is reasonable. It is better not to accept requests that would fail anyway
• If only some of them need it, you should not put it in. You would also block the requests that could be served from a static response or a cache

And once you decide to put it in, there is one thing you absolutely must check. If every pod goes NotReady at the same time, the service ends up with zero endpoints and clients get their connections refused outright. That can be a worse experience than a 5xx, and because there is no health signal at all at the moment of recovery, diagnosis gets harder too.

kubectl get endpointslice -n payments -l kubernetes.io/service-name=checkout -o wide

NAME             ADDRESSTYPE   PORTS   ENDPOINTS   AGE
checkout-x7k2m   IPv4          8080    <unset>     3h44m

If ENDPOINTS is empty, this is the state you are in. When you put a dependency check in readiness, you also have to consider a policy that keeps at least one instance in, or a design that treats a dependency failure as partial degradation.

Choosing a Probe Type and the Cost of exec

There are four methods, and they differ in cost and reliability.

httpGet — the default choice. The kubelet sends the request directly to the pod IP. It treats 200 through 399 as success. There is one trap here. If an auth proxy sends a 302 redirect from the health path to a login page, the probe is judged a success. It succeeds even when the application is dead. The health path must be excluded from authentication and must return 200 directly.

tcpSocket — it only looks at whether the port is open. The kernel maintains the listener socket, so it succeeds even when every application thread has stalled. It is the least reliable, so use it only when there is no other option.

grpc — for a gRPC service, the kubelet calls the standard health protocol directly. There is no need to bake a health check binary into the image and call it through exec the way you used to.

readinessProbe:
  grpc:
    port: 9000
    service: readiness
  periodSeconds: 5

exec — the most expensive. Every time the probe runs, it creates a new process inside the container. The container runtime pays the cost of creating a task, setting up namespaces, and reaping the process every single time. Put an exec probe with a 10 second period on 500 pods and the cluster as a whole sees 50 process creations per second, continuously.

The cost shows up in two places. The kubelet and runtime CPU on the node climb, and because the probe process belongs to the cgroup of the container, it eats into the CPU and memory budget of the application. For an exec probe that invokes a shell script, the memory of the shell itself is included in that. In a container with a tight memory limit, the probe can even become the trigger for an OOM.

Counting what you actually have is usually a surprise.

kubectl get pods -A -o json | \
  jq -r '.items[].spec.containers[].livenessProbe | select(.!=null) | keys[]' | \
  grep -v Seconds | grep -v Threshold | sort | uniq -c | sort -rn

    412 httpGet
     87 exec
     23 tcpSocket

A good share of those 87 exec probes are scripts that call curl, and that is a cost that disappears outright if you switch to httpGet.

If you use a service mesh, there is one more thing. Because the kubelet sends the request directly to the pod IP, the probe bypasses the mTLS policy of the sidecar. If the mesh requires strict mutual TLS the probe fails, so most meshes offer an option to rewrite the probe path so the sidecar receives it instead. If every probe suddenly starts failing while the application is perfectly fine, suspect this.

The Termination Path — The Race Condition Between Endpoint Removal and SIGTERM

No matter how well you tune the probes, if errors spike on every rolling update, the cause is in the termination path.

When a pod is deleted, two things start simultaneously and with no coordination between them.

• The kubelet path — it runs the preStop hook, sends SIGTERM to the main container process, and sends SIGKILL once the grace period ends
• The endpoint path — the controller removes the pod from the EndpointSlice, and that change propagates to the kube-proxy on every node, to the ingress controllers, and to the service mesh sidecars

The problem is that the second one is slow. It takes hundreds of milliseconds to several seconds for the controller to notice, to update the EndpointSlice, and for every node to receive it and rewrite its rules. With a large cluster or an external load balancer in the mix, it takes longer.

Meanwhile the container has already received SIGTERM and closed its listener. Requests that are still routed to this pod meet a connection refusal. This is exactly why a small number of 502s and connection resets show up during a deploy.

The fix is to delay SIGTERM on purpose. The preStop hook runs before SIGTERM and SIGTERM is held back until it completes, so you put a delay here that waits for propagation.

spec:
  terminationGracePeriodSeconds: 60
  containers:
    - name: checkout
      lifecycle:
        preStop:
          exec:
            command: ['sh', '-c', 'sleep 15']

Recent versions let you use a dedicated handler that does not invoke a shell. It is especially useful in a distroless environment where the image has no shell.

lifecycle:
  preStop:
    sleep:
      seconds: 15

Here you absolutely have to calculate the grace period budget. terminationGracePeriodSeconds is the entire budget, including the preStop execution time.

• 15 seconds of preStop delay
• up to 30 seconds to drain in-flight requests
• 15 seconds of slack
• 60 seconds in total

If you skip this calculation, the grace period is exhausted the moment preStop ends and SIGKILL flies. That is the situation where exit code 137 gets stamped with Reason Error, and because it shares an exit code with the memory problem covered in the OOMKilled article, it is easy to misdiagnose.

The application side has to handle SIGTERM properly too. Refuse new connections but finish the in-flight requests, and let background workers wrap up their current job before exiting. An application with no SIGTERM handler dies instantly under the default behavior, so no matter how long you make preStop, in-flight requests get cut off.

kubectl exec -it checkout-7d9f6c5b4d-2xk9p -n payments -- \
  sh -c 'grep SigCgt /proc/1/status'

SigCgt: 0000000000014002

Whether bit 15 is set in this bitmask tells you if a SIGTERM handler is registered. For containers launched through a shell, you also have to check the classic problem where the shell becomes PID 1 and does not forward signals to its children.

A Standard Template and a Checklist

When you build a new service, use the below as a starting point and adjust only the values to fit the characteristics of the service.

spec:
  terminationGracePeriodSeconds: 60
  containers:
    - name: app
      lifecycle:
        preStop:
          sleep:
            seconds: 15
      startupProbe:
        httpGet: { path: /healthz, port: http }
        periodSeconds: 5
        failureThreshold: 60
        timeoutSeconds: 3
      readinessProbe:
        httpGet: { path: /readyz, port: http }
        periodSeconds: 5
        timeoutSeconds: 3
        failureThreshold: 2
      livenessProbe:
        httpGet: { path: /healthz, port: http }
        periodSeconds: 10
        timeoutSeconds: 5
        failureThreshold: 6

Check five things before you deploy.

• Does the liveness handler stay clear of calling external dependencies
• Has timeoutSeconds been left at 1
• Does the worst case startup time fit inside the startupProbe budget
• Is terminationGracePeriodSeconds larger than the sum of the preStop delay and the drain time
• Does the application receive SIGTERM and start draining

As a metric for preventing recurrence, tracking probe failures themselves is useful. They get caught at the stage before they lead to a restart.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: probe-alerts
  namespace: monitoring
spec:
  groups:
    - name: probes
      rules:
        - alert: LivenessProbeFailing
          expr: increase(prober_probe_total{probe_type="Liveness",result="failure"}[15m]) > 5
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "{{ $labels.namespace }}/{{ $labels.pod }} liveness failing without restart yet"
        - alert: ServiceHasNoEndpoints
          expr: kube_endpoint_address_available == 0
          for: 3m
          labels:
            severity: critical

Closing — A Probe Is Not a Safety Device but a Delegation of Authority

Configuring a probe means handing Kubernetes the authority to cut traffic and the authority to kill a process. We are the ones who decide on what grounds that authority gets exercised. If the grounds are weak, a system with nothing wrong with it collapses under its own defense mechanism.

Here is the one sentence to remember. liveness must answer only "does restarting make it better", and every condition whose answer is no has to be sent to readiness or taken out of the probes entirely.