Kubernetes OOMKilled(137) Memory Troubleshooting — What to Check Before You Raise the Limit

Introduction — It Died With 137, but the Log Has No Exception

The container keeps restarting, yet the last line in the log is an ordinary info message.

kubectl get pod -n search

NAME                      READY   STATUS             RESTARTS       AGE
indexer-6c4d9f7b8-h2klp   0/1     CrashLoopBackOff   4 (94s ago)    22m

kubectl logs indexer-6c4d9f7b8-h2klp -n search --previous | tail -3

2026-07-26T00:38:41.220Z INFO  segment merge started shard=7 docs=1284000
2026-07-26T00:38:44.902Z INFO  segment merge progress shard=7 pct=61

No exception, no shutdown message. It means the process vanished without getting a chance to speak, and that is the signature of SIGKILL.

kubectl describe pod indexer-6c4d9f7b8-h2klp -n search | grep -A6 "Last State"

    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Sun, 26 Jul 2026 09:36:02 +0900
      Finished:     Sun, 26 Jul 2026 09:38:45 +0900
    Restart Count:  4

What 137 Tells You and What It Does Not

137 is 128 plus 9, and signal number 9 is SIGKILL. POSIX shells and container runtimes report the exit code of a process terminated by a signal as 128 plus the signal number.

There is something you have to distinguish precisely here. 137 only says that someone forcibly killed this process. It does not tell you whether memory was the reason. Several parties can send SIGKILL.

• The cgroup OOM killer in the kernel — when the container went past its memory limit
• The global OOM killer in the kernel — when the entire node ran out of memory
• kubelet — against a container that does not terminate within the grace period after SIGTERM
• Another management tool on the node, or a person

Whether memory is the cause is decided by the Reason field, not by the exit code. If Reason is OOMKilled it is memory, and if it is Error with 137 it is a forced kill after the grace period ran out. Giving the latter more memory has no effect at all.

kubectl get pods -A -o custom-columns='NS:.metadata.namespace,POD:.metadata.name,REASON:.status.containerStatuses[*].lastState.terminated.reason,CODE:.status.containerStatuses[*].lastState.terminated.exitCode' \
  | grep -v '<none>' | head -5

NS          POD                            REASON      CODE
search      indexer-6c4d9f7b8-h2klp        OOMKilled   137
payments    ledger-5f8b7c6d9-nm3xt         Error       137
media       transcode-79c5d6f8b-vn4pq      Error       143

The second pod shows 137, but it is not an OOM. It died while ignoring the termination signal, and the fix belongs on the graceful shutdown side.

The most reliable evidence is the counter the cgroup keeps itself.

kubectl exec -it indexer-6c4d9f7b8-h2klp -n search -- cat /sys/fs/cgroup/memory.events

low 0
high 0
max 3421
oom 2
oom_kill 2

If oom_kill is greater than zero, a process really was killed by OOM inside this cgroup. The max counter matters too. That value counts how many times usage touched the limit and the kernel tried to reclaim, so you use it to find containers that have not died yet but are running extremely close to the edge.

Container OOM, or Node-Wide OOM

The symptoms look similar, but the response is completely different.

A container cgroup OOM kills only that container. Going past the limit is that container own doing, and the other pods on the node are unharmed.

A node-wide OOM is the situation where the entire node ran out of memory and the kernel picks a victim. The process that dies may not be the one that caused it, and several pods die at once.

A kubelet eviction is the stage before that. When node memory falls below the eviction threshold, kubelet picks pods and terminates them, and in that case the status is Evicted, not OOMKilled.

One line of kernel message on the node settles it.

kubectl debug node/ip-10-0-1-14 -it --image=busybox -- sh

/ # dmesg -T | grep -i "out of memory" | tail -3
[Sun Jul 26 09:38:45 2026] Memory cgroup out of memory: Killed process 21847 (java) total-vm:9421884kB, anon-rss:2088412kB, file-rss:2104kB, shmem-rss:0kB, UID:1000 pgtables:5120kB oom_score_adj:939

The prefix Memory cgroup out of memory is decisive. If that phrase is there it is a container limit overrun, and if you get only Out of memory: Killed process without it, it is a node-wide OOM.

Evictions are confirmed through events.

kubectl get events -A --field-selector reason=Evicted --sort-by=.lastTimestamp | tail -3

NAMESPACE   LAST SEEN   TYPE      REASON    OBJECT                        MESSAGE
media       4m12s       Warning   Evicted   pod/transcode-79c5d6f8b-9klm  The node was low on resource: memory. Threshold quantity: 100Mi, available: 84Mi. Container transcode was using 3187Mi, request is 512Mi.

The message prints usage and request together. It means a pod that requested only 512Mi while using 3187Mi put the node at risk, and in this case what you fix is not the limit but the honesty of the request.

Who dies first in a global OOM is decided by the QoS class. kubelet sets oom_score_adj differently per container according to QoS.

• Guaranteed — requests and limits are equal for every container. oom_score_adj is -997, so it effectively dies last
• BestEffort — no requests and no limits. oom_score_adj is 1000, so it dies first
• Burstable — in between. The larger the memory request relative to node capacity, the lower the score and the better the odds of survival

kubectl get pods -A -o custom-columns='NS:.metadata.namespace,POD:.metadata.name,QOS:.status.qosClass' | grep BestEffort | head -3

NS          POD                             QOS
default     debug-shell                     BestEffort
media       thumbnailer-5c8d7f9b6-p4wnt     BestEffort

When the Runtime Misreads the Container Limit

You gave the container 2Gi, but the application frequently mistakes the 64Gi of the node for its own share.

JVM

Recent JVMs enable container awareness by default, so it is rare for the heap to be sized past the limit. The problem runs the other way. The default MaxRAMPercentage is 25 percent, so in a 2Gi container the heap ceiling lands at 512Mi.

kubectl exec -it indexer-6c4d9f7b8-h2klp -n search -- \
  java -XX:+PrintFlagsFinal -version 2>/dev/null | grep -E "MaxHeapSize|MaxRAMPercentage"

   size_t MaxHeapSize     = 536870912    {product} {ergonomic}
 double MaxRAMPercentage  = 25.000000    {product} {default}

If 137 happened here, the cause is not the heap. If the heap ceiling is 512Mi and the container died at 2Gi, it means the remaining 1.5Gi is being used outside the heap. Metaspace, thread stacks, the code cache, direct byte buffers, GC data structures, and native libraries belong here.

kubectl exec -it indexer-6c4d9f7b8-h2klp -n search -- jcmd 1 VM.native_memory summary

Total: reserved=4183MB, committed=1971MB
-                 Java Heap (reserved=512MB, committed=512MB)
-                     Class (reserved=1084MB, committed=68MB)
-                    Thread (reserved=418MB, committed=418MB)
-                      Code (reserved=250MB, committed=42MB)
-                        GC (reserved=76MB, committed=76MB)
-                  Internal (reserved=812MB, committed=804MB)

Threads are using 418MB. That is 400 threads created on the default 1MB stack. Growing the heap here only makes the situation worse.

A common wrong answer: seeing 137 and raising the heap size. A bigger heap means GC runs less often, so actual resident memory grows, and if off-heap was the cause you only pull the moment of death earlier. You have to look at what is big first.

The recommended setting is a combination of a percentage and ceilings, not absolute values.

env:
  - name: JAVA_TOOL_OPTIONS
    value: >-
      -XX:MaxRAMPercentage=70.0
      -XX:MaxMetaspaceSize=256m
      -XX:MaxDirectMemorySize=256m
      -XX:+ExitOnOutOfMemoryError
      -XX:+HeapDumpOnOutOfMemoryError
      -XX:HeapDumpPath=/dumps
resources:
  requests:
    memory: 2Gi
  limits:
    memory: 2Gi

-XX:+ExitOnOutOfMemoryError matters. Without this option, the JVM does not die even when the heap runs short, it stays alive burning all of its time in GC. It becomes the state that is hardest to diagnose, Running on the surface but not responding. It is better to die cleanly and restart.

Node.js

The old generation heap ceiling in V8 is decided from the system memory the process perceives, and combinations that fail to reflect the container limit accurately still exist. Stating it explicitly is safer.

kubectl exec -it api-gateway-7d8f9c6b5-x2mnp -n edge -- \
  node -e "console.log(require('v8').getHeapStatistics().heap_size_limit / 1048576)"

4144.0

The container limit is 1Gi, but V8 believes it can grow to 4GB. In this state V8 does not hurry its GC and the container dies first. You do not even get a JavaScript heap out of memory exception, just 137.

env:
  - name: NODE_OPTIONS
    value: '--max-old-space-size=768'
resources:
  requests:
    memory: 1Gi
  limits:
    memory: 1Gi

Give roughly 75 percent of the limit to the heap and leave the rest for buffers, native addons, and stacks. If the application uses Buffer heavily you have to go lower. Buffer is allocated outside the heap, so it is not counted toward max-old-space-size.

Python

Python has no concept of a heap ceiling. Instead, two things frequently cause trouble.

First, the per-thread memory arenas of glibc. With many threads the arenas multiply, and RSS grows far beyond actual usage.

env:
  - name: MALLOC_ARENA_MAX
    value: '2'

Second, worker processes accumulate leaks. Until the root fix lands, periodic recycling is a practical buffer.

command:
  - gunicorn
  - --workers=4
  - --max-requests=2000
  - --max-requests-jitter=200
  - app:application

Without --max-requests-jitter, the workers restart at the same moment and requests are cut off all at once.

When the Container Lives but a Feature Dies

The cgroup OOM killer does not kill the container, it kills the process with the highest score inside the cgroup. If a child worker that is not PID 1 is sacrificed, the container still shows as Running, RESTARTS does not go up, and only part of the functionality quietly stops.

kubectl exec -it api-gateway-7d8f9c6b5-x2mnp -n edge -- cat /sys/fs/cgroup/memory.events | grep oom_kill

oom_kill 3

RESTARTS is 0 but oom_kill is 3. When you see this combination it means child processes are dying. The fix is to make the supervising process detect a child death and exit on its own. Only then can Kubernetes step in.

working set and RSS, and the Page Cache

The memory metrics have similar names but different meanings. Put an alert on the wrong metric and you will suffer false positives every day.

• memory.current (cgroup v2) — everything this cgroup occupies. Anonymous memory and the page cache are both included. This is the value the limit check is based on
• working set — memory.current minus the inactive file cache. This is the value kubectl top shows, and the basis of the kubelet eviction decision
• RSS — the pages the process actually placed in physical memory. Shared libraries are double counted and the page cache is left out

Here is where the trap appears. An application that reads many files piles up page cache, so memory.current sticks to the limit. Looking at the graph alone you would think it is about to die, but the cache is reclaimable, so it does not actually die.

kubectl exec -it indexer-6c4d9f7b8-h2klp -n search -- sh -c \
  'cat /sys/fs/cgroup/memory.max; cat /sys/fs/cgroup/memory.current; grep -E "^(anon|file|slab) " /sys/fs/cgroup/memory.stat'

2147483648
2091483136
anon 1183842304
file 874512384
slab 33124352

Of the 2091MB total, anonymous memory is 1183MB and the file cache is 874MB. Looking at anonymous memory alone, there is room to spare.

kubectl top pod indexer-6c4d9f7b8-h2klp -n search

NAME                      CPU(cores)   MEMORY(bytes)
indexer-6c4d9f7b8-h2klp   410m         1216Mi

This is why kubectl top reports 1216Mi. It is the value with the inactive file cache subtracted.

A common wrong answer: putting the alert on container_memory_usage_bytes. This metric includes the page cache, so it climbs close to the limit even during normal file I/O. Alerts have to be wired to container_memory_working_set_bytes.

There is one caveat, though. When the cache is of a kind that is not reclaimed, for example data placed on tmpfs or shared memory, it is not a reclaim target and pushes straight against the limit. A volume with emptyDir and medium set to Memory is the classic case. Everything written to this volume counts toward the container memory limit.

volumes:
  - name: scratch
    emptyDir:
      medium: Memory
      sizeLimit: 256Mi

Always specify sizeLimit. Without it, it can use up to half of the node memory.

How to Decide requests and limits

When it comes to memory, setting requests and limits to the same value is usually right. The reason is that memory is an incompressible resource. CPU merely gets slower when it is short, but memory kills when it is short. A pod that uses more than its requests betrays the calculation the scheduler made, and the price is paid by the other pods on the same node.

resources:
  requests:
    cpu: 500m
    memory: 2Gi
  limits:
    memory: 2Gi

There is a common argument here, whether you should match requests and limits across the board to obtain Guaranteed QoS. Summed up, it goes like this.

• Keep memory equal — it is sacrificed last under node-wide pressure, and it prevents chain accidents caused by overcommit
• Specify CPU requests, be careful with CPU limits — a CPU limit triggers cgroup bandwidth throttling. Latency spikes show up from hitting the limit even on a node with room to spare, so for latency-sensitive services it is common to omit CPU limits and secure only the share through requests
• That said, if you omit CPU limits you cannot obtain Guaranteed QoS. It is a question of which matters more, node-wide OOM defense or latency stability, and neither side is always right

A common wrong answer: doubling the limit every time 137 shows up. If it is a leak you only stretch the interval between deaths, and in the meantime you need one more node. Look at the time series first. If it rises and falls in a sawtooth it is a normal GC pattern and the limit really is too small. If it rises monotonically and then drops off a cliff, it is a leak.

Recurrence Prevention — Build the Evidence and Catch It Early

Ground It in Actual Usage With VPA

Running it in recommendation mode alone is worth plenty. Automatic application involves recreating pods, so it calls for care.

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: indexer
  namespace: search
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: indexer
  updatePolicy:
    updateMode: 'Off'

kubectl get vpa indexer -n search \
  -o jsonpath='{.status.recommendation.containerRecommendations[0]}'

{"containerName":"indexer","lowerBound":{"cpu":"320m","memory":"1300Mi"},"target":{"cpu":"480m","memory":"1730Mi"},"uncappedTarget":{"cpu":"480m","memory":"1730Mi"},"upperBound":{"cpu":"1100m","memory":"2600Mi"}}

target is a recommendation based on the observed usage distribution, and upperBound comes out more generous the shorter the observation window. You need at least a week, and ideally a window that includes the traffic peak, for the values to mean anything.

Get the Alert Before It Dies

An alert that arrives after death is a post-mortem notice. You have to catch it at the stage where it approaches the limit.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: memory-alerts
  namespace: monitoring
spec:
  groups:
    - name: memory
      rules:
        - alert: ContainerOOMKilled
          expr: increase(kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}[10m]) > 0
          labels:
            severity: critical
          annotations:
            summary: "{{ $labels.namespace }}/{{ $labels.pod }} was OOMKilled"
        - alert: ContainerMemoryNearLimit
          expr: |
            container_memory_working_set_bytes{container!="",container!="POD"}
              / on(namespace,pod,container) container_spec_memory_limit_bytes{container!=""} > 0.9
          for: 15m
          labels:
            severity: warning
        - alert: ContainerMemoryGrowsMonotonically
          expr: |
            deriv(container_memory_working_set_bytes{container!="",container!="POD"}[6h]) > 0
              and increase(container_memory_working_set_bytes{container!="",container!="POD"}[6h]) > 200e6
          for: 1h
          labels:
            severity: warning

The third rule is the early warning for leaks. If it rises monotonically for 6h and grows by 200MB or more, it is hard to read that as normal cache warm-up.

Secure the Post-Mortem Material in Advance

A container killed by OOM leaves nothing behind. Attach a volume so the dump goes out before it dies.

volumeMounts:
  - name: dumps
    mountPath: /dumps
volumes:
  - name: dumps
    persistentVolumeClaim:
      claimName: heap-dumps

For the JVM you secure it with the HeapDumpOnOutOfMemoryError seen earlier, for Node.js with a signal-based snapshot, and for Python with memray or tracemalloc. Note, though, that writing a heap dump needs additional memory, so if the limit has no headroom the dump itself can fail.

Closing — Look at Reason First, Raise the Limit Last

The order for responding to OOMKilled is fixed. Confirm that Reason really is OOMKilled, back it up with the oom_kill counter of the cgroup, split container OOM from node-wide OOM with dmesg, inspect whether the runtime recognizes the limit properly, and see whether the working set time series is a sawtooth or a monotonic rise. Only after you get this far does adjusting the limit become a decision with evidence.

The one sentence to remember is this. 137 is not a diagnosis but a cause of death unknown, and the only things that confirm memory are the Reason field and the oom_kill counter.