Split View: OOM Killer가 프로세스를 죽였을 때 — dmesg 로그 해독부터 cgroup OOM 구분까지

When the OOM Killer Kills a Process — From Decoding the dmesg Report to Telling cgroup OOM Apart

Introduction — The Application Log Is Clean to the Very End, but the Process Is Gone

Incident reports usually arrive like this. "The service died but there is no error in the application log. The last line is a normal request being served." And in a container environment the exit code reads 137.

The fact that there is nothing in the application log is the key clue. SIGKILL cannot be caught and no handler runs. The process evaporates in the middle of executing a single instruction. So the trace is left by the kernel, not by the application.

sudo dmesg -T | grep -iE 'killed process|out of memory' | tail -5
# [Sat Jul 25 03:14:22 2026] Out of memory: Killed process 21874 (java) total-vm:12582912kB, anon-rss:11536876kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:22912kB oom_score_adj:0

If this one line shows up, it is confirmed. If it does not, this was probably not an OOM. This post starts from finding that line, moves on to why it had to be that particular process, and ends with what you should change so it does not happen again.

How to Find the OOM Log and How to Read It Line by Line

Finding it comes first. The dmesg ring buffer is finite, so entries are pushed out as time passes. To query across a reboot, use journald.

# Search the kernel messages of the current boot session
sudo journalctl -k --since "2 hours ago" --grep "Out of memory"

# Previous boot session (if the machine was rebooted after the OOM)
sudo journalctl -k -b -1 --grep "oom"

# Everything, not just the kernel log (includes systemd-oomd and the container runtime)
sudo journalctl --since "2026-07-25 03:00" --until "2026-07-25 03:30" | grep -iE 'oom|killed'

If journalctl -k has nothing and dmesg has nothing either, the Storage=volatile setting may have made the log disappear on reboot. In that case Storage=persistent in /etc/systemd/journald.conf becomes the first item on your prevention list.

Now read the whole report. The OOM report the kernel prints is made of four parts.

[Sat Jul 25 03:14:22 2026] java invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
[Sat Jul 25 03:14:22 2026] CPU: 3 PID: 21874 Comm: java Not tainted 6.8.0-45-generic #45-Ubuntu
[Sat Jul 25 03:14:22 2026] Call Trace:
[Sat Jul 25 03:14:22 2026]  dump_stack_lvl+0x48/0x70
[Sat Jul 25 03:14:22 2026]  dump_header+0x4f/0x240
[Sat Jul 25 03:14:22 2026]  oom_kill_process+0x10d/0x1c0
[Sat Jul 25 03:14:22 2026]  out_of_memory+0x246/0x580
[Sat Jul 25 03:14:22 2026]  __alloc_pages_slowpath+0xb6c/0xf70
[Sat Jul 25 03:14:22 2026] Mem-Info:
[Sat Jul 25 03:14:22 2026] active_anon:3812044 inactive_anon:118233 isolated_anon:0
[Sat Jul 25 03:14:22 2026]  active_file:214 inactive_file:302 unevictable:0
[Sat Jul 25 03:14:22 2026]  slab_reclaimable:38210 slab_unreclaimable:91043
[Sat Jul 25 03:14:22 2026]  free:41283 free_pcp:812 free_cma:0
[Sat Jul 25 03:14:22 2026] Node 0 Normal free:165132kB min:162044kB low:202552kB high:243060kB
[Sat Jul 25 03:14:22 2026] Tasks state (memory values in pages):
[Sat Jul 25 03:14:22 2026] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[Sat Jul 25 03:14:22 2026] [   1042]     0  1042    28901     1204     126976        0             0 systemd-journal
[Sat Jul 25 03:14:22 2026] [   1863]     0  1863     4521      612      73728        0         -1000 sshd
[Sat Jul 25 03:14:22 2026] [  21874]  1000 21874  3145728  2884219   23461888        0             0 java
[Sat Jul 25 03:14:22 2026] [  22910]   999 22910   612344   498231    4145152        0           200 postgres
[Sat Jul 25 03:14:22 2026] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/app.service,task=java,pid=21874,uid=1000
[Sat Jul 25 03:14:22 2026] Out of memory: Killed process 21874 (java) total-vm:12582912kB, anon-rss:11536876kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:22912kB oom_score_adj:0

Here is what to pull out of each part.

The first line, the one with invoked oom-killer, is the process that triggered the OOM. It can be different from the process that died. The side that asked for an allocation and failed is the trigger, while the side that dies is decided by score. order=0 means a single 4KB page was requested, and a large order value (3 or more) means a physically contiguous large block could not be found, so the problem may be fragmentation rather than total capacity.

The Mem-Info block shows whether reclaimable memory was really gone. In the example above active_file and inactive_file are 214 and 302 pages respectively. That means barely 1MB of page cache was left, which is evidence that the cache had been dropped as far as it could go and it still was not enough. free sitting near the min watermark tells the same story. Conversely, if active_file is tens of GB and an OOM still happened, it is not a total capacity problem but a problem with a specific zone or node.

The Tasks state table is measured in pages. On 4KB pages, the rss of 2884219 pages for java is about 11.0GiB. That matches anon-rss:11536876kB on the last line. This table is a full snapshot of the moment of the OOM, so it is the only material that reveals the real culprit when the culprit is not the process that died. Read it sorted by the rss column.

The oom-kill: line carries the most information and is ignored the most often. If constraint is CONSTRAINT_NONE it is a system-wide memory shortage, if it is CONSTRAINT_MEMCG it is a cgroup limit, and if it is CONSTRAINT_CPUSET or CONSTRAINT_MEMORY_POLICY it is a NUMA node constraint. The global_oom token and the task_memcg path live here as well.

The last line splits RSS into three useful categories. anon-rss is heap and stack, file-rss is mapped files, and shmem-rss is shared memory and tmpfs. If shmem-rss is large, suspect /dev/shm or a tmpfs mount. Data written to tmpfs looks like a file but occupies memory and is never pushed out to disk.

The Calculation That Decides Who Dies — badness and oom_score_adj

The kernel scores every candidate and kills the highest one. The calculation goes like this.

points = rss + swapents + (pgtables_bytes / PAGE_SIZE)      # unit: pages
adj    = oom_score_adj * (totalpages / 1000)
points = points + adj

Three things follow immediately.

First, the score is based on real resident memory (RSS) and not virtual memory (total_vm). A process that merely reserves 64GB with mmap and actually touches only 200MB is not a candidate. Naming a culprit by looking at VIRT in top is almost always wrong.

Second, oom_score_adj ranges from -1000 to 1000 and is a bonus or penalty measured in thousandths of total memory. On a 64GiB machine, oom_score_adj=200 adds a score equivalent to about 12.8GiB. Even a process whose real RSS is 2GB can outrank an 11GB JVM through that one adjustment. The postgres in the log above had oom_score_adj=200, but the RSS of java was so large that java was the one selected.

Third, oom_score_adj=-1000 is special. It is excluded from the score calculation altogether and is never selected. That is why distributions set this value on sshd. You can confirm it on the sshd line of the log.

Here is how to see the current values.

# Sort processes by score together with their adjustment value
for p in /proc/[0-9]*; do
  pid=${p#/proc/}
  [ -r "$p/oom_score" ] || continue
  printf '%8s %7s %6s %s\n' \
    "$(cat "$p/oom_score" 2>/dev/null)" \
    "$(cat "$p/oom_score_adj" 2>/dev/null)" \
    "$pid" \
    "$(tr -d '\0' < "$p/comm" 2>/dev/null)"
done | sort -rn | head -8
# 2884219       0  21874 java
#  501431     200  22910 postgres
#   38210       0   3311 containerd
#    1204       0   1042 systemd-journal
#       0   -1000   1863 sshd

The scale of oom_score differs between kernel versions. Recent kernels are page-count based, while older kernels normalized to a 0 through 1000 range. Do not use the absolute value as a threshold; use it only for relative comparison between processes.

There are three ways to change the value.

# 1) Directly on a running process
echo -500 | sudo tee /proc/21874/oom_score_adj

# 2) On a systemd unit (survives restarts)
sudo systemctl edit myapp.service
# [Service]
# OOMScoreAdjust=-500

# 3) When launching the process
sudo choom -n -500 -- /usr/bin/myapp

In a container environment the runtime sets this value for you. Kubernetes assigns it by QoS class as follows.

The Burstable result is clamped between -996 and 999. A practically important conclusion falls out of this. The larger you set the memory request, the higher the odds of surviving a node-wide OOM. A pod that writes a small request and actually uses a lot is the first to die the moment the node comes under pressure. A request is not a number for fooling the scheduler; it is a survival ranking.

A Container OOM and a System-Wide OOM Are Different Events

This is the point most often confused in practice. Both produce exit code 137 and both leave an OOM log in dmesg, but the cause and the response are completely different.

A cgroup OOM log looks like this.

[Sat Jul 25 04:02:11 2026] node invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=969
[Sat Jul 25 04:02:11 2026] memory: usage 524288kB, limit 524288kB, failcnt 1842
[Sat Jul 25 04:02:11 2026] swap: usage 0kB, limit 0kB, failcnt 0
[Sat Jul 25 04:02:11 2026] Memory cgroup stats for /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a1c8f2e.slice/cri-containerd-3f7b91ac.scope:
[Sat Jul 25 04:02:11 2026] anon 521142272
[Sat Jul 25 04:02:11 2026] file 1048576
[Sat Jul 25 04:02:11 2026] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=cri-containerd-3f7b91ac.scope,mems_allowed=0,oom_memcg=/kubepods.slice/...,task_memcg=/kubepods.slice/...,task=node,pid=33127,uid=1000
[Sat Jul 25 04:02:11 2026] Memory cgroup out of memory: Killed process 33127 (node) total-vm:1284736kB, anon-rss:508924kB, file-rss:36104kB, shmem-rss:0kB, UID:1000 pgtables:2048kB oom_score_adj:969

There are three ways to tell them apart, and any one of them alone is conclusive.

• Whether the last line reads Out of memory: or Memory cgroup out of memory:
• Whether constraint on the oom-kill: line is CONSTRAINT_NONE or CONSTRAINT_MEMCG
• Whether the memory: usage / limit / failcnt line exists. This line appears only on a cgroup OOM

If you could not get the log, counters confirm it too. cgroup v2 accumulates events.

# Find the cgroup path of the pod first
cat /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a1c8f2e.slice/memory.events
# low 0
# high 0
# max 1842
# oom 7
# oom_kill 3

max is the number of times the limit was hit and reclaim happened, oom is the number of times reclaim failed and the OOM path was entered, and oom_kill is the number of times something was actually killed. If only max is large and oom_kill is 0, it means the group is holding on near the limit by reclaiming continuously. Performance is degrading but nothing has died yet. Catching this state is far better than a postmortem.

Here is the second common misconception. Exit code 137 is not evidence of an OOM. 137 is 128 plus 9, which only means the process was killed by SIGKILL. It is also 137 when a container dies after the grace period following a failed liveness probe, when you remove it with kubectl delete, and when it dies during node shutdown. To confirm, you have to look at the reason field of the container status or at the kernel log above.

kubectl get pod api-7d9f8-x2k4l -o jsonpath='{.status.containerStatuses[0].lastState.terminated}' | python3 -m json.tool
# {
#     "containerID": "containerd://3f7b91ac...",
#     "exitCode": 137,
#     "finishedAt": "2026-07-25T04:02:11Z",
#     "reason": "OOMKilled",
#     "startedAt": "2026-07-25T03:11:04Z"
# }

There is a trap in the opposite direction too. If a node-wide OOM kills a process inside a container, that pod also receives 137. The reason then reads OOMKilled, but the memory limit may not have been exceeded at all. Raising the limit will not stop it from happening again. If constraint=CONSTRAINT_NONE, the answer is to grow the node itself or to evict other pods.

What overcommit_memory 0/1/2 Actually Changes

The difference between the three values is when and in what form the failure appears. They do not change the total amount of memory.

The ceiling of mode 2 is calculated as follows.

sysctl vm.overcommit_memory vm.overcommit_ratio
# vm.overcommit_memory = 0
# vm.overcommit_ratio = 50

grep -E 'CommitLimit|Committed_AS|MemTotal|SwapTotal' /proc/meminfo
# MemTotal:       65806232 kB
# SwapTotal:             0 kB
# CommitLimit:    32903116 kB
# Committed_AS:   48192044 kB

CommitLimit is all of swap plus the overcommit_ratio percentage of physical memory. In the example above there is no swap, so it is 32GiB, which is 50% of 64GiB. Committed_AS is already 46GiB, so switching to vm.overcommit_memory=2 right now would make new allocations start failing immediately. Turning on mode 2 with the default ratio on a machine without swap is a textbook own goal. If you want to use it, raise vm.overcommit_ratio to 90 or higher, or specify an absolute value with vm.overcommit_kbytes.

The classic case that needs mode 1 is Redis. BGSAVE creates a child with fork, and because of copy on write the real extra usage is far smaller, but the heuristic of the kernel may see it as a commit the same size as the parent. That is why Redis prints a warning at startup recommending vm.overcommit_memory=1.

# Apply permanently
echo 'vm.overcommit_memory = 1' | sudo tee /etc/sysctl.d/60-overcommit.conf
sudo sysctl --system

One caution here. Mode 1 calls the OOM Killer more often. Since nothing is refused at allocation time, it blows up when the pages are actually touched. If the application is designed to handle a malloc failure gracefully, mode 2 is better; if it is not (which is most of the time), keeping mode 0 and isolating with cgroup limits is the realistic choice.

The Myth That "Adding Swap Means No More OOM"

This is the most widespread misconception and it is only half right. Swap does not prevent OOM, it delays it. And the state the system goes through during that delay is often worse than dying instantly.

A global OOM does not fire "when memory reaches zero" but "when reclaim was attempted and made no progress." With swap available, anonymous pages can be pushed out, so reclaim makes progress and the OOM is postponed. The problem is that in this window the working set thrashes in and out of swap. Processes are alive but get nothing done, health checks time out, and even an SSH login takes several minutes.

Here is how to confirm thrashing.

vmstat 1 5
# procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
#  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
#  1 18  8291840  92104   1024  38912 41216 38912 42104 39204 8210 19204  2  9  4 85  0

If si (swap in) and so (swap out) are both tens of MB per second at the same time, that is thrashing. If only one side is large it may be normal page-out, so you have to look at both directions together. PSI makes it even clearer.

cat /proc/pressure/memory
# some avg10=94.12 avg60=88.03 avg300=61.44 total=182937461283
# full avg10=71.88 avg60=66.21 avg300=44.02 total=91827364512

A full of 71% means that for more than seven of the last ten seconds every task was stalled because of memory. The CPU may look idle and the load may be low, but the system is effectively frozen.

Two more points. Inside a cgroup, memory.swap.max exists separately, and if that value is 0 the group cannot use swap no matter how much swap the host has and goes straight to OOM. Kubernetes disabled node swap by default for a long time and only recently gained limited support, so in container workloads the assumption that "we have swap, so we are fine" generally does not hold.

And zram or zswap are a different animal. They compress inside memory rather than on disk, so latency is far lower, and on workloads that compress well there is a real memory expansion effect. That said, compression itself burns CPU, and it does nothing for data that does not compress.

Preventing Recurrence — Stepping In Before the Kernel Kills

The kernel OOM Killer is a last resort, and a last resort is late. The practical goal is to intervene predictably at an earlier stage.

First, declare limits explicitly with cgroups. A global OOM makes it hard to predict which process will die, but a cgroup OOM is confined to that group. For a systemd service, write it straight into the unit.

sudo systemctl edit myapp.service
# [Service]
# MemoryHigh=6G
# MemoryMax=8G
# OOMPolicy=stop

sudo systemctl daemon-reload
sudo systemctl restart myapp

systemctl show -p MemoryHigh -p MemoryMax -p OOMPolicy myapp.service
# MemoryHigh=6442450944
# MemoryMax=8589934592
# OOMPolicy=stop

The difference between MemoryHigh and MemoryMax matters. MemoryMax (the memory.max of the cgroup) kills once exceeded. MemoryHigh (memory.high) does not kill but puts the allocating side to sleep. A penalty sleep proportional to the excess is applied, so the application slows down sharply but stays alive. Setting both creates a buffer zone of "brake at 6GB, terminate at 8GB", and an alert in that zone buys time for a human to step in.

Second, alert on the pressure itself. Pressure before death is more useful than a counter tallied after death.

# Top 5 cgroups by memory pressure
for f in /sys/fs/cgroup/system.slice/*/memory.pressure; do
  v=$(awk '/^full/ {print $3}' "$f" 2>/dev/null | cut -d= -f2)
  [ -n "$v" ] && printf '%7s  %s\n' "$v" "${f%/memory.pressure}"
done | sort -rn | head -5
#   18.44  /sys/fs/cgroup/system.slice/myapp.service
#    0.31  /sys/fs/cgroup/system.slice/containerd.service

If you use Prometheus, watch the increase of node_vmstat_oom_kill from node_exporter together with container_oom_events_total from cAdvisor. The former is node-wide and the latter is per container, so the two events we separated above are captured by two different metrics.

Third, consider a userspace OOM manager. The kernel waits until reclaim has failed completely, but userspace tools set their own thresholds.

• earlyoom watches the MemAvailable and SwapFree ratios and sends SIGTERM when they drop below a threshold, then SIGKILL if they drop further. It is simple to configure and works on any distribution.
• systemd-oomd is PSI based. It watches memory.pressure per cgroup and, when the threshold is exceeded for a sustained period, cleans up whatever is under the most pressure in that slice. Because it uses pressure as its metric, it has fewer false positives than an absolute capacity threshold.

# earlyoom: step in below 5% available memory and 5% swap
sudo systemctl edit earlyoom.service
# [Service]
# Environment=EARLYOOM_ARGS=-m 5 -s 5 --avoid '(^|/)(sshd|systemd)$' --prefer '(^|/)java$'

# systemd-oomd: per-slice pressure threshold
sudo systemctl edit myapp.service
# [Service]
# ManagedOOMMemoryPressure=kill
# ManagedOOMMemoryPressureLimit=60%

systemd-oomd only acts when the threshold is continuously exceeded for DefaultMemoryPressureDurationSec (30 seconds by default) in /etc/systemd/oomd.conf, so it does not react to momentary spikes.

Fourth, measure the actual usage again. Raising the limit is not a root cause fix. For a JVM, check whether -XX:MaxRAMPercentage recognizes the container limit; for Go, check whether GOMEMLIMIT is set; and check whether native heap fragmentation or a glibc arena issue is at play. smem or /proc/PID/smaps_rollup shows you the composition of RSS.

sudo cat /proc/21874/smaps_rollup
# 7f2c00000000-7ffd1a3e5000 ---p 00000000 00:00 0 [rollup]
# Rss:            11536876 kB
# Pss:            11498210 kB
# Anonymous:      11402344 kB
# AnonHugePages:   2097152 kB
# Shared_Clean:      34120 kB
# Private_Dirty:  11402344 kB

If Anonymous is most of RSS it is the application heap, and if Shared_Clean is large it is shared libraries, so the real cost is closer to Pss.

Closing — One constraint Line in the Log Decides Your Response

If you remember one thing, make it this. The first fork in the road of OOM debugging is global or cgroup, and the answer is already written in the constraint field of the oom-kill: line.

• CONSTRAINT_NONE means the node is short on memory. Raising the pod limit will not stop the recurrence. You have to grow the node or lower the density.
• CONSTRAINT_MEMCG means it is a limit problem for that container. The node is fine. Raising the limit or reducing application usage is the right move.

And then the remaining rules.

1. Do not conclude OOM from exit code 137 alone. Confirm with the reason field or the kernel log.
2. Find the culprit by rss, not by total_vm. Virtual memory does not enter the score.
3. Write the memory request honestly. The oom_score_adj of a Burstable pod is derived from that value, and that is your survival ranking under node pressure.
4. Do not set only MemoryMax; build a buffer zone with MemoryHigh. Slowing down before dying gives you time to diagnose.
5. Alert on full in memory.pressure, not on the oom_kill counter. You need a warning in advance, not a notification after the fact.