Why Load Average Is Not CPU Utilization — load average 24 With the CPU at 30%

Introduction — load average 24, and yet the CPU is at 30%

The alert arrives before dawn. The load average on an 8 core server has gone past 24. You log in, open top, and more than 60% of the CPU is still idle. It is easy to slide into one of two conclusions here. "Monitoring is broken" or "we need three times the CPU". Both are wrong.

The load average is not CPU utilization. Even the unit is different. Utilization is a ratio (0 to 100%) and load is a count (a number of tasks). And when Linux counts that number, it counts one more thing that other Unix systems do not. That one extra thing is half of this article.

uptime
#  03:41:52 up 12 days,  3:21,  2 users,  load average: 24.31, 22.08, 14.95

nproc
# 8

cat /proc/loadavg
# 24.31 22.08 14.95 2/1284 48210

Look at 2/1284 on the last line. The 2 in front is the number of threads that are runnable at this instant, and 1284 is the total number of threads. The load is 24, yet only 2 threads are runnable. Where are the other 22? The answer to that question is the starting point of the diagnosis.

What the three numbers really are — 5 second sampling and an exponentially damped moving average

The kernel samples the number of active tasks on each CPU run queue once per LOAD_FREQ period, that is, once every 5 seconds (precisely 5 seconds plus 1 tick, deliberately so that the sampling does not stay in phase with periodic interrupts). Active tasks are computed as follows.

nr_active = rq->nr_running + rq->nr_uninterruptible

nr_running is the tasks that are currently running or runnable, and nr_uninterruptible is the tasks in uninterruptible sleep. Summing this across every CPU gives the sample for that instant.

The update formula that follows is the heart of it. The kernel computes the following in fixed point arithmetic.

load = (load * EXP + active * (FIXED_1 - EXP)) >> FSHIFT

FSHIFT  = 11,  FIXED_1 = 2048
EXP_1   = 1884   (2048 * e^(-5/60))
EXP_5   = 2014   (2048 * e^(-5/300))
EXP_15  = 2037   (2048 * e^(-5/900))

Checking the constants by hand makes this click faster. To update every 5 seconds and still obtain a 1 minute time constant, the decay factor has to be e to the power of minus 5/60, that is, 0.9200. Multiply 0.9200 by 2048 and you get 1884. The 5 minute and 15 minute values come out as 2014 and 2037 the same way.

One property that matters in practice falls out of this. This is not an arithmetic mean but an exponentially damped average. When load jumps like a step function, the 1 minute average reaches exactly 63.2% of the final value after 1 minute, and reaching 99% takes about 5 minutes. By the same logic, the 15 minute average keeps getting dragged along for more than an hour.

That is why the idiomatic reading "if the 1 minute average is larger than the 5 and 15 minute ones, load is rising" holds. Conversely, it is normal for only the 15 minute average to stay high right after an incident ends. Put an alert on that number and it will keep ringing for 20 minutes after recovery.

The decisive Linux difference — D state is counted too

Traditional Unix defined the load average purely as run queue length, that is, the number of tasks waiting for CPU. In 1993 Linux added the TASK_UNINTERRUPTIBLE state to that. The intent was explicit. The goal was to measure demand across system resources in general, not demand for CPU alone.

TASK_UNINTERRUPTIBLE is the state that shows up as D in ps. It is a wait inside the kernel that not even a signal can break, and the representative cases are these.

• Waiting for block device I/O to complete (io_schedule)
• Waiting on a page cache lock (folio_wait_bit_common)
• Waiting for a response from a hard mounted NFS server
• Waiting on certain kernel mutexes and semaphores
• Waiting for a firmware response in a device driver

CPU is not on that list. So the load average can climb as high as you like while the CPU sits completely idle. If a single NFS server stops responding, every process that touched that mount piles up in D state, and an 8 core server hits a load of 200 while the CPU graph stays flat. This is correct behavior.

One exception is useful to know. Since kernel 4.2, TASK_IDLE (TASK_UNINTERRUPTIBLE | TASK_NOLOAD) was introduced, so kernel threads that sit waiting with nothing to do are not counted in the load. The ones that appear as I in ps belong here.

ps -eo state,pid,comm --no-headers | awk '{c[$1]++} END {for (s in c) print c[s], s}' | sort -rn
#    1183 S
#      68 I
#      22 D
#       2 R
#       9 Z

D is 22 and R is 2. That lines up exactly with the load of 24 seen earlier. At this point there is only one conclusion. This server is not short on CPU, it is waiting on I/O somewhere.

Here it is worth naming a wrong explanation that turns up often. The sentence "the load average is the length of the CPU wait queue" is right on other Unix systems and wrong on Linux. And the rule that "load below the core count is safe" guarantees no safety either, as long as D state is mixed into that value.

Where "divide by the core count" breaks down

Dividing the load by nproc and calling under 1.0 headroom and over 1.0 saturation is usable as a starting point. But it breaks down in the following five situations.

First, D state contamination. Exactly as in the previous section. 24/8 = 3.0, yet that has nothing to do with CPU saturation.

Second, the lag of the average. A 30 second spike does not surface as even half of itself in the 1 minute average. Conversely, an incident that already ended lingers in the 15 minute average.

Third, SMT (hyperthreading). nproc counts logical CPUs. With 8 physical cores and SMT enabled you get 16, but the actual processing capacity is not 16 cores. A load of 12 does not mean 75% of 16 cores.

lscpu | grep -E '^CPU\(s\)|Thread|Core|Socket'
# CPU(s):                          16
# Thread(s) per core:              2
# Core(s) per socket:              8
# Socket(s):                       1

Fourth, cgroup quotas. A task throttled by a CPU limit on a container drops out of the run queue altogether, so it never registers in the load average. On a 64 core host, a pod capped at 0.5 CPU can be severely slow while the host load stays quiet. This combination comes up again later.

Fifth, the absence of attribution. The load is a single system wide scalar. It tells you nothing about which container, which service, or which resource is responsible. You can raise an alert on it, but the only thing that alert lets you do is go in and look.

High load with an idle CPU — the diagnostic path

There is an order to this. Top to bottom, three minutes and you are done.

# Step 1: check the composition of the load — few R and many D means the I/O path
vmstat 1 5
# procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
#  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
#  2 22      0 512344  84120 6231044    0    0     0     0 2103 4102  4  3 21 72  0
#  1 23      0 511208  84120 6231180    0    0 132096   412 2288 4390  3  4 19 74  0
#  2 22      0 510992  84120 6231212    0    0 128512   380 2251 4301  4  3 20 73  0

The b column is 22 to 23. That column is exactly the number of tasks in uninterruptible sleep. wa (iowait) at 72% tells the same story. iowait is "the time in which the CPU was idle while a task on that CPU was waiting on I/O", so it is really a kind of idle time. A high value is not bad in itself, but read together with the b column the direction is unmistakable.

# Step 2: find out which device is saturated
iostat -x 1 3
# Device      r/s     rkB/s  rrqm/s %rrqm r_await rareq-sz    w/s   wkB/s  w_await  aqu-sz  %util
# nvme0n1   1842.0   58944.0    0.0  0.00   12.85    32.00  210.0  3360.0     3.02   24.30  99.60
# nvme1n1      2.0      64.0    0.0  0.00    0.31    32.00    1.0    12.0     0.22    0.00   1.20

There are three things to read. Whether %util is pinned at 100% (the device never rests), how many times the usual value r_await/w_await have become (latency per request), and how deep aqu-sz is (average number of requests queued). On NVMe, %util at 100% may not mean saturation because of parallelism, so make the call from await and queue depth.

# Step 3: check the D state processes one by one
ps -eo state,pid,ppid,comm --no-headers | awk '$1 == "D"' | head
# D  4821  4102 postgres
# D  4822  4102 postgres
# D  7710     1 kworker/u33:2

# Step 4: find where in the kernel that process is stuck (root privileges required)
sudo cat /proc/4821/stack
# [<0>] io_schedule+0x46/0x80
# [<0>] folio_wait_bit_common+0x131/0x330
# [<0>] filemap_fault+0x5f0/0xa50
# [<0>] __do_fault+0x39/0x120
# [<0>] handle_mm_fault+0xd4e/0x1090

If the top of the stack is io_schedule it is block I/O, if rpc_wait_bit_killable or nfs_wait_on_request shows up it is NFS, and if it is in the __lock_page family it is page cache contention. If /proc/PID/stack is empty or returns a permission error, the kernel was built without CONFIG_STACKTRACE or you lack the privileges.

When NFS is the suspect, look at the mount statistics directly.

# Check per server RTT and retransmits
nfsstat -c | head -20
mountstats --nfs /mnt/shared 2>/dev/null | grep -A3 'READ:'

# Find the server that is not answering (a hard mount waits forever)
grep nfs /proc/mounts
# 10.0.9.4:/export/data /mnt/shared nfs4 rw,hard,proto=tcp,timeo=600,retrans=2 0 0

A hard mount pins processes in D state until the server comes back. In that state even kill -9 has no effect. Operationally, consider softerr or a shorter timeo, but confirm the data consistency requirements first.

Low load and yet slow — the things load cannot see

The opposite direction is common too. The load is 0.8 and p99 is 3 seconds. Listing what load structurally cannot see narrows the candidates.

• Interruptible waits (S state) are not counted in the load. Time spent waiting on a socket response, that is, latency in a downstream API or database, all belongs here. Most of the reasons an application is slow live on this side.
• Userspace lock contention is mostly S state as well. A futex wait is interruptible, so no matter how bad it gets the load does not rise.
• cgroup CPU throttling, as noted above, is invisible because it removes tasks from the run queue.
• Single thread bottlenecks are load 1 by definition. Even with 64 cores the load never goes above 1.
• Hypervisor steal time means the CPU was taken away, not that the queue got longer.

The commands that separate these five are as follows.

# What an S state task is waiting on (socket waits do not show up here, so use the stack)
sudo cat /proc/9931/stack | head -5

# Lock contention: whether context switches have exploded
vmstat 1 3        # how many times the usual value the cs column has become
pidstat -w -p 9931 1 3
# UID  PID   cswch/s nvcswch/s  Command
# 1000 9931  18422.0   34110.0  java

# Throttling (cgroup v2)
cat /sys/fs/cgroup/cpu.stat
# nr_periods 86400
# nr_throttled 41287
# throttled_usec 2914837261

# steal
mpstat -P ALL 1 3 | tail -3

A large nvcswch/s (involuntary context switches) means CPU contention, and a large cswch/s (voluntary) means a pattern of falling asleep and waking again on locks or I/O.

PSI — the metric that answers what load average could not

The fundamental problem with the load average is that it says only how many are waiting and never how long anything was stalled. PSI, which arrived in kernel 4.20, measures exactly the latter.

cat /proc/pressure/io
# some avg10=78.23 avg60=71.04 avg300=48.92 total=8172635241
# full avg10=61.17 avg60=55.83 avg300=37.10 total=6019283746

cat /proc/pressure/cpu
# some avg10=2.41 avg60=1.88 avg300=1.02 total=91827364
# full avg10=0.00 avg60=0.00 avg300=0.00 total=0

cat /proc/pressure/memory
# some avg10=0.00 avg60=0.00 avg300=0.00 total=142
# full avg10=0.00 avg60=0.00 avg300=0.00 total=88

Here is how to read it.

• some is the fraction of time in which at least one task was stalled because of that resource.
• full is the fraction of time in which every runnable task was stalled at once. That is, during that time the system did no useful work at all.
• avg10, avg60, and avg300 are percentages over 10 second, 60 second, and 300 second windows, and total is cumulative microseconds.
• The full line for the system wide cpu is meaningless by definition and is always 0 (the line itself has been printed since kernel 5.13).

What the output above says is clear. I/O full at 61% means that for more than 6 of the last 10 seconds the whole system was stalled on I/O. That is far more direct information than the number 24, and above all it translates straight into lost throughput.

The second advantage of PSI is that it exists per cgroup. That is the attribution the load average does not have.

# Which cgroup was stalled on I/O the longest
for f in /sys/fs/cgroup/**/io.pressure; do
  v=$(awk '/^full/ {print $2}' "$f" | cut -d= -f2)
  printf '%6s  %s\n' "$v" "${f%/io.pressure}"
done 2>/dev/null | sort -rn | head -5
#  58.12  /sys/fs/cgroup/system.slice/postgresql.service
#   3.04  /sys/fs/cgroup/system.slice/containerd.service
#   0.00  /sys/fs/cgroup/user.slice

If /proc/pressure is missing, either CONFIG_PSI is off, or the kernel was built with CONFIG_PSI_DEFAULT_DISABLED=y and needs the boot parameter psi=1.

Wrapping up — load average is queue length, not utilization

If you remember only one thing, remember this. The Linux load average is an exponentially damped moving average of tasks waiting for CPU and tasks waiting for I/O summed together with no distinction. So this one number alone cannot answer whether you need more CPU.

Boiled down into practical rules, it comes out like this.

1. When the load is high, first look at the fourth field of /proc/loadavg and the b column of vmstat 1. The ratio of R to D sets the direction.
2. If D dominates, go down into the await values of iostat -x and into /proc/PID/stack. Adding CPU is not the answer.
3. Put the alert on PSI, not on load. cpu.some avg60 and io.full avg60 crossing a threshold correlates with real latency at far fewer false positives.
4. Leave load on the dashboard but use it only as a trend indicator. If the 1 minute value is above the 5 and 15 minute ones, load is rising, and read exactly that much and no more.
5. In a container environment, host load does not explain pod performance. You have to look at the cgroup cpu.stat and *.pressure.