CPU steal time and throttling — telling apart the st column in top, burstable credits, and CFS quota

Introduction — CPU is at 40% but p99 is 3 seconds

There is one sentence that comes up in capacity meetings more than any other. "CPU utilization is only 40%, so this is not a CPU problem." That sentence is wrong in three situations. And those three situations are what this whole article is about.

top -bn1 | head -4
# top - 14:22:31 up 9 days,  4:12,  1 user,  load average: 3.21, 3.04, 2.88
# Tasks: 214 total,   2 running, 212 sleeping,   0 stopped,   0 zombie
# %Cpu(s): 31.2 us,  4.1 sy,  0.0 ni, 42.6 id,  0.4 wa,  0.0 hi,  0.5 si, 21.2 st

The 21.2 st in the last column is where this article starts. Idle looks like 42.6%, but 21.2% is time that has already been taken away by someone else, and the CPU the application could have used shrank by exactly that much. And there is a third cause that does not appear here at all. Container throttling shows up nowhere on this screen.

What steal time means — the time the hypervisor did not give you

st is the share of time a vCPU was ready to run but waited because the hypervisor did not assign it a physical CPU. It is not something that happened inside the guest kernel but something that happened outside the guest, and the guest is merely notified after the fact.

The mechanism is a paravirtualized interface. On KVM the guest kernel registers a shared memory structure through MSR_KVM_STEAL_TIME, and the hypervisor writes into it the accumulated nanoseconds for which it scheduled the vCPU out. The guest kernel reads that value and reflects it in the 8th field of /proc/stat. Xen does the same thing with its runstate information.

head -2 /proc/stat
# cpu  9284712 1832 1204831 88213904 42193 0 18922 1928374 0 0
# cpu0 1160589  229  150604 11026738  5274 0  2365  241046 0 0
#      user    nice system idle    iowait irq softirq steal guest guest_nice

The eighth number is the accumulated steal ticks. top and mpstat only show you the delta of this value.

mpstat -P ALL 1 3 | tail -5
# 14:23:03  CPU   %usr  %nice  %sys %iowait  %irq %soft %steal %guest %gnice  %idle
# 14:23:03  all  30.51   0.00  4.02    0.38  0.00  0.51  21.34   0.00   0.00  43.24
# 14:23:03    0  29.90   0.00  3.96    0.50  0.00  0.99  22.28   0.00   0.00  42.37
# 14:23:03    1  31.14   0.00  4.08    0.26  0.00  0.03  20.40   0.00   0.00  44.09

If every CPU sits evenly near 21%, the whole host is oversubscribed; if it is concentrated on one particular vCPU, there is a competitor on the physical core that vCPU landed on. The latter is often resolved just by restarting the instance so it moves to a different host.

Here is the most common misunderstanding. A high st does not mean your process is using a lot of CPU. It is the opposite. It is time you wanted to use but could not, and the cause sits outside the instance. No amount of optimizing application code brings this number down.

The second misunderstanding matters just as much. An st of 0 does not mean there is no hypervisor contention. Steal accounting only works when the hypervisor exposes that information to the guest. Guests on VMware generally show st as 0, and the real contention is visible only in the host-side %RDY (CPU ready) metric. Concluding that "st is 0, so nothing is wrong" is the classic misdiagnosis in on-premises virtualization.

# Check whether the paravirtualized clock and steal accounting are active
grep -o 'kvm\|xen\|vmware\|hyperv' /sys/hypervisor/type 2>/dev/null
systemd-detect-virt
# kvm

grep -E 'CONFIG_PARAVIRT_TIME_ACCOUNTING' /boot/config-$(uname -r)
# CONFIG_PARAVIRT_TIME_ACCOUNTING=y

At what percentage does st become a problem, and what can you do

There is no absolute threshold, but the operational rules of thumb are fairly consistent.

• Under 1%: normal. On shared infrastructure a flat 0 is actually the rare case.
• 1~5%: worth watching. For a latency-sensitive service, p99 may already be affected.
• 5~10% sustained: worth acting on. Throughput really is cut by that much.
• Over 10% sustained: move immediately. No tuning works on that host.

What matters is not the instantaneous value but the persistence. Spiking for a few minutes because of a backup window or a neighbor batch job is common. Alerts should fire on 5-minute or 15-minute averages to cut down false positives.

There are four responses.

# 1) Relocate the instance — stop then start and it lands on a different physical host
#    (a reboot stays on the same host, so it has no effect)
aws ec2 stop-instances --instance-ids i-0abc123def4567890
aws ec2 start-instances --instance-ids i-0abc123def4567890

The difference between a reboot and a stop-then-start is the key point. A reboot from inside the guest does not change the physical host. Only a stop and a start through the cloud API makes placement happen again.

The other three are structural. Moving up to an instance size that occupies an entire physical core makes the neighbors disappear altogether. A dedicated host or a bare metal instance removes hypervisor sharing entirely, at a much higher cost. And separating latency-sensitive workloads onto instance families with low st is another practical option.

One trap. Adding vCPUs when st is high often backfires. If the host is already oversubscribed, the extra vCPUs simply have to wait for more scheduling opportunities, and inside the guest even spinlock contention increases. Raising the instance size, meaning the share of the host you occupy, is the right move rather than the core count.

Credit exhaustion on burstable instances — same symptom, different cause

This is the second cause. The AWS T family, and the equivalent tiers on other clouds, run on two concepts: baseline performance and CPU credits. Use less than the baseline and credits accumulate; use more than the baseline and credits are consumed.

One credit is enough to run a single vCPU at 100% performance for one minute. A t3.medium earns 24 per hour and has 2 vCPUs, so the sustainable usage is 2 times 20%, which comes out to 40% of a single vCPU. Keep using more than that and the balance is certain to reach 0.

The symptom is distinctive. Everything is fine for the first few hours, and then from some moment on, performance drops in a step and never recovers. The load is unchanged, yet response times multiply. In standard mode, once credits run out the hypervisor clamps you to baseline performance, and the guest perceives that as steal time. So st goes up, but the cause is not a neighbor, it is your own budget.

Telling them apart is simple. If steal rises and falls at random over time it is a neighbor; if steal climbs like a staircase and stays there from the moment the credit balance touches 0, it is credits.

aws cloudwatch get-metric-statistics \
  --namespace AWS/EC2 --metric-name CPUCreditBalance \
  --dimensions Name=InstanceId,Value=i-0abc123def4567890 \
  --start-time 2026-07-25T00:00:00Z --end-time 2026-07-26T00:00:00Z \
  --period 300 --statistics Average \
  --query 'sort_by(Datapoints, &Timestamp)[].[Timestamp,Average]' --output text | tail -6
# 2026-07-25T09:00:00Z	142.3
# 2026-07-25T10:00:00Z	88.1
# 2026-07-25T11:00:00Z	31.4
# 2026-07-25T12:00:00Z	0.0
# 2026-07-25T13:00:00Z	0.0
# 2026-07-25T14:00:00Z	0.0

It is 0 from 12:00 onward. If the latency graph turned bad from exactly that moment, the diagnosis is finished.

There are three options. Switch to unlimited mode and the instance keeps bursting even without credits, with the excess billed separately (the CPUSurplusCreditsCharged metric shows the actual cost). If you are continuously running above the baseline, moving to a fixed-performance instance in the M or C family ends up cheaper. If the bursts really are intermittent, keep the T family but put the credit balance on an alert.

# Switch to unlimited
aws ec2 modify-instance-credit-specification \
  --instance-credit-specifications \
  'InstanceId=i-0abc123def4567890,CpuCredits=unlimited'

The cost criterion is this. If overage charges in unlimited mode happen all the time, compare that amount against the price difference to the next fixed-performance instance up. In a permanently bursting state the latter is usually cheaper, and its performance is predictable.

Container CFS quota throttling — the third cause that top never shows

This is the most important section. The first two causes at least leave a trace called st. Throttling leaves no trace at all.

Here is how it works. When a cgroup has a CPU ceiling, the kernel hands out a quota every period, 100ms by default. If the tasks in that group use up the quota within the period, the kernel pulls the entire group off the run queue for the remaining time. Nothing runs until the next period begins.

Let us look at what happens in that state. A pod with a CPU limit of 1 core has 8 threads. If all 8 run at once, they burn through the 100ms quota in 12.5ms. For the remaining 87.5ms this pod is completely stopped. If a request happens to land in that window, 87.5ms is added to its response. And the average CPU utilization does not even look like 100%, meaning 1 core out of 1. It looks like something near 12.5%.

That is why you get a graph like this. Average CPU utilization is low, steal is 0, the load average is quiet, and only p99 spikes at regular intervals. A throttled task is taken off the run queue, so it is not caught by the load average either.

Checking takes a single file, cpu.stat.

# cgroup v2 (from inside the container; if the cgroup namespace is private you see it as is)
cat /sys/fs/cgroup/cpu.max
# 100000 100000

cat /sys/fs/cgroup/cpu.stat
# usage_usec 4128374621
# user_usec 3719283746
# system_usec 409090875
# nr_periods 86400
# nr_throttled 41287
# throttled_usec 2914837261

The two numbers in cpu.max are the quota and the period, in microseconds. 100000 100000 means a 100ms quota on a 100ms period, that is, 1 core. With no limit the first value is max.

There is one ratio you need to read.

throttling ratio = nr_throttled / nr_periods = 41287 / 86400 = 47.8%
average stall time per period = throttled_usec / nr_throttled = 2914837261 / 41287 = 70.6ms

Throttling occurred in half of all periods, and each time it occurred the group stopped for 70ms on average. That is where the p99 latency comes from. As a practical rule, above a 5% ratio it is worth investigating, and above 10% it is worth acting on.

On cgroup v1 the files are different.

cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us
# 100000
cat /sys/fs/cgroup/cpu/cpu.cfs_period_us
# 100000
cat /sys/fs/cgroup/cpu/cpu.stat
# nr_periods 86400
# nr_throttled 41287
# throttled_time 2914837261000

throttled_time on v1 is in nanoseconds and throttled_usec on v2 is in microseconds. That is a factor of 1000, and it is the spot people get wrong most often when porting a dashboard.

To sweep per pod from the host, do this.

for f in /sys/fs/cgroup/kubepods.slice/*/*/cpu.stat; do
  awk -v p="${f%/cpu.stat}" '
    /^nr_periods/  {np=$2}
    /^nr_throttled/{nt=$2}
    END {if (np > 0 && nt/np > 0.05) printf "%5.1f%%  %s\n", nt*100/np, p}
  ' "$f"
done | sort -rn | head -5
#  47.8%  /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a1c8f2e.slice
#  12.3%  /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d22b71a.slice

In Prometheus you use the cAdvisor metrics.

sum by (pod) (rate(container_cpu_cfs_throttled_periods_total[5m]))
  / sum by (pod) (rate(container_cpu_cfs_periods_total[5m]))

I often see dashboards that watch only container_cpu_cfs_throttled_seconds_total, but that value answers "how long was it stopped" when what you actually need is "how many periods out of how many were stopped". You have to chart both.

The kernel version is another value to check. Kernels before 5.4 had a bug in CPU slice expiration handling that caused throttling even when the quota had not been used up. That expiration logic was removed in 5.4 and the symptom dropped sharply. If you see unexplained throttling on an old node, check the kernel first.

uname -r
# 6.8.0-45-generic

From 5.14 onward, cpu.max.burst was added, so unused quota can be banked up to a limit and then spent above the ceiling for a moment. It is very effective for short, spiky workloads.

# Allow banking up to 50ms
echo 50000 | sudo tee /sys/fs/cgroup/kubepods.slice/.../cpu.max.burst

A table that separates the three causes, and the Kubernetes CPU limit debate

Here are the three causes so far in a single table. The symptoms look similar, but the diagnostic point and the response are entirely different.

The "load average does not rise" in the third row is especially counterintuitive. A throttled task is removed from the run queue, so it is never caught in the queue. Load, CPU utilization, and steal are all quiet, and only the service is slow. When you see that combination, look at cpu.stat.

Now for the debate. There is an argument for removing CPU limits in Kubernetes altogether. You need to look at both sides precisely.

The argument for removal. CPU is a compressible resource. Unlike memory, a shortage does not kill anything, it only makes things slower. And request is already translated into cpu.weight (cpu.shares on v1), which guarantees a proportional share under contention. In other words, when the node is busy you split it by the request ratio, and when the node is idle, with no limit you can simply use the CPU that is left over. With a limit you are forcibly stopped even when the node has CPU to spare. That is pure waste.

The argument for keeping them. Without a limit, one runaway pod really does slow down its neighbors on the same node. Weight guarantees only a minimum share under contention and sets no ceiling, so a pod that creates hundreds of threads degrades the effective performance of its neighbors through context switches and cache pollution. Also, without a limit the performance characteristics shift with how much room the node has, which destroys the reproducibility of capacity planning and load test results. On a multi-tenant cluster, isolation is not negotiable. And to be given dedicated cores by the static policy of the CPU manager you need Guaranteed QoS, meaning limit has to equal request.

The balance point can be summarized as follows.

1. Always set request. This is not up for debate. Both scheduling placement and the share under contention come from here.
2. The candidates for removing a limit are latency-sensitive, trustworthy first-party services. Remove it, but you have to watch node-wide CPU utilization and neighbor pod latency at the same time.
3. The ones to keep a limit on are batch jobs, low-trust workloads, and multi-tenant namespaces. Here predictability matters more than efficiency.
4. If you keep a limit, size it to the parallelism. A limit of 1 core with 8 threads structurally produces throttling.
5. Tell the runtime about the ceiling. As a single action this has the biggest effect of all.

Let me expand on number 5. The JVM and the Go runtime look at the total core count of the host by default. Inside a container with a CPU limit of 1 on a 64-core node, a Go program sets GOMAXPROCS to 64, and 64 threads share a quota worth 1 core and hit throttling immediately.

# Go: use an automatic tuning library, or set the environment variable directly
GOMAXPROCS=1 ./myapp

# JVM: container awareness is on by default, but it needs verifying
java -XX:+PrintFlagsFinal -version | grep -E 'ActiveProcessorCount|UseContainerSupport'
#      intx ActiveProcessorCount   = -1
#      bool UseContainerSupport    = true

# Set it explicitly
java -XX:ActiveProcessorCount=2 -jar app.jar

The container awareness of the JVM rounds up when converting a CPU limit into a core count. A limit of 1.5 cores is seen as 2. If you use fractional limits, check whether this rounding is what creates the throttling.

There are two node-level options as well. The kubelet flag --cpu-manager-policy=static assigns dedicated cores exclusively to Guaranteed pods that requested whole CPUs, bypassing the CFS quota entirely. Reducing --cpu-cfs-quota-period from 100ms to around 10ms shortens each individual stall, which lowers the amplitude of latency spikes. That said, a shorter period increases scheduling overhead, and this flag is not a stable feature yet, so verify it before applying it.

Preventing recurrence — what to put on the dashboard

Each of the three causes is caught by a different metric, so you have to chart all three. Leave out even one and that cause stays invisible forever.

# 1) steal — per node, 5-minute average
avg by (instance) (rate(node_cpu_seconds_total{mode="steal"}[5m])) > 0.05

# 2) throttling ratio — per pod
sum by (namespace, pod) (rate(container_cpu_cfs_throttled_periods_total[5m]))
  / sum by (namespace, pod) (rate(container_cpu_cfs_periods_total[5m])) > 0.10

# 3) CPU pressure — PSI. Catches both throttling and contention
avg by (instance) (rate(node_pressure_cpu_waiting_seconds_total[5m])) > 0.20

There is a reason to chart PSI alongside them. cpu.pressure directly measures "time that could have been spent running but was not", so it catches the outcome regardless of whether the cause is steal, throttling, or run queue contention. It exists per cgroup too.

cat /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a1c8f2e.slice/cpu.pressure
# some avg10=68.42 avg60=61.03 avg300=44.18 total=182937461283
# full avg10=52.11 avg60=47.88 avg300=33.02 total=91827364512

some at 68% means that out of the last 10 seconds, some task in this pod waited for CPU for 6.8 of them. Read alongside the 47.8% throttling ratio, you are confirming the same fact from two angles.

Leaving it behind as a routine check script means you do not have to dig through your memory during an incident.

#!/usr/bin/env bash
# cpu-triage.sh — check the three causes of CPU latency in one pass
set -u

echo "== 1. steal time (5-second average) =="
mpstat 5 1 | awk '/Average/ {printf "  %%steal = %s\n", $(NF-3)}'

echo "== 2. virtualization environment =="
printf '  '; systemd-detect-virt

echo "== 3. cgroup throttling (v2) =="
if [ -r /sys/fs/cgroup/cpu.stat ]; then
  awk '/^nr_periods/{np=$2} /^nr_throttled/{nt=$2} /^throttled_usec/{tu=$2}
       END {
         if (np > 0)
           printf "  ratio %.1f%%  (%d/%d), average %.1fms when it happens\n",
                  nt*100/np, nt, np, (nt>0 ? tu/nt/1000 : 0)
       }' /sys/fs/cgroup/cpu.stat
  printf '  cpu.max = %s\n' "$(cat /sys/fs/cgroup/cpu.max 2>/dev/null)"
fi

echo "== 4. CPU pressure (PSI) =="
[ -r /proc/pressure/cpu ] && sed 's/^/  /' /proc/pressure/cpu

Wrapping up — low utilization does not mean the CPU is idle

If you remember one thing, remember this. CPU utilization tells you how much was used, not how much could have been used. The time you wanted to use but could not shows up in three forms, steal, credit exhaustion, and throttling, and of those, throttling leaves no trace anywhere on the utilization graph.

Compressed into a decision order, it goes like this.

1. Look at %steal first with mpstat -P ALL 1. Above a 5-minute average of 5% it is an instance problem, and there is nothing you can do from inside the guest.
2. If steal climbed like a staircase from a particular moment and stayed there, check the credit balance. It is a budget problem, not a neighbor.
3. If steal is 0 but latency spikes, look at nr_throttled in cpu.stat. Utilization and load average cannot show you this problem.
4. Once throttling is confirmed, match the parallelism of the runtime before raising the limit. Adjusting GOMAXPROCS or ActiveProcessorCount usually has a bigger effect.
5. Removing CPU limits is not a universal cure. Setting request accurately comes first, and removal should be confined to trustworthy workloads and carried out together with observation.
6. Alert on all three, steal, throttling ratio, and PSI. The three causes do not cover for one another.