A Complete Guide to Linux Performance Tools: How to Read the Numbers in the Output

• Introduction
• 1. The Order in Which to Pick a Tool — The USE Method
• 2. top and htop — Three Places That Create an Illusion
• 3. vmstat — Summarizing the Whole System in One Line
• 4. mpstat and pidstat — Two Tools That Narrow the Scope
• 5. iostat — Look at await and Be Suspicious of %util
• 6. sar — Rewinding Time That Has Already Passed
• 7. Network — Throughput, Queues, Retransmits
• 8. perf — Which Function Is Spending the Time
• 9. Tool Selection Summary Table
• Quiz: check your understanding
• Closing
• References
• Further reading

Introduction

Learning performance tools and reading the output of performance tools are two different things. Plenty of people know how to type iostat -x, but few can answer precisely, when they see %util at 100 percent, "so is this disk saturated?" That holds even though the man page states outright that the value does not mean saturation on a modern SSD.

This blog already has a Linux performance engineering guide, and that article covers profiling techniques, eBPF, flame graphs and kernel tuning broadly. This article digs into the point where it does not overlap. Its subject is what every single column each tool prints actually means, and which values create an illusion. This is not an article that introduces new tools; it is an article that keeps you from misreading the output of tools you already know.

The reference environment is sysstat 12 or later and kernel 5.x or later. Column names change between sysstat versions (for example, the old avgqu-sz became aqu-sz), so if your output differs, check the man page for the version you have installed.

1. The Order in Which to Pick a Tool — The USE Method

The easiest way to break the habit of firing up top first is the USE checklist. For every resource, you ask three questions: Utilization, Saturation, Errors.

The value of this table lies in the saturation column. If you look only at utilization, you conclude "CPU is at 60 percent, so we have headroom", but if the run queue is three times the core count, delay is already happening. Utilization is an average; saturation is waiting. What users feel is the waiting side.

2. top and htop — Three Places That Create an Illusion

top is the first tool you bring up, and also the easiest one to misread.

top -b -n 1 | head -20
top -H -p 1234
top -o %MEM

-b is batch mode, which is good for logging, -H displays per thread, and -o sets the sort key.

Illusion 1: %CPU is measured against a single core. On an 8-core system one process can show 800 percent. There is no reason to be alarmed by values above 100 percent. Pressing Shift+I while top is running turns off Irix mode and switches to values based on all cores.

Illusion 2: VIRT is reserved virtual address space. The Java and Go runtimes map large regions in advance that they never actually use. Memory judgments should be based on RES, or more accurately on PSS.

Illusion 3: the %CPU on the first screen is a cumulative value since boot. The value you capture the instant you open top is meaningless; you have to read from at least the second refresh.

To see memory accurately per process, PSS, which apportions shared pages, is better.

sudo grep -H '^Pss:' /proc/1234/smaps_rollup

RSS is the total amount of pages that process has resident in physical memory, but it counts library pages shared by several processes in full for each of them. That is why simply summing RSS on a web server running 20 worker processes gives a figure several times the real memory usage. PSS divides shared pages by the number of referencing processes and distributes them, so the sum of PSS is far closer to actual usage. When sizing capacity or setting container memory limits, always judge on a PSS basis.

If you use htop, it is a good idea to turn on thread display and tree display in the display options. In particular, when you run it inside a container you see only the processes in that namespace rather than the whole host, so when judging resource contention from the host's point of view you have to check again on the host.

3. vmstat — Summarizing the Whole System in One Line

vmstat 1 has the highest information density among performance tools. As the man page states, the first report is the average since the last reboot, so always read from the second line onward.

vmstat -w 1 10
vmstat -s
vmstat -d

-w gives wide output, -s gives the event counter table (no repetition), and -d gives disk statistics.

The reading rules come out like this.

• If r is consistently and substantially larger than the core count, that is CPU saturation. Momentary spikes are normal.
• If b is not zero, it means there are processes waiting for I/O to complete.
• If si/so are not zero, swapping is actually happening. Even if swpd is large, when si/so are zero it just means pages pushed out in the past are still there and it is not a current problem. This distinction matters.
• bi/bo are block device reads and writes (KiB/s).
• If cs (context switches) is abnormally large, suspect lock contention or excessive threads.
• If in (interrupts) is large, look at network traffic or timers.
• st is time stolen by the hypervisor. It is a problem you cannot solve from inside the guest, so if this value stays high, changing the instance type or migrating is the answer.

The ratio of us to sy is useful too. If sy is larger than us, it is a sign that time is going into system calls, context switches and interrupt handling rather than application logic.

4. mpstat and pidstat — Two Tools That Narrow the Scope

mpstat breaks things down per core. This tool exists to expose the imbalance that an average hides.

mpstat -P ALL 1 5

If one of eight cores is at 100 percent, the overall average is 12.5 percent and looks idle, but in reality it is a single-threaded bottleneck. The case where interrupts pile onto one core also shows up here (the %irq and %soft columns).

pidstat gives per-process interval statistics. Unlike top, it keeps printing without clearing the screen, which makes it suitable for writing to a file.

pidstat -u 1 5
pidstat -r 1 5
pidstat -d 1 5
pidstat -w 1 5
pidstat -t -p 1234 1 5

The columns for each mode are as follows.

• -u: %usr, %system, %guest, %wait, %CPU, CPU. Here %wait is "the share of time the process was ready to run but waiting on a CPU". When this value is high the process is not slow — CPU contention is severe, so the response is completely different.
• -r: minflt/s, majflt/s, VSZ, RSS, %MEM. majflt/s (major faults) is the key one. It counts how often a page had to be read from disk, so if it is not zero, memory is short or the file cache is being pushed out. Minor faults occur constantly under normal operation.
• -d: kB_rd/s, kB_wr/s, kB_ccwr/s, iodelay. kB_ccwr/s is the amount of writes that were cancelled, which shows up when a file is written and then deleted, among other cases.
• -w: context switch statistics. Voluntary switches (cswch/s) are due to waiting, and involuntary switches (nvcswch/s) are due to exhausting the time slice. Many of the latter means CPU contention.
• -t: breaks output down per thread.

5. iostat — Look at await and Be Suspicious of %util

iostat -xz 1 5
iostat -xmdz -p ALL 1 3

-x extended statistics, -z skip inactive devices, -m MiB units, -p ALL down to partitions, -y skip the first report.

The definitions of the key columns are as follows.

• r/s, w/s: the number of read and write requests completed per second after merging.
• rrqm/s, wrqm/s: the number of requests merged per second. A large value means the kernel is doing a good job of coalescing sequential I/O.
• r_await, w_await: average service time in milliseconds, including queue wait time. This is the closest to what the user perceives.
• aqu-sz: average queue length.
• rareq-sz, wareq-sz: average request size in KiB. Around 4KiB means random I/O; hundreds of KiB means sequential I/O.
• %util: the share of time during which requests were issued to the device.

The man page's warning about %util is worth quoting as is. On devices that serve requests serially, close to 100 percent means saturation, but on devices that process in parallel, such as RAID arrays or modern SSDs, this value does not reflect a performance ceiling. NVMe runs dozens of queues at the same time, so there is plenty of headroom even at %util 100 percent.

It is safer to set your practical criteria like this.

In cloud environments the last item is especially common. When burst credits run out, IOPS suddenly drops to the baseline, so the request count is low while responses are slow.

6. sar — Rewinding Time That Has Already Passed

Half of all performance problems are already over by the time a human logs in. sar is the tool for rewinding to that moment.

sar -u -s 03:00:00 -e 04:00:00
sar -r
sar -q
sar -b
sar -n DEV
sar -n EDEV
sar -W
sar -f /var/log/sa/sa14 -u ALL

• -u CPU, -r memory, -q run queue and load, -b I/O transfer rates, -n DEV interface throughput, -n EDEV interface errors, -W swapping statistics.
• Use -s/-e to specify a time range and -f to specify the per-day file.

The collection file path differs by distribution. On the RHEL family it is /var/log/sa/, and on the Debian/Ubuntu family it is /var/log/sysstat/. On the Debian family you have to enable collection in /etc/default/sysstat before data starts accumulating. Collection interval and retention period are adjusted in /etc/sysstat/sysstat (Debian) or /etc/sysconfig/sysstat (RHEL).

The default 10-minute interval misses momentary spikes. If you have to catch second-scale events, reduce the collection interval to one minute or attach a separate continuous collector. Bear in mind that disk usage and collection overhead go up in exchange.

7. Network — Throughput, Queues, Retransmits

Throughput comes from sar -n DEV, errors from sar -n EDEV, socket state from ss, and kernel counters from nstat.

sar -n DEV 1 5
ip -s link show eth0
nstat -az | grep -i -E 'retrans|listen|prune|collapse'
ss -tin state established | head -20

ss -i shows per-socket TCP internals. From it you can read rtt, cwnd and retrans values. If RTT is large and cwnd is small, throughput will not materialize even with bandwidth to spare. This is the classic case of the bandwidth-delay product being trapped by the window size.

Among the nstat counters, the ones used most often in practice are these.

• TcpExtListenOverflows, TcpExtListenDrops: the accept queue overflowed. Either the application is failing to keep up with accept or the backlog is too small.
• TcpRetransSegs: retransmits. It is only meaningful as a ratio against total transmissions.
• TcpExtTCPRcvCollapsed, TcpExtPruneCalled: receive buffer pressure.

Interface-level drops are visible in RX errors/dropped from ip -s link. When these climb, you need to look at the NIC or the ring buffer rather than at anything above the kernel.

8. perf — Which Function Is Spending the Time

If you have come this far and are thinking "I can see it is using CPU, but I do not know what it is doing inside", it is perf time.

sudo perf top
sudo perf stat -d -p 1234 -- sleep 10
sudo perf record -F 99 -g -p 1234 -- sleep 30
sudo perf report --stdio | head -40

• perf top: shows in real time the symbols using the most CPU.
• perf stat: summarizes hardware counters over the interval a command runs. Look at IPC (instructions per cycle), cache misses, and branch mispredictions. An IPC well below 1 is a strong signal that the CPU is waiting on memory.
• perf record -g: writes samples to a file including call stacks. -F 99 samples 99 times per second; 99 is used instead of 100 to avoid falling in step with timers.

If perf does not work in a container or in the cloud, it may be because of the kernel's observability permission level.

cat /proc/sys/kernel/perf_event_paranoid

The value has to be 2 or lower for ordinary profiling to be possible, and containers need separate privileges granted. Lowering this value on a production server is a security decision, so do not change it on your own; handle it according to policy. Check the exact permission requirements in the documentation for the distribution and kernel version you are running.

If symbols show up only as question marks, the debug symbol package is missing. The RHEL family needs the debuginfo package and the Debian family needs the dbgsym repository added.

9. Tool Selection Summary Table

This is a table for choosing a tool starting from a symptom. In practice, working down this order yields the answer most of the time.

Better than pinning this table to the wall is adding a line for every incident your team has actually experienced. Every organization hits different bottlenecks regularly, so that list ends up far more accurate.

The last piece most often missing from measurement is something to compare against. Knowing only that response time is 200 milliseconds right now tells you nothing about whether that is good or bad. If the usual value was 40 milliseconds this is serious; if the usual value was 220 milliseconds, nothing is happening. That is why the first thing to do when introducing performance tools is not to set thresholds but to record normal values for a few days and build a baseline. Without a baseline every number stays uninterpretable.

Container environments need one more layer. The free or nproc values you see inside a container often report the host's values directly, while the actual limits are enforced by cgroups. Reading the limits and actual usage directly from cgroup v2 files is the reliable approach.

cat /sys/fs/cgroup/memory.max
cat /sys/fs/cgroup/memory.current
cat /sys/fs/cgroup/cpu.max
cat /sys/fs/cgroup/cpu.stat

If the throttling counters in cpu.stat keep increasing, the container is periodically stalling even when the host CPU is idle. In that case the CPU utilization observed on the host comes out low, so you can never find the cause of latency inside the container from host metrics alone.

Quiz: check your understanding

mpstat -P ALL 1 5
pidstat -u 1 5

iostat -xz 1 5

mpstat -P ALL 1 5
pidstat -t -p <PID> 1 5

sudo perf stat -d -p 1234 -- sleep 10

sar -q -s 02:50:00 -e 03:30:00
sar -u -s 02:50:00 -e 03:30:00
sar -r -s 02:50:00 -e 03:30:00
sar -n DEV -s 02:50:00 -e 03:30:00
journalctl --since '2026-08-15 02:50' --until '2026-08-15 03:30' -p warning

Closing

The output of a performance tool tells you facts, but it does not tell you conclusions. %util at 100 percent is a fact; "the disk is saturated" is an interpretation. Knowing that this interpretation can be wrong depending on the device type is what it means to be able to use the tool.

Here are three principles worth remembering. First, look at saturation before utilization. Second, split the imbalance an average hides apart per core and per process. Third, continuous collection has to be on, because no tool is any use if you cannot look at time that has already passed.

References

• vmstat(8) — man7.org (verified 2026-08-15)
• iostat(1) — man7.org (verified 2026-08-15)
• pidstat(1) — man7.org (verified 2026-08-15)
• ss(8) — man7.org (verified 2026-08-15)
• dmesg(1) — man7.org (verified 2026-08-15)

Further reading

• Previous: A Complete Guide to Linux Incident Response Commands
• Next: A Complete Guide to Processes and Signals
• A Complete Guide to Linux Performance Engineering — all the way through eBPF and flame graphs
• A Guide to Tuning Linux Kernel Parameters — the adjustment that comes after measurement
• Linux Terminal — practice commands in the browser
• Linux Command Quiz — check how well you remember the options