Split View: 리눅스 장애 대응 명령어 완전 가이드: 첫 60초부터 근본 원인까지

The complete Linux troubleshooting command guide: from the first 60 seconds to root cause

• Introduction
• 1. The first 60 seconds — the standard triage sequence
• 2. What load average actually tells you
• 3. Who is using the CPU
• 4. Memory — which number in free do you believe
• 5. When the disk is full — capacity and inodes
• 6. I/O — finding the evidence of a slow disk
• 7. Network — ports, queues, and retransmits
• 8. Logs — narrow the time window first
• 9. Digging into a process that looks stuck
• 10. Always-on collection for incidents that will not reproduce
• Quiz: check your understanding
• Closing
• References
• Further reading

Introduction

The biggest waste of time during an incident is not caused by not knowing a command. It is caused by not knowing which command to run in which order. You bring up top, stare at it for 30 seconds, run df, come back to top — and five minutes are gone.

This post lays out, in order, the commands to run from the moment you get the call that says "the server is slow", "it stopped responding", or "something is off since the deploy". This blog already has a Linux performance engineering guide, but that one covers profiling and tuning methodology. This one is different. The subject here is the order in which you eliminate candidates one at a time while you still do not know what the problem is.

There is a single premise. Diagnosis is not about finding the culprit, it is about eliminating suspects. If it is not CPU, erase CPU. If it is not memory, erase memory. What remains is the answer.

Distribution notation is as follows. RHEL-family means RHEL/Rocky/AlmaLinux 8 and later; Debian-family means Debian 12 and Ubuntu 22.04 and later.

1. The first 60 seconds — the standard triage sequence

The "first 60 seconds" checklist published by the Netflix performance engineering team has effectively become the industry standard. The key is to fire ten commands at once and look at the whole picture first, and only then dig deep.

uptime
dmesg --level=err,warn --ctime | tail -20
vmstat 1 5
mpstat -P ALL 1 3
pidstat 1 3
iostat -xz 1 3
free -m
sar -n DEV 1 3
ss -s

Here is the suspect each command rules out.

mpstat, pidstat, iostat, and sar all live in the sysstat package. It is frequently not part of a default install, so put it on the image when you build the server.

# RHEL family
sudo dnf install -y sysstat
# Debian family
sudo apt install -y sysstat

To read historical data with sar, the collector daemon has to be running. On Debian-family systems, collection only starts once you turn the enable value on in /etc/default/sysstat.

sudo systemctl enable --now sysstat

2. What load average actually tells you

The three numbers from uptime are the 1-minute, 5-minute, and 15-minute load averages. Two things here are easy to misread.

First, the Linux load average does not count CPU wait alone. It includes processes in uninterruptible sleep, the D state, as well as runnable (R) ones. D state is usually disk I/O or a wait on an NFS response. That is how you end up with a load of 40 while the CPU sits idle. In that case the culprit is storage, not CPU.

Second, the slope matters more than the absolute value. If the 1-minute figure is larger than the 15-minute figure, things are getting worse right now; if it is smaller, you are already recovering. In the latter case there is no reason to rush a restart.

uptime

 14:22:31 up 41 days,  3:11,  2 users,  load average: 12.44, 6.80, 3.15

This output means "load quadrupled over the last 15 minutes and is still climbing". It only means something relative to the core count, so check the core count first.

nproc
lscpu | grep -E '^CPU\(s\)|Thread|Core|Socket|Model name'

3. Who is using the CPU

The first line of vmstat 1 is the average since boot, so ignore it and start reading at the second line. This behaviour is stated in the man page.

vmstat 1 5

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 8  1      0 210488  91232 3120440    0    0    12    34  980 2210 71 14  9  6  0

Read it in this order.

• r: the number of processes waiting to run. If it stays consistently above the core count, the CPU is saturated.
• b: the number of processes blocked waiting for I/O to complete. When this is large, suspect storage.
• si/so: swap in and out. Anything other than zero means memory pressure is already costing you performance.
• wa: the share of time spent waiting on I/O. High means disk; low but still slow means it is not the disk.
• st: time stolen by the hypervisor. On a cloud VM, a value that keeps exceeding 5 means the host is oversubscribed, and there is nothing you can solve from inside your own server.

Whether everything is busy or just one core is settled with mpstat.

mpstat -P ALL 1 3

If exactly one core sits at 100 percent, you have a single-threaded bottleneck. Scaling up will not fix it; you have to look at the code or the configuration.

Find the guilty process with pidstat. Unlike top, it keeps printing per-interval values rather than cumulative ones, which makes it good for capturing into a log.

pidstat -u 1 5
pidstat -t -p 1234 1 5

The -t option breaks the numbers down per thread. For thread-heavy processes such as Java or Go, you can see which thread is spinning. %wait is the share of time a task was ready to run but could not get a CPU, so when that is high the process is not slow, CPU contention is severe.

The classic ps combination is still useful too.

ps -eo pid,ppid,stat,pcpu,pmem,etime,rss,args --sort=-pcpu | head -15

In the stat column, D means uninterruptible sleep, Z is a zombie, and T is stopped.

4. Memory — which number in free do you believe

The column beginners misread most often in free -m is the free column. Linux uses spare memory as page cache. A small free is normal, and the column to look at is available.

free -m

               total        used        free      shared  buff/cache   available
Mem:           15884       10233         412         188        5238        5102
Swap:           4095         120        3975

available is the kernel estimate of "how much could be reclaimed and handed over if a new process asked for it right now". If that value is a healthy share of the total, you are not short on memory.

If you suspect memory pressure, start by checking what the kernel killed.

dmesg --ctime | grep -i -E 'out of memory|oom-kill|killed process'
journalctl -k --since '2 hours ago' | grep -i oom

dmesg --ctime (-T) converts timestamps to a human-readable clock, but the man page warns that timestamps can be inaccurate after a system suspend or resume. When you need precise times, use journalctl -k.

For a detailed breakdown, look at /proc/meminfo.

grep -E 'MemAvailable|Dirty|Writeback|Slab|SReclaimable|Committed_AS' /proc/meminfo

If Dirty is large and not shrinking, writes are failing to make it down to disk. If Slab is abnormally large, suspect a kernel object leak. For real per-process usage, RSS double-counts shared pages, so read PSS where you can.

sudo grep -H '^Pss:' /proc/1234/smaps_rollup

5. When the disk is full — capacity and inodes

Disk exhaustion comes in two kinds: block exhaustion and inode exhaustion. Looking only at df -h, answering "there is plenty free", and then discovering the inodes are actually gone is not a rare experience.

df -h
df -i

When IUse% in df -i reads 100 percent, you cannot create a new file no matter how much space is left. The cause is usually a directory where millions of small files pile up, such as session files or a mail queue.

When hunting for large directories, it is safer to restrict the recursion to a single filesystem.

sudo du -x -h --max-depth=1 /var | sort -h | tail -20

-x keeps it from crossing into other filesystems. Without it, you end up walking /proc and network mounts and it takes forever.

If df and du disagree badly, the reason is files that were deleted but are still open. Deleting a file does not return the space while a process that has it open is still alive.

sudo lsof +L1
sudo lsof -nP +L1 | awk '{print $1, $2, $7, $9}' | sort -k3 -n -r | head

+L1 lists open files whose link count is below 1, that is, files that have already been unlinked. The fix is to send the process a log-reopen signal or restart it. The mechanics are covered in this series in the file descriptors and inodes guide.

6. I/O — finding the evidence of a slow disk

iostat -xz 1 3

-x gives extended statistics and -z omits devices with no activity. The first report covers the period since boot, so drop it with -y or start reading at the second report.

The columns to read are these.

• r_await / w_await: average response time in milliseconds, including the time a request waited in the queue. On NVMe, anything above single-digit milliseconds is odd; on spinning disks, 10 to 20 is still a normal range.
• aqu-sz: average queue length. Well above 1 means the device is not digesting requests.
• rareq-sz / wareq-sz: average request size in KiB. Tells small random I/O apart from large sequential I/O.
• %util: the share of time during which I/O was issued to this device.

%util needs care. The man page warns about it explicitly. On a device that serves requests serially, 100 percent means saturation, but on devices that process in parallel, such as a RAID array or a modern SSD, 100 percent is not the performance ceiling. In that case the number you judge by is await, not %util.

Which process is generating the I/O is answered by pidstat -d.

pidstat -d 1 5

kB_rd/s and kB_wr/s are per-process reads and writes, and iodelay shows block I/O delay in clock ticks. If iotop is installed, it is more comfortable to watch interactively.

sudo iotop -oPa

7. Network — ports, queues, and retransmits

netstat is a relic of the net-tools package, and the standard on current distributions is ss.

ss -tulpn
ss -tan state established | head
ss -s

• -t TCP, -u UDP, -l listening only, -a everything, -n no name resolution, -p show the process using the socket.
• Seeing process names with -p usually requires root.

Recv-Q and Send-Q mean different things depending on the socket state. On a listening socket, Recv-Q is the length of the completed queue waiting to be accepted, and Send-Q is the backlog maximum. If Recv-Q on a listening socket stays full, the application is failing to keep up with accept. On an established connection they are, respectively, received data not yet read and sent data not yet acknowledged. Note, though, that the meaning of these two columns is not spelled out in the body of the ss manual published on man7.org; it rests on iproute2 implementation behaviour. Before you lean on it as evidence, it is safer to confirm it once against the man page of the iproute2 version you have installed.

Whether this is a path problem or an application problem gets settled by splitting it up by layer.

ip -brief addr
ip route get 10.0.3.14
ping -c 4 10.0.3.14
mtr -rwc 20 10.0.3.14
curl -sS -o /dev/null -w 'dns:%{time_namelookup} conn:%{time_connect} tls:%{time_appconnect} ttfb:%{time_starttransfer} total:%{time_total}\n' https://example.com

The curl timing variables show which stage is slow in a single line. A slow DNS stage is a name resolution problem, a slow time_connect is the path or a firewall, and a slow time_starttransfer on its own is the server application.

Check whether TCP retransmits are high.

nstat -az | grep -i -E 'retrans|drop|overflow'
ss -ti state established | grep -E 'retrans|rtt' | head

If ListenOverflows or ListenDrops is climbing, the backlog is overflowing.

8. Logs — narrow the time window first

Do not read logs from the beginning. Look at only the five minutes either side of the incident.

journalctl --since '2026-08-15 14:10' --until '2026-08-15 14:30' -p err
journalctl -u nginx.service -n 200 --no-pager
journalctl -u nginx.service -f
journalctl -k -b -1 -p warning
journalctl -g 'timeout|refused|denied' --since today

• -p err shows only err priority and above. The priority names, in order, are emerg, alert, crit, err, warning, notice, info, debug.
• -b -1 is the log from the previous boot. When a server reboots out of nowhere, the cause is in there.
• -g applies a regular expression to the MESSAGE field.

Listing boots and checking the reason for a reboot goes like this.

journalctl --list-boots
last -x reboot shutdown | head

If last -x shows reboots with no clean shutdown record, suspect a kernel panic, a power problem, or a forced hypervisor restart. Designing the log pipeline itself is covered in the Linux logging operations guide.

9. Digging into a process that looks stuck

A process that stops responding is one of three things: burning CPU, waiting on something, or stuck on a lock.

Start with the wait point the kernel reports. These are reads with no side effects.

cat /proc/1234/status | grep -E 'State|Threads|voluntary'
sudo cat /proc/1234/stack
sudo cat /proc/1234/wchan; echo
ls -l /proc/1234/fd | head

A State of D (disk sleep) is uninterruptible sleep. A process in that state will not die even to SIGKILL. It only unblocks when the resource it is waiting on, usually storage or NFS, responds.

If you need to look at the syscall level, attach strace, but do so knowing full well that it makes the process substantially slower in production, and keep it short.

sudo strace -f -p 1234 -tt -T -e trace=network,file 2>&1 | head -50
sudo strace -c -f -p 1234

-c emits only a summary, so its overhead is comparatively low, and it shows at a glance which syscall is eating the time. If strace will not attach, the kernel ptrace restriction setting may be the reason.

cat /proc/sys/kernel/yama/ptrace_scope

A value of 1 or higher means even the same user cannot attach to an arbitrary process, and root is required.

Being stuck on a file lock is also common.

cat /proc/locks | head
sudo lsof /var/lib/myapp/data.db

10. Always-on collection for incidents that will not reproduce

The most common failure is "we did not capture the logs at the time". A momentary load spike is over before a human can log in, so collection has to already be running under normal conditions.

sar was built for exactly this. The default collection interval varies by distribution configuration and is usually 10 minutes.

sar -u -f /var/log/sa/sa15
sar -r -s 14:00:00 -e 15:00:00
sar -n DEV -s 14:00:00 -e 15:00:00
sar -q

-f points at a specific day file. The file path differs: /var/log/sa/ on RHEL-family, /var/log/sysstat/ on Debian-family.

A simple approach that snapshots only when a threshold is crossed works too. The following writes a process list to a file when load goes over a threshold.

#!/usr/bin/env bash
set -euo pipefail
THRESHOLD=20
LOAD=$(awk '{print int($1)}' /proc/loadavg)
if [ "$LOAD" -ge "$THRESHOLD" ]; then
  TS=$(date +%Y%m%d-%H%M%S)
  OUT="/var/log/spike-$TS.txt"
  {
    uptime
    ps -eo pid,stat,pcpu,pmem,etime,args --sort=-pcpu | head -30
    ss -s
    vmstat 1 3
  } > "$OUT"
fi

Hang that script off a one-minute cron job or a systemd timer, and the next load spike leaves evidence behind even with nobody there. For how to write timers, see systemd timers, fully explained.

Finally, take care that the commands you run while diagnosing do not make the incident worse. Running du -x from the root while forgetting the flag, or launching a full find / scan against a disk whose I/O is already saturated, makes the situation worse. Remember that when a disk is already 100 percent utilised, read work has to queue too.

Quiz: check your understanding

ps -eo pid,stat,wchan:20,args --sort=-pcpu | awk '$2 ~ /D/'
iostat -xz 1 3
vmstat 1 5

df -i
df -h /var/lib/myapp
mount | grep myapp

sudo lsof -nP +L1 | head
sudo systemctl reload rsyslog

ps -o pid,ppid,stat,wchan:24,args -p 1234
sudo cat /proc/1234/stack

curl -sS -o /dev/null -w 'dns:%{time_namelookup} conn:%{time_connect} tls:%{time_appconnect} ttfb:%{time_starttransfer} total:%{time_total}\n' https://api.example.com/health

Closing

The difference between people who are good at incident response and people who are not is not the number of commands they know. It is whether, every single time, they are conscious of what the result of this command lets them rule out.

If you decide in advance, before typing vmstat, that "if wa is low here I erase the disk", then the moment the output appears your next move is already determined. Without that, you can look at the same screen three times and narrow nothing at all.

Do not copy this list of commands into your internal wiki as-is. Turn the first-60-seconds block into a single script and roll it out to every server. Running that one script during an incident is far faster than recalling nine commands.

References

• vmstat(8) — man7.org (verified 2026-08-15)
• iostat(1) — man7.org (verified 2026-08-15)
• pidstat(1) — man7.org (verified 2026-08-15)
• ss(8) — man7.org (verified 2026-08-15)
• lsof(8) — man7.org (verified 2026-08-15)
• dmesg(1) — man7.org (verified 2026-08-15)
• journalctl(1) — man7.org (verified 2026-08-15)

Further reading

• Next: The complete Linux performance tools guide — how to read the numbers in the output
• The complete Linux performance engineering guide — the theory side, going as far as profiling and eBPF
• The complete systemd service management guide — diagnosing a service that will not start
• Debugging DNS resolution order — when it is only name resolution that fails
• Linux terminal — practise the commands straight from the browser
• Linux command quiz — check how well you remember the options
• curl builder — build curl commands with the timing options attached