Processes and Signals, the Complete Guide: Tracing Why a Process Will Not Exit

• Introduction
• 1. Process state — the one letter in the STAT column of ps
• 2. The signal table — number, default action, purpose
• 3. kill, pkill, killall — who are you sending it to
• 4. SIGTERM and SIGKILL — designing the grace period
• 5. PID 1, zombies, and orphans
• 6. Process groups, sessions, and what happens when the terminal closes
• 7. Handling signals in an application
• 8. Tracing a process that will not die, layer by layer
• 9. Priorities and resource limits
• Quiz: check your understanding
• Closing
• References
• Further reading

Introduction

Your deploy script sends kill and the process does not die. You send kill -9 and it is still sitting in the list. You close the terminal and the background job dies along with it. You stop a container and the application is cut off halfway through writing data.

All four come from the same gap in knowledge: how a signal is delivered to a process, who is allowed to ignore it, and in which states it is never delivered at all.

This article is not about memorising a list of signals. It walks in order through the real operational situations that signals produce once they are entangled with process groups and sessions, the controlling terminal, and parent-child relationships. It ends with how to design a graceful shutdown in an application and in a container.

The baseline is Linux kernel 5.x or newer, on x86-64 and ARM. Signal numbers differ per architecture. The numbers in this article are the x86/ARM values; Alpha, MIPS, SPARC and others use different ones. In scripts it is always safer to use names rather than numbers.

1. Process state — the one letter in the STAT column of ps

The STAT column of ps summarises, in a single letter, what is happening to that process.

ps -eo pid,ppid,stat,wchan:24,etime,args --sort=-pcpu | head -20

The extra characters carry information too. s means session leader, + means foreground process group, l means multi-threaded, < means high priority, and N means low priority.

If you see the D state, signals will not fix it. Find out what the kernel is waiting for.

sudo cat /proc/1234/stack
sudo cat /proc/1234/wchan; echo
cat /proc/1234/status | grep -E '^State|^SigQ|^SigPnd|^SigBlk|^SigIgn|^SigCgt'

SigPnd is the set of pending signals, SigBlk the blocked ones, SigIgn the ones set to be ignored, and SigCgt the bitmask of signals with a handler registered. They are hexadecimal bitmasks and awkward to read, but they are the definitive way to answer "is this process catching SIGTERM?"

2. The signal table — number, default action, purpose

Of the standard signals defined in the signal(7) man page, these are the ones that actually come up in operations. Numbers are the x86/ARM values.

The default actions mean Term (terminate), Core (terminate and dump core), Ign (ignore), Stop (stop), and Cont (resume).

The single most important sentence in the man page is this one. SIGKILL and SIGSTOP cannot be caught, blocked, or ignored. Every other signal can have its handling changed by the application. That is why programs exist for which kill -TERM does nothing, and that may be design rather than a bug.

SIGPIPE frequently causes trouble in pipelines.

yes | head -1

Once head has read one line and exited, yes is writing into a pipe with no reader, and it is terminated by SIGPIPE. That is correct behaviour. A program that has been configured to ignore SIGPIPE gets an EPIPE error instead, so if it does not handle that error it can spin in an infinite loop.

3. kill, pkill, killall — who are you sending it to

kill -l
kill -TERM 1234
kill -s TERM 1234
kill -9 1234
kill -0 1234

kill -0 sends no signal at all and only checks whether delivery would be possible (does the process exist, do you have permission). It is a good fit for liveness checks in scripts.

There are two kinds of tool for sending by name, and they behave differently.

pkill -TERM -f 'java.*myapp'
pkill -u appuser -TERM nginx
killall -TERM nginx
pgrep -a -f 'java.*myapp'

• pgrep/pkill do pattern matching, and with -f they match against the full command line. Without -f they only look at the process name, which is usually truncated to 15 characters.
• killall matches the exact name. Be aware that on some Unix systems (Solaris and others) killall means something entirely different. If portability matters, pkill is the safe choice.

Destructive command warning: if the pattern is too broad, pkill -f will kill processes you did not intend. It is especially dangerous when run as root. Always confirm the targets with pgrep first, then run pkill with exactly the same pattern.

# Step 1: confirm the targets (kills nothing)
pgrep -a -f 'java.*myapp'
# Step 2: only run this once the list is correct
pkill -TERM -f 'java.*myapp'

To send to an entire process group, put a minus sign in front of the PID. A negative PID means a process group ID.

kill -TERM -12345

This form is useful for cleaning up children in one shot, but getting the group ID wrong has very wide-reaching effects. In particular, kill -9 -1 sends SIGKILL to every process your permissions reach, and run as root it effectively halts the system. Never run it.

4. SIGTERM and SIGKILL — designing the grace period

A correct shutdown procedure always has two stages. Send SIGTERM first, allow a grace period, and only send SIGKILL if it is still alive.

#!/usr/bin/env bash
set -euo pipefail
PID="$1"
TIMEOUT="${2:-30}"

kill -TERM "$PID" 2>/dev/null || exit 0
for _ in $(seq "$TIMEOUT"); do
  if ! kill -0 "$PID" 2>/dev/null; then
    echo "graceful shutdown complete"
    exit 0
  fi
  sleep 1
done
echo "timeout, escalating to SIGKILL" >&2
kill -KILL "$PID"

SIGKILL gives the application no chance to clean up. Buffers on open files are not flushed, transactions are cut off, and temp files and lock files are left behind. Sending SIGKILL to a database or a queue consumer is effectively pulling the power cord. A generous grace period is always the better trade.

For systemd services this procedure is available as configuration.

[Service]
KillSignal=SIGTERM
TimeoutStopSec=60
KillMode=mixed
SendSIGKILL=yes

KillMode=mixed sends SIGTERM only to the main process, and after the timeout sends SIGKILL to every process in the cgroup. It suits applications that manage their own child processes. The exact meaning and defaults of each directive can change between systemd versions, so check the systemd.kill documentation for the version you have installed.

Containers follow the same model. docker stop sends SIGTERM, waits 10 seconds by default, then sends SIGKILL. Kubernetes uses the termination grace period from the Pod spec.

docker stop --time 60 mycontainer
kubectl delete pod mypod --grace-period=60

There is a trap people hit constantly in containers. If you use the shell form (CMD myapp), the shell becomes PID 1 and the application becomes its child. SIGTERM then goes to the shell, and the shell does not forward it to the child. The end result is that the application is killed by SIGKILL with no grace period at all. Use the exec form (CMD ["myapp"]), or add a proper init process.

5. PID 1, zombies, and orphans

When a process exits, the kernel keeps its exit status around. The process entry only disappears once the parent calls wait and reaps it. The state before that reaping is the zombie state (Z).

ps -eo pid,ppid,stat,args | awk '$3 ~ /Z/'

A zombie uses almost no memory, but it occupies a PID. If thousands of zombies pile up, PIDs are exhausted and no new process can be created.

cat /proc/sys/kernel/pid_max
ls /proc | grep -c '^[0-9]'

You cannot kill a zombie. It is already dead. The fix is to make the parent reap it, or to terminate the parent so the zombie is adopted by PID 1. PID 1 is responsible for adopting orphaned processes and reaping them automatically.

If the parent dies first, the child becomes an orphan and is adopted by PID 1 (or by a process registered as a subreaper). If zombies keep accumulating, the parent program has a bug: it is not handling SIGCHLD.

Zombie accumulation is especially common in containers. It happens when the application runs as PID 1 and was never written to reap children. Using the init option provided by the container runtime is the standard fix.

docker run --init myimage

6. Process groups, sessions, and what happens when the terminal closes

Why does closing the terminal kill your background job? There are three layers involved.

• Process group: one pipeline is usually one group. It is the unit of shell job control.
• Session: one login is one session. The session leader owns the controlling terminal.
• Controlling terminal: when the terminal goes away, the kernel sends SIGHUP to the session leader, and the session leader (usually the shell) propagates SIGHUP to its jobs.

This is how you inspect the current structure.

ps -eo pid,ppid,pgid,sid,tty,stat,args | head -20
ps -o pid,pgid,sid,tty,args -p $$

There are several ways to survive a terminal disconnect, and they differ.

nohup ./long-job.sh > /var/log/long-job.log 2>&1 &
setsid ./long-job.sh > /var/log/long-job.log 2>&1 &
disown -h %1
systemd-run --user --unit=long-job ./long-job.sh

• nohup sets SIGHUP to be ignored and then runs the command. It still belongs to the same session.
• setsid creates a new session, severing the relationship with the controlling terminal entirely. It is more reliable.
• disown -h removes an already-running job from the shell list that receives propagated SIGHUP.
• systemd-run runs the work as a separate unit entirely, making it independent of the session. It is the safest choice for long-running work on a production server.

If you run long jobs over SSH, tmux or screen is the practical answer. The session survives a dropped connection and you can reattach later.

7. Handling signals in an application

In shell scripts you handle them with trap.

#!/usr/bin/env bash
set -euo pipefail

cleanup() {
  echo "cleaning up..."
  rm -f /tmp/myjob.lock
}
trap cleanup EXIT
trap 'echo "received SIGTERM"; exit 143' TERM
trap 'echo "received SIGINT"; exit 130' INT

while true; do
  sleep 1
done

trap ... EXIT runs no matter which path the script exits by, which makes it the right place for cleanup logic. The exit code convention is 128 plus the signal number. Dying from SIGTERM (15) gives 143, from SIGINT (2) gives 130. Once you know this convention you can see 143 in systemctl status or a CI log and immediately read it as "it received a normal shutdown request".

One thing to watch out for: while an external command such as sleep is running, bash does not process the trap immediately — it processes it after that command finishes. A long sleep therefore makes the script slow to react. Running it in the background and using wait gives an immediate response.

sleep 300 &
wait $!

The timeout command is useful too.

timeout 30 ./maybe-hangs.sh
timeout -s KILL 30 ./maybe-hangs.sh
timeout -k 10 30 ./maybe-hangs.sh

-k sends the default signal first and then, if the process is still alive after the extra interval you specify, sends SIGKILL. It can replace the two-stage shutdown script from earlier with a single line.

8. Tracing a process that will not die, layer by layer

When you get a report that "it will not die", narrow it down in this order.

Step 1 — is it actually alive? If it is a zombie, it is already dead.

ps -o pid,ppid,stat,args -p 1234

Step 2 — was the signal delivered? Delivery itself may have failed because of permissions.

kill -0 1234; echo "exit=$?"

If you get Operation not permitted, it is a permissions problem.

Step 3 — is the process catching the signal?

grep -E '^SigIgn|^SigCgt|^SigBlk' /proc/1234/status

Step 4 — is it in the D state? If so, signals are of no use whatsoever.

cat /proc/1234/status | grep '^State'
sudo cat /proc/1234/stack

If the stack shows NFS or block-layer functions, you have to wait for storage to respond. If it is a disconnected NFS mount, a forced unmount is the only way out.

sudo umount -f -l /mnt/nfsshare

Destructive command warning: umount -l (lazy) detaches the mount from the namespace immediately but leaves the in-use references in place. Writes in flight can be lost, so use it only as a last resort.

Step 5 — does it recur after a restart? If it does, this is not a signal problem: it is a structural problem in the application or in storage.

9. Priorities and resource limits

There is also the option of restraining a process instead of killing it.

nice -n 10 ./batch-job.sh
renice -n 10 -p 1234
ionice -c 3 -p 1234
chrt -p 1234

• nice values run from -20 (highest priority) to 19 (lowest). Lowering it into negative values requires root.
• ionice -c 3 is the idle I/O class: it performs I/O only when no other process is using the disk. Applying it to nightly backups or bulk copy jobs dramatically reduces the impact on production traffic.
• chrt deals with real-time scheduling policies. Used incorrectly it can make the system unresponsive, so treat it with care.

Limits are set with ulimit and with systemd directives.

ulimit -a
ulimit -n
cat /proc/1234/limits

In a systemd unit you configure them like this.

[Service]
LimitNOFILE=65535
LimitNPROC=4096
MemoryMax=2G
CPUQuota=200%

When MemoryMax is exceeded, the cgroup OOM killer kills processes only inside that service. That prevents it from spilling over into a system-wide OOM, so it is well worth setting on any memory-hungry service. The available directives and their defaults vary by systemd version, so check the systemd.resource-control documentation for the version you have installed.

Quiz: check your understanding

ps -o pid,ppid,stat,wchan:24,args -p 1234

sudo cat /proc/1234/stack

docker run --init myimage
docker stop --time 60 mycontainer

grep SigCgt /proc/1/status

setsid ./deploy.sh > /var/log/deploy.log 2>&1 &
systemd-run --unit=deploy-2026-08-15 ./deploy.sh
tmux new -s deploy

ps -eo pid,ppid,stat,args | awk '$3 ~ /Z/' | head
cat /proc/sys/kernel/pid_max

sleep 10 &
wait $!

renice -n 19 -p 1234
sudo ionice -c 3 -p 1234

[Service]
Nice=19
IOSchedulingClass=idle
CPUQuota=50%

Closing

Signals look simple, but process groups, sessions, the controlling terminal, cgroups, and the container runtime are layered on top of them. That is why the answer to "why will it not die" comes from a different layer every time.

Three things are worth remembering in practice. Only SIGKILL and SIGSTOP are absolute; everything else is at the application's mercy. In the D state no signal helps at all, so look at storage. Shutdown always starts with SIGTERM and a grace period, and SIGKILL is the last resort.

That last one matters most for services that handle data. If you run with a 10-second termination grace period and SIGKILL lands in the middle of a large transaction, that day's incident was manufactured by your shutdown procedure.

References

• signal(7) — man7.org (verified 2026-08-15)
• proc_pid_fd(5) — man7.org (verified 2026-08-15)
• dmesg(1) — man7.org (verified 2026-08-15)
• systemd official documentation — freedesktop.org (verified 2026-08-15)

Further reading

• Previous: The complete guide to Linux performance tools
• Next: The complete guide to running SSH
• The complete guide to systemd service management — designing shutdown procedures with unit files
• The complete guide to terminal and shell internals — how PTYs and job control work inside
• Linux terminal — practise signal-related commands
• Linux command quiz — check how well you remember the options