Space Left but No space left on device — inode exhaustion, deleted open files, reserved blocks

Introduction — df says 38% is left, yet a single touch fails

The symptom starts like this. The application cannot write its logs, a deployment fails while creating a temporary file, the database cannot write its WAL. You log in to check and the disk has room to spare.

df -h /var
# Filesystem      Size  Used Avail Use% Mounted on
# /dev/nvme0n1p2  100G   61G   35G  64% /

touch /var/log/test
# touch: cannot touch '/var/log/test': No space left on device

Avail is 35GB and yet a single zero-byte file cannot be created. It is easy to conclude here that "the monitoring is broken", but df honestly reported block usage and nothing more. ENOSPC is errno 28, and the kernel has several paths to that value besides a shortage of blocks.

This post lists those paths in order of likelihood, with the command that confirms each one, the fix, and how to keep it from coming back. Walking through them in order takes about five minutes.

What to check first — are you really writing to that filesystem

Spend thirty seconds before starting the diagnosis. The most common mistake is running df -h with no argument, skimming the list it prints, and guessing which entry the failing path belongs to. Ask about the path directly.

# The filesystem this path actually belongs to
findmnt -T /var/log/app/application.log
# TARGET SOURCE          FSTYPE OPTIONS
# /var   /dev/nvme0n1p5  xfs    rw,relatime,attr2,inode64,logbufs=8

df -h /var/log/app/
# Filesystem      Size  Used Avail Use% Mounted on
# /dev/nvme0n1p5   20G   20G     0 100% /var

It was not / but /var that was a separate partition, and that is the one that filled up. If this is your case, you are done here. Environments where /tmp is a tmpfs are just as common.

df -h /tmp /dev/shm
# Filesystem      Size  Used Avail Use% Mounted on
# tmpfs           3.2G  3.2G     0 100% /tmp
# tmpfs           7.9G  6.1G  1.8G  78% /dev/shm

tmpfs uses memory. When it fills up you get ENOSPC with no relation to the disk at all, and that same amount of physical memory is being eaten at the same time.

If the path checks out and Avail still shows room, now we get to the main topic.

Cause 1: inode exhaustion — blocks remain but the file count ran out

The most common cause. Confirming it is one line.

df -i /
# Filesystem        Inodes   IUsed IFree IUse% Mounted on
# /dev/nvme0n1p2   6553600 6553600     0  100% /

IUse% is 100%. ext2/3/4 fix the inode count at format time and it cannot be raised afterwards. The default is decided by the -i option (bytes-per-inode) of mke2fs, and the default profile gives roughly one per 16KB. That is about 6.55 million on a 100GB filesystem. If the average file size of your workload is smaller than 16KB, inodes run out before blocks do.

Here is how to find the guilty directory. du counts size, not the number of files, so it is useless here.

# Top 10 directories by file count (within the same filesystem only)
sudo find / -xdev -printf '%h\n' 2>/dev/null | sort | uniq -c | sort -rn | head -10
# 2841193 /var/spool/postfix/maildrop
#   84120 /var/lib/php/sessions
#   38914 /home/deploy/.cache/pip/http
#   18422 /var/lib/apt/lists
#    9104 /usr/share/man/man3

-xdev matters. Leave it out and the scan walks into the other mounted filesystems, which takes several times longer and pollutes the result. If the tree is large and the command above takes too long, narrow down from the top level.

for d in /*/; do
  printf '%9s  %s\n' "$(sudo find "$d" -xdev 2>/dev/null | wc -l)" "$d"
done | sort -rn | head -5
#   2925104  /var/
#     91208  /usr/
#     42011  /home/

The usual suspects are a fixed set. Mail queues (maildrop and deferred of postfix), the PHP session directory, package manager caches, applications that leave session or lock files behind, and container layers that never get cleaned up.

The fix has two stages. For right now, delete. When the file count is large rm -rf runs into the argument length limit, so hand the work to find.

# Delete session files older than 30 days (a safe form when the count is large)
sudo find /var/lib/php/sessions -type f -mtime +30 -delete

# Watching the deletion progress (when there are millions of files)
sudo find /var/spool/postfix/maildrop -type f -mtime +7 -print -delete | pv -l > /dev/null

The root fix is recreating or migrating the filesystem. Raising the inode count on ext4 means formatting again.

# One inode per 4KB (four times the default). All the data is lost
sudo mkfs.ext4 -i 4096 /dev/nvme0n1p6

# Or specify the inode count directly
sudo mkfs.ext4 -N 20000000 /dev/nvme0n1p6

Moving to XFS is a practical choice too. XFS allocates inodes dynamically, so this problem is effectively gone. It is not completely gone, though, because imaxpct caps the share of space that inodes may use.

xfs_info /var | grep -E 'imaxpct|isize'
# meta-data=/dev/nvme0n1p5 isize=512  agcount=4, agsize=1310720 blks
#          =               sectsz=512 attr=2, projid32bit=1
# data     =               bsize=4096 blocks=5242880, imaxpct=25

# Raise the cap if you need to (possible while mounted)
sudo xfs_growfs -m 50 /var

Cause 2: files that were deleted but are still open

When df and du disagree by a lot, this is the one.

df -h /var
# Filesystem      Size  Used Avail Use% Mounted on
# /dev/nvme0n1p5   20G   20G     0 100% /var

sudo du -shx /var
# 3.1G	/var

df says 20GB is used and du finds only 3.1GB. The 17GB gap is files whose directory entry is gone. On Unix a file is released only when its link count is 0 and the count of open descriptors is 0 as well. rm removes only the link. If somebody is holding that file open the space is not returned, and du walks paths, so it cannot count that file.

Here is the command that confirms it.

sudo lsof -nP +L1 | head
# COMMAND    PID USER  FD  TYPE DEVICE     SIZE/OFF NLINK    NODE NAME
# java     21874  app  12w REG  259,5    8589934592     0 1183241 /var/log/app/application.log (deleted)
# nginx    14022 www   9w  REG  259,5    4294967296     0 1183502 /var/log/nginx/access.log (deleted)
# rsyslogd  1288 root  7w  REG  259,5     429496729     0 1183210 /var/log/syslog.1 (deleted)

+L1 means print only the open files whose link count is below 1, and the 0 in the NLINK column is exactly that state. The SIZE/OFF column is the number of bytes that are not being returned. The three above add up to about 12GB.

On a minimal image where lsof is not installed, look at /proc directly.

sudo ls -l /proc/[0-9]*/fd/* 2>/dev/null | grep '(deleted)' | awk '{print $9, $10, $11}' | head
# /proc/21874/fd/12 -> /var/log/app/application.log
# /proc/14022/fd/9 -> /var/log/nginx/access.log

The cause is almost always the same. The process never reopened the file after log rotation. When logrotate rotates with the create method it moves the original aside and creates a new file, while the process is still writing to the old inode. A signal has to be sent from the postrotate script, and that part is either missing or failing.

There are two fixes.

# A) Send the reopen signal to the process (cleanest)
sudo systemctl reload nginx        # nginx uses USR1
sudo kill -HUP 1288                # rsyslogd

# B) If a restart is impossible, empty it directly through the fd
sudo truncate -s 0 /proc/21874/fd/12

Option B has a side effect. If the process opened the file with O_APPEND the next write continues at the end of the file, which is now 0, so there is no problem; otherwise it keeps writing at the old offset and the file becomes a sparse file. The blocks actually occupied went down, but the size shown by ls -l still looks just as large. Use it as first aid only; the proper fix is option A.

Fixing the logrotate configuration is what prevents a recurrence.

cat /etc/logrotate.d/app
# /var/log/app/*.log {
#     daily
#     rotate 14
#     compress
#     delaycompress
#     missingok
#     notifempty
#     copytruncate
# }

copytruncate copies the original and then truncates the original to 0. The inode is preserved, so the process has to do nothing. In exchange, the logs written between the copy and the truncation can be lost. If that loss is unacceptable, send a signal from postrotate instead of using copytruncate, and make sure that script does not fail silently.

    postrotate
        systemctl reload app.service || systemctl kill -s USR1 app.service
    endscript

Cause 3: reserved blocks — the 5% only root can use

ext2/3/4 keep a fixed share of all blocks for root only. The default is 5%. There are two purposes: letting an administrator log in and clean up even when the disk is full, and leaving the allocator room to find contiguous blocks so that fragmentation is delayed.

The symptom in this case is distinctive. Root can create the file but the service account cannot.

sudo touch /var/lib/app/probe && echo "root: ok"
# root: ok

sudo -u appuser touch /var/lib/app/probe2
# touch: cannot touch '/var/lib/app/probe2': No space left on device

tune2fs is what confirms it.

sudo tune2fs -l /dev/nvme0n1p2 | grep -E 'Block count|Reserved block count|Block size|Free blocks'
# Block count:              26214400
# Reserved block count:     1310720
# Free blocks:              1310698
# Block size:               4096

Reserved block count is 1310720 and Free blocks is almost the same. That means the entire remaining space is the reservation. 1310720 times 4096 is about 5.0GiB, and 1310720 out of 26214400 is exactly 5%.

Knowing how df presents this situation removes a lot of confusion. The Avail of df is the value with the reservation subtracted, and Use% is computed as used divided by (used plus Avail). So the moment only the reservation is left, Avail becomes 0 and Use% becomes 100%. In other words, with this cause df usually shows 100%, and the impression that "there is space left" comes from the gap against du or from a write that succeeds as root.

On a 100GB data volume, 5% is 5GB. For a pure data volume that is not the root filesystem, that ratio is excessive.

# Lower it to 1%. It applies immediately while mounted and the data is safe
sudo tune2fs -m 1 /dev/nvme0n1p2
# Setting reserved blocks percentage to 1% (262144 blocks)

# If you want to specify an absolute count
sudo tune2fs -r 262144 /dev/nvme0n1p2

The recommended baseline is this. Keep 5% on the root filesystem. There is no reason to create a situation the administrator cannot get into. A volume dedicated to logs or data can safely drop to 1%, and on a multi-terabyte volume you can set it to 0 and put solid monitoring in place instead. Note, though, that with the reservation at 0 fragmentation accelerates once the filesystem is nearly full, so it is not recommended for a volume you intend to run above 90% at all times.

Causes 4 and 5: mount shadowing, container overlays, and fake ENOSPC

If df and du disagree but there are no deleted open files, this is the candidate that remains. Mounting another filesystem on a directory that already holds files makes the original files invisible. Those files still occupy space on the filesystem underneath, though. du only walks the mounted top, so it will never find them.

The typical scenario goes like this. Logs had piled up under /var/log, then a new disk was attached and mounted at /var/log. Now 40GB of old logs are trapped inside the root filesystem and invisible to every tool.

The way to check is to bind mount the root somewhere else and look at the original that is not shadowed.

sudo mkdir -p /mnt/rootfs
sudo mount --bind / /mnt/rootfs

# The files that were hidden are visible here
sudo du -xhd1 /mnt/rootfs/var | sort -h | tail -5
# 1.2G	/mnt/rootfs/var/lib
# 2.8G	/mnt/rootfs/var/cache
# 41G	/mnt/rootfs/var/log
# 46G	/mnt/rootfs/var

sudo ls -la /mnt/rootfs/var/log | head
# -rw-r----- 1 syslog adm 18253611008 Mar  2 04:11 syslog.1
# -rw-r----- 1 syslog adm 22091571200 Feb 18 03:52 kern.log.1

# Clean up, then release
sudo rm -f /mnt/rootfs/var/log/*.1
sudo umount /mnt/rootfs

To check whether mounts overlap in the first place, use findmnt.

findmnt --list -o TARGET,SOURCE,FSTYPE,SIZE,USED,AVAIL | head
# TARGET      SOURCE          FSTYPE  SIZE  USED AVAIL
# /           /dev/nvme0n1p2  ext4    100G   96G     0
# /var/log    /dev/nvme1n1p1  ext4    200G   12G  178G
# /var/lib/docker /dev/nvme2n1p1 xfs   500G  310G  190G

Containers add one more layer on top of this. Run df inside a container and you see the overlay filesystem, and that value is usually the value of the filesystem that holds the host /var/lib/docker (or the containerd root).

# Inside the container
df -h /
# Filesystem      Size  Used Avail Use% Mounted on
# overlay         500G  310G  190G  63% /

It shows 190GB left, but the amount you can actually use may be different, because three separate caps apply.

• The storage cap of the runtime: docker run --storage-opt size=10G. On the overlay2 driver it works only when the backing filesystem is XFS mounted with the prjquota option. If the condition is not met, the option itself is rejected.
• The ephemeral-storage of Kubernetes: this is not a cap but an eviction threshold. Exceeding it does not produce ENOSPC, it gets the pod Evicted. The error message is completely different, so the two are easy to tell apart.
• The sizeLimit of emptyDir: a memory-backed emptyDir is a tmpfs, so exceeding it produces a real ENOSPC.

Here is how to see what is eating space on the host side.

docker system df -v | head -20
# TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
# Images          148       12        182.4GB   161.2GB (88%)
# Containers      31        12        41.2GB    38.1GB (92%)
# Local Volumes   22        6         84.9GB    71.3GB (83%)
# Build Cache     412       0         38.2GB    38.2GB

# Reclaim safely (nothing in use is touched)
docker image prune -a --filter 'until=168h'
docker builder prune --filter 'until=168h'

Container logs are frequently forgotten too. The json-file driver grows without bound by default.

sudo du -sh /var/lib/docker/containers/*/*-json.log | sort -h | tail -3
# 8.2G	/var/lib/docker/containers/3f7b91ac.../3f7b91ac...-json.log

# Cap it with the daemon-wide configuration
cat /etc/docker/daemon.json
# {
#   "log-driver": "json-file",
#   "log-opts": { "max-size": "100m", "max-file": "5" }
# }

This setting applies from newly created containers onward. Existing containers have to be recreated.

Cause 5: the same wording produced by inotify

The last trap. inotify_add_watch returns ENOSPC when it reaches the watch count limit. So a tool that uses file watching prints "No space left on device" with nothing to do with the disk at all. If the filesystem is perfectly fine and only one particular process throws this error, suspect this one.

sysctl fs.inotify.max_user_watches fs.inotify.max_user_instances
# fs.inotify.max_user_watches = 65536
# fs.inotify.max_user_instances = 128

# Find the processes holding many watches
sudo find /proc/[0-9]*/fdinfo -type f 2>/dev/null \
  | xargs grep -l '^inotify' 2>/dev/null \
  | cut -d/ -f3 | sort -u \
  | while read -r pid; do
      n=$(grep -hc '^inotify' /proc/"$pid"/fdinfo/* 2>/dev/null | paste -sd+ | bc)
      printf '%7s %6s %s\n' "$n" "$pid" "$(tr -d '\0' < /proc/"$pid"/comm)"
    done | sort -rn | head -5
#   58210   9931 node
#    4102   3311 containerd

echo 'fs.inotify.max_user_watches = 524288' | sudo tee /etc/sysctl.d/60-inotify.conf
sudo sysctl --system

Preventing a recurrence — a decision table and two alerts

Here is the whole thing as a decision table. Check from the top down.

If you have made it this far, the last job is making sure it never comes back.

Watch both blocks and inodes. Most dashboards look only at block usage. node_exporter exports both metrics, so all you have to do is add the second alert.

# Blocks
(1 - node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}
   / node_filesystem_size_bytes{fstype!~"tmpfs|overlay"}) > 0.85

# inode — many places do not have this alert
(1 - node_filesystem_files_free{fstype!~"tmpfs|overlay"}
   / node_filesystem_files{fstype!~"tmpfs|overlay"}) > 0.85

Watch the growth rate along with it. Even if the alert fires at 85% usage, it is too late when 100% is 30 minutes away. Taking the projected exhaustion time from predict_linear as the criterion is more practical.

predict_linear(node_filesystem_avail_bytes{mountpoint="/var"}[6h], 4 * 3600) < 0

Verify the log rotation for real. Having a configuration file and having it work are different things.

# Check which files are targeted without running it
sudo logrotate -d /etc/logrotate.conf 2>&1 | grep -E 'considering|rotating|error'

# Force a run to verify the postrotate script as well
sudo logrotate -vf /etc/logrotate.d/app

# Right after the run, check whether deleted open files remain
sudo lsof -nP +L1 | grep -c deleted
# 0

Make the regular check a one-liner. Leaving it behind as a script is better than digging through memory during an incident.

#!/usr/bin/env bash
# fs-health.sh — sweeps the ENOSPC cause candidates in one pass
set -u

echo "== Block usage =="
df -hT -x tmpfs -x devtmpfs

echo; echo "== inode usage =="
df -i -x tmpfs -x devtmpfs

echo; echo "== Deleted but still open files (top 5) =="
lsof -nP +L1 2>/dev/null | awk 'NR>1 {print $7, $1, $2, $NF}' | sort -rn | head -5

echo; echo "== Reserved block percentage =="
for dev in $(lsblk -pnro NAME,FSTYPE | awk '$2 ~ /^ext[234]$/ {print $1}'); do
  tune2fs -l "$dev" 2>/dev/null \
    | awk -v d="$dev" '/Block count/{t=$3} /Reserved block count/{r=$3}
                       END {if (t) printf "%s  %.1f%%\n", d, r*100/t}'
done

echo; echo "== Mount shadowing candidates (df and du mismatch) =="
findmnt --list -o TARGET,SOURCE,FSTYPE,USE% -t ext4,xfs

Closing — do not judge from df alone

If you remember one thing, let it be this. ENOSPC does not mean "there are no blocks". It means "the kernel has no resource to accept this write", and that resource may be blocks, or inodes, or the free space excluding the reservation, or inotify watches.

Compressed into a single diagnostic order, it goes like this.

1. Pin down the filesystem the path belongs to with findmnt -T. Half the cases end here.
2. Look at inodes with df -i. They are a resource separate from blocks.
3. Compare df and du -x. A mismatch means either a deleted open file or mount shadowing. Separate the two with lsof +L1 for the former and a bind mount for the latter.
4. If root works and the service account does not, it is reserved blocks. Adjust it with tune2fs -m.
5. If the filesystem is perfectly fine and only one particular process throws this error, suspect inotify.
6. Set two alerts, one for blocks and one for inodes. And the projected exhaustion time is more useful than the usage percentage.