Split View: 파일 디스크립터와 inode 완전 가이드: df와 du가 다를 때 무슨 일이 벌어지는가

A Complete Guide to File Descriptors and Inodes: What Happens When df and du Disagree

• Introduction
• 1. Name and substance — what an inode is
• 2. Why df and du disagree
• 3. When you delete and the space does not come back
• 4. inode exhaustion — when free space is not enough to create a file
• 5. File descriptors — what a process has open
• 6. Limits — fixing Too many open files
• 7. Diagnostic order summary
• Quiz: check your understanding
• Closing
• References
• Further reading

Introduction

Run systems long enough and you meet situations that look like they violate common sense. You deleted the logs and the usage reported by df has not moved. There is 30 percent of the volume free and the system says it cannot create a file. All you did was rename a file, and the application keeps writing to the old one. You raised ulimit -n and you still get "Too many open files".

None of the four can be explained without the same two concepts: inodes and file descriptors. And once you know those two, all four look like obvious consequences.

This article is not a filesystem textbook; it is a reference for operators. It explains the concepts only as far as it has to, and spends everything else on "in this situation, what do you check and what do you run".

The baseline is Linux kernel 5.x or later, ext4 and XFS. Where behaviour differs by filesystem type, it is called out at that point. In particular the inode characteristics of ext4 and XFS differ substantially, and that difference is exactly where it becomes a real operational problem.

The reading order goes from concepts to symptoms. The first two sections establish the relationship between inodes and directory entries, and from there each section covers a situation you meet in the field. If you are in a hurry, starting from the diagnostic order summary at the end and working back up to the section you need is also fine.

1. Name and substance — what an inode is

In Linux a file is split into two parts.

• inode: the substance of the file. It holds size, permissions, owner, timestamps, link count, and the location of the data blocks. The file name is not in here.
• directory entry: an item pairing a name with an inode number. A directory is, in the end, a list of such pairs.

This separation explains almost every peculiarity of how files behave on Linux.

ls -li /var/log/messages
stat /var/log/messages

ls -i and stat show the inode number and the link count. In stat output, Links is the link count.

Deleting a file is, precisely, deleting a directory entry. The result is that the link count drops by one. Only when the link count reaches zero and no process has the file open does the kernel release the data blocks. Those two conditions are the crux.

The difference between hard links and symbolic links falls out of this too.

echo hello > original.txt
ln original.txt hardlink.txt
ln -s original.txt symlink.txt
ls -li original.txt hardlink.txt symlink.txt

• A hard link is another name pointing at the same inode. The inode number is identical and the link count becomes 2. You can still reach the data through the hard link after deleting the original.
• A symbolic link is a separate file containing a path string. Its inode number is different, and it breaks when the original is deleted.

Hard links come with restrictions. They can only be made within the same filesystem, and they cannot be placed on directories. That is because inode numbers are only unique within a filesystem. It is also the reason generational backups built on the --link-dest option of rsync are only efficient within one filesystem.

There is one more consequence that falls out of this. Write permission on a file and permission to delete it are separate things. Deleting a file is deleting a directory entry, so what you need is write permission on the directory, not on the file. Even a read-only file can be deleted if you can write to the directory. This is where the accident of someone else's file disappearing from a shared directory comes from.

The standard fix for this problem is the sticky bit.

ls -ld /tmp
sudo chmod 1777 /srv/shared

In a directory with the sticky bit set, only the file owner, the directory owner, and root can delete a file. /tmp is protected this way. When you create a directory several users share, consider this bit along with the rest.

2. Why df and du disagree

df reports the usage tracked by the filesystem superblock; du walks a path and adds up the sizes of the files it meets. When the two disagree, the reason is usually one of three.

Reason 1 — a file that is deleted but still open. This is the most common. There is no directory entry so du cannot find it, but a process has it open so the blocks are not released.

sudo lsof -nP +L1
sudo lsof -nP +L1 | awk 'NR>1 {print $1, $2, $7, $9}' | sort -k3 -nr | head

According to the documentation, +L1 lists open files whose link count is less than 1, which is to say files that have already been unlinked.

Reason 2 — files hidden under a mount. If you put files in a directory and then mount another filesystem over it, the files underneath become invisible while still taking up space.

sudo mkdir -p /mnt/check
sudo mount --bind / /mnt/check
sudo du -x -sh /mnt/check/var/log
sudo umount /mnt/check

Mounting the root again with a bind mount makes the hidden files visible. Be sure to unmount afterwards.

Reason 3 — reserved blocks. By default ext4 reserves a certain percentage for root only. That is why the available space reported by df is smaller than total minus used.

sudo tune2fs -l /dev/sda1 | grep -i 'reserved'

Caution: the tune2fs -m command that reduces the reserved percentage is commonly suggested on the internet, but reserved blocks exist to prevent fragmentation and to guarantee root an emergency working space. Unless this is a data-only volume, do not casually set it to zero.

3. When you delete and the space does not come back

This is the situation you hit most often, so here is the procedure.

Step 1 — confirm.

df -h /var
sudo du -x -sh /var
sudo lsof -nP +L1 | head -20

Step 2 — identify which process.

sudo lsof -nP +L1 | awk 'NR>1 {print $2}' | sort -u | while read -r P; do
  printf '%s\t%s\n' "$P" "$(ps -o comm= -p "$P")"
done

Step 3 — release it. Try these in order.

# 3-1. Reopen signal (safest)
sudo systemctl reload rsyslog
sudo kill -USR1 "$(cat /run/nginx.pid)"

# 3-2. Empty the file descriptor directly (the process stays up)
sudo truncate -s 0 /proc/1234/fd/7

# 3-3. Restart the service (last resort)
sudo systemctl restart myapp

Destructive command warning: truncate -s 0 /proc/PID/fd/N immediately empties the contents of the file that descriptor points at. Specify the wrong descriptor number and you wipe out a different, live file. Always verify the target link before you run it.

sudo ls -l /proc/1234/fd/7

Conversely, you can exploit this same property to recover a deleted file. If a process still has it open, the contents are alive.

sudo ls -l /proc/1234/fd | grep deleted
sudo cp /proc/1234/fd/7 /backup/recovered.log

The symbolic link for a deleted file carries a marker after the path indicating that it was deleted. Copying with cp in that state brings the contents back. This is the first thing to try when you delete a log file by mistake.

4. inode exhaustion — when free space is not enough to create a file

df -i
df -ih

If IUse% is at 100 percent, you cannot create a new file even with blocks left over. If you get a No space left on device error while df -h looks roomy, this is it.

Now find the culprit. It is the problem of finding directories with large file counts.

sudo find /var -xdev -type f 2>/dev/null | awk -F/ '{print "/"$2"/"$3}' | sort | uniq -c | sort -nr | head -20

-xdev keeps it from crossing into another filesystem. If the system is already under load, this scan is itself a burden, so narrow the scope before you run it.

The typical causes are as follows.

• PHP session files or application caches piling up without being cleaned
• A backed-up mail queue
• Batch jobs that create temporary files and never delete them
• The many small files in container image layers

On ext4, the inode count is fixed at the moment the filesystem is created. It cannot be increased later, so once it is exhausted your only options are to delete files or recreate the filesystem. You can adjust the density at creation time.

sudo mkfs.ext4 -i 8192 /dev/sdb1
sudo mkfs.ext4 -N 10000000 /dev/sdb1

-i is the number of bytes per inode (smaller means more inodes), and -N specifies the inode count directly.

XFS is different. It allocates inodes dynamically, so exhaustion is effectively a non-issue. That said, older kernels or particular configurations may have a restriction that confines inodes to a specific region, so for volumes handling large numbers of small files, considering XFS is the safer choice. Check the exact options and constraints in the mkfs man pages for the distribution you are running.

Destructive command warning: mkfs erases all data on the target device. Get one letter of the device name wrong and a production volume disappears. Always verify before running it.

lsblk -f
sudo blkid /dev/sdb1
findmnt /dev/sdb1

5. File descriptors — what a process has open

A file descriptor is an integer pointing at a file a process has opened. By convention 0, 1, and 2 are standard input, standard output, and standard error respectively.

ls -l /proc/1234/fd
ls -l /proc/1234/fd | wc -l
sudo lsof -p 1234
sudo lsof -p 1234 | wc -l

According to the documentation, each entry under /proc/PID/fd is a symbolic link pointing at the actual file. Sockets and pipes are shown as a type and an inode number instead of a file path.

Besides numbers, the FD column of lsof shows these values.

• cwd: current working directory
• rtd: root directory
• txt: program text (the executable)
• mem: memory-mapped file

The letter after the number is the access mode. r is read, w is write, and u is read/write.

Here are some useful query combinations.

sudo lsof -nP -iTCP -sTCP:LISTEN
sudo lsof -nP -i :5432
sudo lsof -u appuser
sudo lsof +D /var/lib/myapp
sudo lsof -c nginx -a -u www-data
sudo lsof -t -c nginx

• -a combines conditions with AND. Without it the conditions are treated as OR, which produces far more results than you expected.
• -t prints only PIDs, which makes it easy to pipe into another command.
• +D recursively searches everything under the given directory. It is slow, so narrow the scope.

6. Limits — fixing Too many open files

The cause of this error spans four layers. Fixing only one layer often does not solve it.

Layer 1 — the per-process soft and hard limits.

ulimit -Sn
ulimit -Hn
cat /proc/1234/limits | grep 'open files'

Reading /proc/PID/limits is the reliable way, because changing ulimit in a shell does not affect processes that are already running.

Layer 2 — the login session limit. This is /etc/security/limits.conf or a file under /etc/security/limits.d/.

appuser soft nofile 65535
appuser hard nofile 65535

This configuration applies only to login sessions that go through PAM. It does not apply to services started by systemd. Not knowing this leaves you stuck for a long time on "I configured it, so why does it not work".

Layer 3 — the systemd unit setting.

[Service]
LimitNOFILE=65535

The limit for a process run as a service is decided here. Applying it requires both a daemon reload and a service restart.

sudo systemctl daemon-reload
sudo systemctl restart myapp
cat /proc/"$(systemctl show -p MainPID --value myapp)"/limits | grep 'open files'

Layer 4 — the system-wide limit.

cat /proc/sys/fs/file-nr
cat /proc/sys/fs/file-max
sysctl fs.file-max

file-nr shows three values: the number of allocated descriptors, the allocated-but-unused portion, and the maximum. When the first approaches the third, you have reached the system-wide limit. On modern systems the default file-max is large enough that hitting it here is rare.

The inotify watch limits produce problems of the same family. Development tools and log collectors that use a lot of file watching hit these often.

cat /proc/sys/fs/inotify/max_user_watches
cat /proc/sys/fs/inotify/max_user_instances

To change the values permanently, put them in a sysctl configuration file. For kernel parameter tuning in general, see the Linux kernel parameter tuning guide.

7. Diagnostic order summary

Here is what to check first, organised by symptom.

To expand on the last row a little: copying a directory full of hard links with a tool that does not understand hard links copies each link as a separate file, multiplying the size several times over.

find /backup -type f -links +1 | head
du -sh --count-links /backup/daily.0
rsync -aH /backup/ /backup2/

The -H option of rsync preserves hard links. tar also recognises hard links by default, but remember that the relationship is only kept if both files are included in the archive.

If you suspect damage to the filesystem itself, run a check.

sudo umount /dev/sdb1
sudo fsck -n /dev/sdb1
sudo xfs_repair -n /dev/sdb1

Destructive command warning: running fsck and xfs_repair on a mounted filesystem can cause corruption. Always unmount first, and perform a read-only check with the -n option before anything else. -n fixes nothing and only reports problems.

Quiz: check your understanding

sudo lsof -nP +L1 | head
sudo systemctl reload rsyslog

sudo ls -l /proc/1234/fd/7
sudo truncate -s 0 /proc/1234/fd/7

df -i
sudo find /var -xdev -type f | awk -F/ '{print "/"$2"/"$3}' | sort | uniq -c | sort -nr | head

[Service]
LimitNOFILE=65535

sudo systemctl daemon-reload
sudo systemctl restart myapp
cat /proc/"$(systemctl show -p MainPID --value myapp)"/limits | grep 'open files'

sudo ls -l /proc/1234/fd | grep deleted
sudo cp /proc/1234/fd/7 /backup/recovered.log

ls -li original.txt hardlink.txt symlink.txt
stat original.txt | grep Links

find /backup -type f -links +1 | head
rsync -aH /backup/ /backup2/

Closing

Reduce this article to one sentence and it comes out like this. On Linux a file name is not the substance, only one reference pointing at the substance.

Everything else follows from that one sentence. Delete a name and the data stays alive as long as another reference remains. That is why the space taken by a deleted file does not come back, that is why you can recover a deleted file, and that is why a process keeps writing to the old file after you rename it.

Moved into an operational checklist it is three lines. For capacity problems, look at df -h and df -i together. When a deletion does not reduce usage, look at lsof +L1. For limit problems, check /proc/PID/limits rather than the shell. Those three lines are this article's practical summary.

References

• lsof(8) — man7.org (verified 2026-08-15)
• proc_pid_fd(5) — man7.org (verified 2026-08-15)
• rsync(1) — man7.org (verified 2026-08-15)
• tar(1) — man7.org (verified 2026-08-15)

Further reading

• Previous: A Complete Guide to TLS Certificates
• Next: A Complete Guide to Linux Firewalls and Access Control
• A Complete Guide to Linux Incident Response Commands — first-pass diagnosis of a full disk
• A Guide to Tuning Linux Kernel Parameters — adjusting system limits
• Linux Terminal — practice lsof and stat
• chmod Calculator — check permission bits