Too many open files solved for good — why raising ulimit does not help

Introduction — accept4() failed (24: Too many open files)

The symptom shows up like this.

2026/07/25 14:02:11 [alert] 14022#14022: accept4() failed (24: Too many open files)
2026/07/25 14:02:11 [alert] 14022#14022: accept4() failed (24: Too many open files)

Or, in Java, like this.

java.net.SocketException: Too many open files
	at java.base/sun.nio.ch.Net.accept(Native Method)
Caused by: java.io.IOException: Too many open files

At this point almost everybody does the same thing. They run ulimit -n 65536, restart the service, and see exactly the same error. The reason is that there is not one limit but three, and the value you changed in the shell never reaches the service.

errno 24 is EMFILE. You have to read the wording precisely.

errno 24 23
# EMFILE 24 Too many open files
# ENFILE 23 Too many open files in system

If in system is attached, you are looking at a different problem. That one difference splits half the diagnosis for you.

There is not one limit but three

Three limits exist independently of one another, and hitting any single one of them stops you from opening a file.

Their relationship looks like this.

fs.file-max   : how many files the whole system can hold open at once (the sum over every process)
   └─ fs.nr_open : the ceiling that the RLIMIT_NOFILE of one process cannot exceed
        └─ RLIMIT_NOFILE hard : the upper bound an administrator set for that process
             └─ RLIMIT_NOFILE soft : the value actually in effect. A process can raise it up to hard on its own

Look at the current state in one go.

sysctl fs.file-max fs.nr_open
# fs.file-max = 6553600
# fs.nr_open = 1048576

cat /proc/sys/fs/file-nr
# 13984	0	6553600

The three numbers in file-nr are, in order, the count of allocated file handles, the count allocated but unused (always 0 on recent kernels), and the maximum. When the first approaches the third you have a system-wide problem, and what surfaces then is ENFILE. In the example above it is 13984 against 6.55 million, so the headroom is overwhelming. On most modern Linux systems fs.file-max is not the bottleneck. The kernel computes it automatically in proportion to the amount of memory. Even so, plenty of tuning documents tell you to raise this value first. Raising it is harmless, but the symptom stays exactly where it was.

fs.nr_open is a different story. Any ulimit -n above the default of 1048576 fails to be set at all.

ulimit -n 2000000
# bash: ulimit: open files: cannot modify limit: Operation not permitted

sudo sysctl -w fs.nr_open=2097152
# fs.nr_open = 2097152
ulimit -n 2000000   # now it succeeds

Checking the value actually in effect — /proc/PID/limits, not ulimit

This is the crux. ulimit -n shows the value of the shell you are in right now. It is not the value of the process that is failing. The two differ all the time.

ulimit -n
# 1048576

pgrep -x nginx
# 14022
# 14025

sudo grep 'Max open files' /proc/14025/limits
# Max open files            1024                 524288               files

The shell says a million while the soft limit of the nginx worker is 1024. That gap is the whole identity of the problem. From now on, when you meet this symptom, do not even look at ulimit -n — run this command first.

# every process matching a name, in one shot
for pid in $(pgrep -x nginx); do
  printf '%6s  %s\n' "$pid" "$(awk '/Max open files/ {print $4, $5}' /proc/"$pid"/limits)"
done
#  14022  1024 524288
#  14025  1024 524288

Look at how many are in use right now at the same time.

# the real number of open fds (the most accurate)
sudo ls /proc/14025/fd | wc -l
# 1021

# utilization on a single line
for pid in $(pgrep -x nginx); do
  used=$(sudo ls /proc/"$pid"/fd 2>/dev/null | wc -l)
  soft=$(awk '/Max open files/ {print $4}' /proc/"$pid"/limits)
  printf '%6s  %6s / %-8s (%d%%)\n' "$pid" "$used" "$soft" $((used * 100 / soft))
done
#  14022    1021 / 1024     (99%)
#  14025    1019 / 1024     (99%)

If you use lsof -p PID | wc -l the number comes out larger. lsof also counts the current working directory, the root directory, the executable and memory-mapped libraries, and none of those count against RLIMIT_NOFILE. When you compare against the limit, use the count from /proc/PID/fd.

Why systemd ignores your ulimit settings

This is the most common trap. If you have ever added the configuration below, restarted, and watched nothing change, this section is the answer.

cat /etc/security/limits.d/99-nofile.conf
# *    soft    nofile    1048576
# *    hard    nofile    1048576

This file is read by the PAM module pam_limits. And pam_limits works only inside a login session. SSH logins, su and login qualify. A system service that systemd starts at boot is not a login session, so it never reads this file at all.

What decides the limit of a service is systemd. The precedence is LimitNOFILE in the unit, then DefaultLimitNOFILE in /etc/systemd/system.conf if that is absent, and the systemd built-in default if that is absent too.

# the value applied to the unit (hard and soft are exposed separately)
systemctl show -p LimitNOFILE -p LimitNOFILESoft nginx.service
# LimitNOFILE=524288
# LimitNOFILESoft=1024

# the global default
systemctl show -p DefaultLimitNOFILE -p DefaultLimitNOFILESoft
# DefaultLimitNOFILE=524288
# DefaultLimitNOFILESoft=1024

Here it pays to know the default behaviour of systemd 240 and later. systemd sets the hard limit generously at 524288 while keeping the soft limit low, at 1024. The reason is compatibility. select() cannot handle fd numbers of 1024 and above, and some older programs run a loop at startup that closes every fd from 0 up to the soft limit, so if that limit is a million, starting up takes several seconds. The design intent of systemd is therefore that a program which needs more should raise it for itself.

A good number of modern runtimes really do that. The Go runtime raises the soft limit to the hard limit at startup, which is why on the same machine a Go service can be perfectly healthy while only the other services hit EMFILE.

Here is the correct way to configure it.

sudo systemctl edit nginx.service
# write the following into the editor that opens
# [Service]
# LimitNOFILE=65536

sudo systemctl daemon-reload
sudo systemctl restart nginx

Two points deserve emphasis. systemctl reload does not change rlimits. The value is applied at process creation time, so it has to be restart. And when you write a single value like LimitNOFILE=65536, both the soft and the hard limit become that value. To give them separately, use the LimitNOFILE=65536:524288 form.

Confirm that it took effect.

sudo grep 'Max open files' /proc/$(pgrep -x nginx | head -1)/limits
# Max open files            65536                65536                files

Sometimes the application has a setting of its own on top of that. nginx carries a separate directive called worker_rlimit_nofile, and that value cannot exceed the hard limit systemd handed it.

# /etc/nginx/nginx.conf
worker_rlimit_nofile 65536;
events {
    worker_connections 32768;
}

Setting worker_connections higher than worker_rlimit_nofile is meaningless. A single connection can use one client socket and one upstream socket, so for a reverse proxy it is safe to set worker_rlimit_nofile to at least twice worker_connections.

There is also a way to put out the fire without a restart.

sudo prlimit --pid 14025 --nofile=65536:65536
sudo prlimit --pid 14025 --nofile
# RESOURCE DESCRIPTION                             SOFT  HARD UNITS
# NOFILE   max number of open files               65536 65536 files

prlimit changes the limit of a running process. That said, if the application read the limit at startup and sized its internal data structures from it (the nginx connection pool does exactly that), it may have no effect. The real fix is editing the unit file and restarting.

Containers play by different rules again. The RLIMIT_NOFILE of a process inside a container comes from process.rlimits in the OCI runtime spec, and that value is decided by the container runtime daemon. Neither the shell ulimit nor the limits.conf on the host has any say.

# checking from inside the container is the only reliable method
docker exec api-server cat /proc/1/limits | grep 'Max open files'
# Max open files            1048576              1048576              files

# specify it per container
docker run --ulimit nofile=65536:65536 myimage

# the daemon-wide default
cat /etc/docker/daemon.json
# {
#   "default-ulimits": { "nofile": { "Name": "nofile", "Soft": 65536, "Hard": 65536 } }
# }

The Kubernetes pod spec has no ulimit field. To adjust it per pod you have to lower it with ulimit -n in the container entrypoint (which can only go below the hard limit), change the containerd configuration on the node, or adjust LimitNOFILE on the container runtime unit. Many distributions ship that runtime unit with LimitNOFILE=infinity, and in that case the container inherits a very large value. It looks generous and comfortable, but for a program that runs the close-every-fd loop described above it becomes the reason startup takes tens of seconds.

What is eating the fds — sockets, files and pipes, broken down

Before you raise the limit you have to see what is occupying it. The symlink targets under /proc/PID/fd tell you the kind.

sudo ls -l /proc/14025/fd | awk '{print $NF}' \
  | sed -E 's#^(socket|pipe|anon_inode):.*#\1#; s#^/.*#regular-file#' \
  | sort | uniq -c | sort -rn
#   61204 socket
#    3891 regular-file
#     412 anon_inode
#       8 pipe
#       3 /dev/null

Sixty thousand sockets means the network side. Dig further into which sockets they are.

# socket count by state
ss -tan | awk 'NR>1 {print $1}' | sort | uniq -c | sort -rn
#   38210 ESTAB
#   21044 CLOSE-WAIT
#    2118 TIME-WAIT
#     412 LISTEN

There are twenty thousand CLOSE-WAIT sockets. This is conclusive evidence of an fd leak. CLOSE-WAIT is the state where the peer sent a FIN and closed the connection while the application on our side never called close(). The kernel waits indefinitely in that state for the application to close it. There is no timeout. Do not confuse it with TIME-WAIT. TIME-WAIT is cleaned up by the kernel on its own and consumes no fd.

Looking at which peer the connections belong to narrows down the offending code.

sudo ss -tanp state close-wait | head -5
# Recv-Q Send-Q   Local Address:Port      Peer Address:Port  Process
# 1      0        10.0.3.14:44120        10.0.9.31:6379     users:(("java",pid=21874,fd=1042))
# 1      0        10.0.3.14:44122        10.0.9.31:6379     users:(("java",pid=21874,fd=1043))

# aggregate by peer address
sudo ss -tan state close-wait | awk 'NR>1 {print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn
#   20981 10.0.9.31
#      63 10.0.9.42

They are all the same peer, Redis. That means the Redis client pool is not returning connections, and that is what decides where the code review starts.

If files are the culprit instead, count which files they are.

sudo ls -l /proc/21874/fd | awk '$NF ~ /^\// {print $NF}' \
  | sed -E 's#/[^/]+$##' | sort | uniq -c | sort -rn | head -5
#    3204 /var/lib/app/uploads/tmp
#     512 /data/kafka/logs/orders-0
#      88 /usr/lib/x86_64-linux-gnu

There are 3204 files open under /var/lib/app/uploads/tmp. That means a code path creates temporary files and never closes them.

inotify and epoll are separate limits

The anon_inode family consumes RLIMIT_NOFILE too, but it has ceilings of its own. That is why failures happen while there is still plenty of fd headroom.

inotify has three ceilings.

sysctl fs.inotify
# fs.inotify.max_queued_events = 16384
# fs.inotify.max_user_instances = 128
# fs.inotify.max_user_watches = 65536

max_user_instances is how many inotify instances (inotify_init) a user may create, and exceeding it gives EMFILE. The error text is exactly the same as the one from ulimit -n, but the cause is different. max_user_watches is how many paths a user may watch, and exceeding it gives ENOSPC, that is, "No space left on device". It has nothing to do with the disk and yet it looks like a disk error.

What matters is that both ceilings are per user. Dozens of containers running under the same UID share a single budget. The classic pattern where you add pods to a node and watch-based tools suddenly start dying comes from exactly this.

Here is how to count the current usage.

# the processes holding the instances
sudo find /proc/[0-9]*/fd -lname 'anon_inode:*inotify*' 2>/dev/null \
  | cut -d/ -f3 | sort | uniq -c | sort -rn | head -5
#      42   9931
#       8   3311
#       2   1042

# the real watch count (the number of inotify lines in fdinfo is the watch count)
sudo grep -c '^inotify' /proc/9931/fdinfo/* 2>/dev/null | grep -v ':0$' | head
# /proc/9931/fdinfo/18:31204
# /proc/9931/fdinfo/22:8102

cat /etc/sysctl.d/60-inotify.conf
# fs.inotify.max_user_watches = 524288
# fs.inotify.max_user_instances = 1024
sudo sysctl --system

epoll has a separate ceiling as well.

cat /proc/sys/fs/epoll/max_user_watches
# 3618421

This value is the number of epoll watch entries a user may register, and the kernel derives it automatically from roughly 4 percent of low memory. A single entry costs tens of bytes of kernel memory, which is why it is not unlimited. Running into this limit in the field is rare, but for a proxy handling millions of connections it belongs on the checklist.

To summarize.

The real cause is usually a leak — how to confirm it and how to triage

There are only two situations that raising the limit actually fixes. One is when you genuinely need that many concurrent connections, the other is when the default is absurdly low. Everything else is a leak, and raising the limit on a leak only pushes the outage a few hours into the future.

The definition of a leak is simple. If fd usage grows monotonically regardless of load, it is a leak. If it rises and falls together with load, it is an ordinary capacity problem. A few minutes of sampling is enough to tell the two apart.

PID=21874
while sleep 60; do
  printf '%s  fd=%-7s estab=%-7s close_wait=%s\n' \
    "$(date +%H:%M:%S)" \
    "$(sudo ls /proc/$PID/fd | wc -l)" \
    "$(ss -tan state established | wc -l)" \
    "$(ss -tan state close-wait | wc -l)"
done
# 14:02:11  fd=18204   estab=8102    close_wait=9931
# 14:03:11  fd=19388   estab=8044    close_wait=11102
# 14:04:11  fd=20511   estab=7981    close_wait=12290
# 14:05:11  fd=21702   estab=8120    close_wait=13401

The connection count (estab) is flat around 8000 while fd climbs by 1200 per minute. And the increase matches the increase in close_wait exactly. The socket leak is confirmed, and once ss -tanp from the previous section has pinned down the peer too, the range of code to investigate is very narrow.

To catch which code path is doing the opening, you need tracing.

# files being opened and the call stack (bcc tools)
sudo opensnoop-bpfcc -p 21874

# frequency of the system calls that open fds
sudo bpftrace -e 'tracepoint:syscalls:sys_exit_openat /pid == 21874 && args->ret > 0/ { @[ustack] = count(); }'

# strace is expensive, so keep it short
sudo timeout 5 strace -f -e trace=openat,socket,close -p 21874 -c
# % time     seconds  usecs/call     calls    errors syscall
# ------ ----------- ----------- --------- --------- ----------------
#  61.02    0.184203          18     10233           socket
#  38.44    0.116041          21      5488           openat
#   0.54    0.001632           4       402           close

socket was called 10233 times while close was called 402 times. The ratio alone gives you the answer.

There are three triage options, and they come in an order.

# 1) restart the process — the surest, but the service goes down
sudo systemctl restart myapp

# 2) buy time by raising only the limit with prlimit (no restart)
sudo prlimit --pid 21874 --nofile=200000:200000

# 3) force a specific fd closed — the last resort, the application may break
sudo gdb -p 21874 -batch -ex 'call (int)close(1042)'

Number 3 closes an fd while the application still believes it is valid, so you have to accept the risk of data corruption or a crash. Use it only when it is the last option left in production.

Wrapping up — three limits and one real cause

If you remember only one thing, remember this. ulimit -n is the value of the shell you are in right now and has nothing to do with the service. What you should look at is always /proc/PID/limits.

Here is the diagnostic order, compressed.

1. Read the error text precisely. If in system is attached it is fs.file-max, otherwise it is the process limit.
2. Read the value actually in effect from /proc/PID/limits and the real usage from ls /proc/PID/fd | wc -l. Those two alone settle immediately whether this is a limit problem or not.
3. For a systemd service, set LimitNOFILE with systemctl edit rather than limits.conf, and restart. A reload will not change it.
4. For containers the runtime daemon decides the value. Check /proc/1/limits directly from inside the container.
5. Break the fd composition down into sockets, files and pipes. A pile of CLOSE-WAIT is definitive evidence of a leak.
6. If usage grows monotonically regardless of load, raising the limit is useless. Raising a limit only buys time to investigate; it is not a fix.
7. If you see "Too many open files" while fd headroom is fine, check the inotify instance limit. The wording is identical.