When 1024 is Not Enough
A default Linux system often sets ulimit -n (open files) to 1024. This is a sensible default for interactive user sessions, but it is catastrophically low for any network-facing service.
The Scenario:
You deploy a new Nginx web server. It works perfectly in testing with a handful of simulated users. On launch day, traffic spikes to 2,000 concurrent users. Suddenly, users report "Connection Refused" errors. You SSH in and check: Nginx CPU usage is under 10%, RAM is 60% free, the disk is idle. Everything looks fine — yet users cannot connect.
The Cause:
Nginx hit the 1,024 open file descriptor limit. Each active HTTP connection consumes one file descriptor. At connection 1,025, the kernel returned EMFILE ("Too many open files") to Nginx. Nginx had no choice but to reject the connection. The error appeared in Nginx's error log, but nobody was watching it.
The Diagnosis:
The actual limit enforced on the running Nginx master process — not just the shell's own default — lives in /proc/<pid>/limits, and the number of descriptors it currently has open is a simple count of entries under /proc/<pid>/fd. Comparing the two shows the process sitting right at its ceiling.
The Fix:
For production servers, set LimitNOFILE=65535 in the Nginx systemd unit file, reload the daemon, and restart the service. It's also worth confirming the kernel's system-wide file descriptor limit (fs.file-max) is high enough to accommodate the raised per-process limit — a low system-wide ceiling can silently cap an otherwise-correct per-process fix.
The lesson: resource exhaustion failures are often invisible in standard monitoring dashboards. CPU and RAM metrics look healthy while the service is completely broken. Always monitor file descriptor usage (/proc/<pid>/fd) for network-facing services.