Operating Systems Interview: What Is a File Descriptor, and How Do You Debug Too Many Open Files?

Prompt and Applicable Roles

A Linux API service runs under steady traffic for several hours, then starts rejecting new connections. Calls to upstream services and attempts to open log files also begin to fail. You know that:

• the service process has a soft RLIMIT_NOFILE of 8,192 and a hard limit of 65,536;
• it starts with about 400 open file descriptors, then grows by roughly 25 per minute;
• just before the incident, /proc/<pid>/fd contains close to 8,192 entries and the logs contain EMFILE: Too many open files;
• restarting the process restores service immediately, but the growth returns;
• the host's /proc/sys/fs/file-nr remains far below /proc/sys/fs/file-max.

Explain the relationship among a file descriptor, the process file-descriptor table, and an open file description. Explain why regular files, sockets, pipes, and epoll all consume descriptors, and distinguish EMFILE from ENFILE. Then give a production-safe diagnostic sequence, decide whether this is an FD leak or legitimate concurrency beyond capacity, propose mitigation and durable fixes, and explain how you would prove that the incident is resolved.

This question fits backend, SRE, infrastructure, systems, and general software-engineering interviews. The values 8,192, 65,536, 400, and 25 per minute are hypothetical interview inputs, not universal recommendations. A real answer must also account for the service manager, container runtime, process permissions, and the application's concurrency model.

What the Interviewer Is Testing

First, can the candidate explain that a file descriptor is a small integer index inside a process, not the pathname, inode, or kernel object itself? A strong answer draws the chain “process FD table → open file description → file, socket, pipe, or another object” and knows that descriptors created by dup or inherited through fork can reference the same open file description.

Second, can the candidate separate limit scopes? EMFILE means the current process reached RLIMIT_NOFILE; ENFILE means the system-wide open-file limit was reached. Neither fs.file-max alone nor ulimit -n in an unrelated shell proves the failing process's effective limit.

Third, does the diagnosis cover count, composition, and trend? One lsof snapshot only shows one instant. Establishing a leak requires sampling the total and object types under stable traffic, then correlating growth with connection pools, request lifecycles, log rotation, child processes, and failure paths.

Fourth, can the candidate separate temporary capacity from root-cause correction? Raising a limit may create an incident-response window or be part of valid capacity planning, but it does not close resources whose ownership was lost. A fixed positive slope will eventually consume the larger limit too.

Finally, is there a verification loop? “The error stopped” is insufficient. A strong answer validates FD utilization and slope, object types, request errors, tail latency, pool state, and repeated lifecycle events under target peak traffic and injected failures.

Questions to Clarify Before Answering

• What is the exact errno? Confirm EMFILE or ENFILE from the application or system-call error instead of inferring the scope from the human-readable message.
• Which PID is failing? A supervisor, worker, sidecar, and short-lived child can inherit different limits and hold different resources.
• What launches the service? An interactive shell, systemd, a container runtime, and a process manager may establish different soft and hard limits. A shell command cannot retroactively change a running service.
• What moves with FD growth? Align it with concurrent connections, upstream requests, queue depth, log rotation, reloads, child count, and error rate.
• Which object type is growing? Sockets, regular files, pipes, anon_inode:[eventpoll], inotify objects, and deleted files point to different ownership paths.
• What is the legitimate capacity model? A high-concurrency proxy may validly need many sockets. A large count is not automatically a leak; it must match configured concurrency and settle when load falls.
• Can /proc be inspected safely? Reading another user's descriptors may be restricted by permissions and ptrace rules. Production investigation should use minimum necessary privilege and avoid long-running, high-overhead tracing.

30-Second Answer

“A file descriptor is a nonnegative integer index in a process's FD table. The table entry references a system-wide open file description, which stores the file offset and status flags and then points to a regular file, socket, pipe, or another kernel I/O object. EMFILE means this process reached RLIMIT_NOFILE; ENFILE means the host reached its system-wide open-file limit.

I would confirm the failing PID and errno, read /proc/<pid>/limits, count and classify /proc/<pid>/fd, sample the categories over time, and check /proc/sys/fs/file-nr. A fixed positive slope under steady traffic, reset by restart and concentrated in one object type, indicates a leak. A count that tracks concurrency, reaches a defined plateau, and falls afterward indicates capacity pressure. I can rate-limit, roll instances, and raise the real service limit after capacity validation, but the durable fix is resource ownership, failure-path cleanup, bounded pools, and correct inheritance. I prove it with FD utilization and slope, object release, and zero related errors at peak load.”

Step-by-Step Deep Dive

Step 1: Build the Three-Layer Reference Model

On success, open() returns a nonnegative integer. Linux normally selects the lowest descriptor number not currently used by that process. By convention, 0, 1, and 2 are standard input, standard output, and standard error. Later numbers are still only indexes in that process.

The core relationship is:

process FD-table entry → open file description → underlying object

The layers hold different state:

Two independent calls to open() on the same path normally create separate open file descriptions. A descriptor returned by dup() references the same open file description, so both descriptors share the offset and file-status flags. After fork(), corresponding parent and child descriptors also reference the same open file descriptions. Descriptor-specific flags such as close-on-exec are separate from that shared state.

This is why “FD 42” has no stable cross-process meaning and why grouping only by pathname can hide the problem. The same path can be independently opened many times, while different descriptor numbers can share one open state.

Step 2: Understand Which Resources Consume Descriptors

Unix-style interfaces expose many I/O resources through readable, writable, or waitable descriptors:

• regular files and directories;
• TCP, UDP, and Unix-domain sockets;
• anonymous pipes and named FIFOs;
• terminals, devices, and some pseudo-files;
• anonymous kernel objects such as epoll, eventfd, timerfd, signalfd, and inotify.

/proc/<pid>/fd contains one symbolic link per descriptor open in the process. A regular file usually shows a path. Sockets and pipes commonly appear as socket:[inode] and pipe:[inode]. Objects without a corresponding inode may appear as anon_inode:[eventpoll] or another anon_inode type.

Creating one epoll descriptor for an event loop does not mean thousands of monitored connections stop consuming descriptors. Every monitored socket still has its own FD. Conversely, a few anon_inode:[eventpoll] entries do not by themselves prove an event-loop leak; identify the category whose count is actually growing.

Step 3: Separate EMFILE, ENFILE, and the Limit Ceilings

RLIMIT_NOFILE has a soft limit and a hard limit. The kernel enforces the soft limit. The hard limit is the ceiling to which an unprivileged process may raise the soft limit. Linux defines the value as one greater than the largest descriptor number that the process may open.

The primary boundaries are:

The first field in /proc/sys/fs/file-nr is the number of allocated file handles; the third corresponds to file-max. This is a count of system-wide open file descriptions, so it need not equal the sum of every process's FD count: multiple descriptors can share an open file description.

The prompt explicitly reports EMFILE, while system-wide usage remains far below file-max. The primary path is therefore a per-process limit. Changing fs.file-max does not address this failure.

Step 4: Collect Low-Risk, Comparable Evidence First

After confirming the PID and permissions, start with a low-overhead /proc snapshot:

pid=12345

grep 'Max open files' /proc/"$pid"/limits
find /proc/"$pid"/fd -maxdepth 1 -type l 2>/dev/null | wc -l
find /proc/"$pid"/fd -maxdepth 1 -type l -exec readlink {} \; 2>/dev/null |
  sed -E 's/socket:\[[0-9]+\]/socket:[id]/; s/pipe:\[[0-9]+\]/pipe:[id]/' |
  sort | uniq -c | sort -nr | head -20
cat /proc/sys/fs/file-nr
cat /proc/sys/fs/file-max

/proc is a live view. The process may open or close descriptors during traversal, and short-lived entries may disappear. Treat these commands as trend diagnostics, not an atomic audit. Sample totals and categories at fixed intervals with timestamps, then align them with application metrics.

If sockets dominate the increase, inspect connection direction, destination, and TCP state. If regular files grow, group paths around logs, temporary files, or configuration reloads. If pipes grow, inspect child processes and IPC lifecycles. If anon_inode objects grow, locate event-loop, watcher, or timer registration and disposal.

Tools such as lsof -p <pid>, ss -tanp, or bounded system-call tracing can add detail after /proc and application metrics narrow the search. Long-running tracing can add production overhead and may still be incomplete without sufficient permission.

Step 5: Use Count, Composition, Slope, and Recovery to Classify the Problem

A high current count alone does not establish a leak. Ask four questions:

1. Does the count match expected concurrency? The budget includes listening sockets, accepted connections, outbound pools, files, pipes, event objects, and safety headroom.
2. Does the composition match the architecture? If an upstream pool is capped at 500 but descriptors for that destination reach 5,000, investigate return and close paths.
3. Does the slope remain positive under steady load? If request rate and concurrency are flat but FDs still grow by 25 per minute, the leak evidence is strong.
4. Does usage return to a plateau when load falls? Request-scoped files, short connections, and temporary pipes should be released. Long-lived pool connections may remain, but at an explicit bound.

In the prompt, the service starts around 400, grows at a fixed rate, resets on restart, and repeats. That pattern strongly favors a leak. Object classification is still required so that natural connection-pool warm-up is not mistaken for an unclosed file.

Insufficient capacity usually looks different: usage follows concurrency, reaches a plateau near configured pool and connection bounds, and falls after load or timeouts subside. The object mix matches the design, and failure appears only when legitimate peak demand exceeds the original budget. Raising limits, adding processes, or reducing per-connection cost may then be durable capacity changes.

Step 6: Trace Resource Ownership Through Common Leak Paths

A strong answer asks “who creates it, who owns it, and who closes it after failure?” instead of stopping at commands. Common causes include:

• an HTTP client does not close a response body, so the connection can neither be reused nor released promptly;
• a database, cache, or upstream connection is checked out but not returned after timeout or exception;
• log rotation or configuration reload repeatedly opens a new file without closing the old handle;
• each request creates a timer, watcher, pipe, or event object but cancellation skips cleanup;
• parent and child processes keep unused ends of standard streams or IPC pipes open;
• a descriptor survives exec unexpectedly, so another process keeps a resource alive;
• retry logic creates a new connection while the earlier attempt remains pending.

Make lifetime ownership visible in the code structure. Use defer, finally, RAII, or the framework's scoped resource management. Establish cleanup immediately after acquisition. Bound pools, concurrency, queues, and retries. Route cancellation, timeout, and early-return paths through the same cleanup logic.

In a multithreaded program, calling open() and setting FD_CLOEXEC in a later operation creates a window in which another thread can fork and exec. Where supported, set O_CLOEXEC atomically at creation time. That prevents an inheritance race; it does not replace ordinary close() ownership.

Step 7: Separate Incident Mitigation from the Durable Fix

Depending on risk, incident mitigation can include:

• rate-limiting new work or reducing per-instance concurrency before the process loses the ability to open logs, control connections, and configuration files;
• rolling the leaking instances while preserving at least one diagnostic sample and avoiding a simultaneous restart;
• raising the actual service process's soft and hard limits after checking memory, kernel overhead, and downstream capacity;
• adding instances so legitimate concurrency is distributed across more processes.

Changing ulimit -n in the current terminal does not affect an already running service. Modify the service manager, container, or runtime that actually creates the process, restart it, and verify /proc/<new-pid>/limits. A configuration file edit alone is not proof that the new limit is active.

The durable fix must target the confirmed growing object: complete close paths, bound pools and concurrency, repair rotation or watcher lifecycles, set useful timeouts, prevent unintended inheritance, and expose current counts plus create/release counters for major resource categories.

Step 8: Prove the Fix with a Capacity Budget and Slope

Create a per-instance FD budget:

baseline FDs + peak inbound connections + peak outbound connections + pools and files + IPC/event objects + safety headroom

The budget must come from the real architecture and load tests. There is no universal utilization percentage for every service. The limit must also preserve room for diagnostics, health checks, logging, and control connections during an incident.

Verification should cover at least:

1. under target peak traffic and injected failures, total FD usage rises to a stable plateau;
2. after load falls and timeouts expire, short-lived descriptors return to the expected baseline;
3. the formerly growing object type no longer has a persistent positive slope;
4. EMFILE, ENFILE, connection failures, and file-open failures remain at zero;
5. request p95/p99, pool wait time, and retry volume do not regress because of overly aggressive limits;
6. repeated deploys, log rotations, configuration reloads, and child-process launches do not create staircase growth;
7. monitoring includes process_open_fds, process_max_fds, utilization, growth rate, and major object pools.

“A ten-minute load test passed” can miss a slow leak. Run long enough to cover the amount of growth that caused the original incident, or amplify the suspect path and prove that create and release counters balance.

Strong Sample Answer

“I would first confirm that the errno is EMFILE and that the API worker itself is failing. A file descriptor is a nonnegative index in the process FD table. The table entry references a system-wide open file description, which stores the offset and file-status flags and then refers to a regular file, socket, pipe, or anonymous kernel object. dup and fork can make multiple descriptors share one open file description, so process FD count and system-wide file-handle count are different metrics.

EMFILE means this process reached RLIMIT_NOFILE; ENFILE means the host reached file-max. Here, the soft limit is 8,192, the pre-incident count is close to that value, and file-nr is far below file-max, so changing the system-wide limit is not targeted.

I would read /proc/<pid>/limits, then sample the count and symlink targets in /proc/<pid>/fd, grouped into sockets, pipes, regular files, and anon_inode objects. I would align each category's slope with inbound connections, upstream pools, file rotation, child processes, and errors. Starting near 400 and growing by 25 per minute under steady traffic, resetting on restart, strongly suggests a leak. If one upstream's sockets dominate, I would inspect response-body close, timeout cancellation, and pool-return paths instead of assuming every connection reflects valid traffic.

For mitigation, I would rate-limit and roll instances while preserving a diagnostic sample. If capacity analysis permits, I can temporarily raise the limit in the real service manager or container configuration, but I must verify the new process's /proc/<pid>/limits. The durable fix makes resource ownership and cleanup structural, bounds pools, retries, timers, and watchers, and prevents unintended inheritance with close-on-exec.

I would verify under target peak load and failure paths. FD usage should reach the planned plateau and return toward baseline when load falls. The formerly growing category must stop accumulating, EMFILE must remain at zero, and tail latency and pool waits must not regress. Those results prove a fix; a restart or larger limit only proves that exhaustion was delayed.”

Common Mistakes

• Treating an FD as a path or inode → it is an index in a process table → explain the FD table, open file description, and underlying object.
• Assuming only disk files consume FDs → sockets, pipes, epoll, timers, and watchers also use them → classify /proc/<pid>/fd targets.
• Changing fs.file-max for every Too many open files error → EMFILE and ENFILE have different scopes → confirm errno and the failing PID first.
• Using the current shell's ulimit -n as the service limit → the running service may have been launched elsewhere → read /proc/<pid>/limits.
• Declaring a leak because the count is high → valid high concurrency can create a high plateau → compare the capacity model, type mix, slope, and recovery after load.
• Only raising 8,192 to 65,536 → a fixed leak slope will consume the new limit too → treat the increase as validated capacity or temporary headroom.
• Checking only the normal return path → timeouts, cancellation, retries, and early returns commonly skip cleanup → define creator, owner, and failure-path cleanup for every resource.
• Using one lsof snapshot → a snapshot cannot prove accumulation → sample at fixed intervals and correlate with workload and pools.
• Accepting temporary error disappearance → restart and higher limits can delay recurrence → verify object type, slope, recovery, and repeated lifecycle events.

Follow-Up Questions

Follow-up 1: Why does ulimit -n show 65,536 while the service still fails at 8,192?

ulimit normally reports the current shell and affects descendants created afterward. A service already launched by systemd, a container runtime, or another process manager is not retroactively changed. A supervisor and its workers may also have different settings. Read the failing PID's /proc/<pid>/limits, update the actual launch boundary, and verify the new PID.

Follow-up 2: Why can two descriptors for the same file affect each other's read position?

If they came from dup, or from the same descriptor before fork, they reference the same open file description and therefore share the offset and file-status flags. If the program calls open() twice, the same path normally produces two independent open file descriptions with independent offsets. A shared pathname does not prove shared open state.

Follow-up 3: Why can a process still use a file through an FD after the file was deleted?

The descriptor references an open file description; I/O does not resolve the pathname again each time. Removing the directory entry does not invalidate existing references. The underlying object can remain until the last reference is closed. Log rotation that unlinks an old file without making the process close it can consume both a descriptor and disk space.

Follow-up 4: Why might raising RLIMIT_NOFILE to a very large number fail?

An unprivileged process cannot raise the soft limit above the hard limit or freely raise its hard limit. Linux also caps RLIMIT_NOFILE with /proc/sys/fs/nr_open. A service manager, container, or permission boundary may add constraints. Check startup errors and the new process's effective limits after every change.

Follow-up 5: If one epoll instance watches many connections, why can the process still exhaust FDs?

The epoll instance itself consumes one FD and appears as anon_inode:[eventpoll]. Every monitored socket remains a separate FD. epoll makes waiting on many connections efficient; it does not combine thousands of sockets into one descriptor or close them for the application.

Follow-up 6: How would you alert on FD usage?

Monitor current open descriptors, the process maximum, utilization, and growth rate per instance. Utilization detects proximity to exhaustion; slope detects slow leaks earlier. Connection pools, files, and watchers should expose their own current counts too. Set thresholds from peak budgets, scaling time, and incident-response lead time instead of copying a universal percentage.