Linux Interview: How Do You Diagnose an OOM Kill?

Prompt and When It Applies

A 64 GiB Linux node still has 18 GiB of MemAvailable, but one container repeatedly exits over six hours. The runtime records OOMKilled and exit code 137. The workload's cgroup v2 memory.max is 4 GiB; before the failure, memory.current approaches that limit and oom_kill increases in memory.events. During the same period, anon in memory.stat grows from 1.2 GiB to 3.5 GiB while file is about 280 MiB. Explain how you would prove the trigger, distinguish a cgroup OOM from a global OOM, locate the source of growth, contain the incident safely, and prevent recurrence.

The 64 GiB, 18 GiB, 4 GiB, six-hour window, and usage figures are interview assumptions, not capacity benchmarks. Assume Linux uses cgroup v2 and that the runtime status and cgroup files refer to the same failure window. This question belongs to general because it tests operating-system reclaim, cgroup accounting, OOM evidence, and process handling; the container platform only supplies the setting.

Public Linux and operating-system interview material published in 2026 directly includes OOM scenarios and asks candidates to reason from kernel logs, resource limits, and process behavior. A complete answer must go beyond “add memory” or “I saw 137”: it needs an evidence chain, safe containment, source attribution, and a reproducible test showing that the repair works.

What the Interviewer Is Evaluating

The first signal is whether the candidate interprets evidence correctly. Under Bash's exit-status convention, 137 can be 128 plus signal 9, so it says that the process ended due to SIGKILL. An administrator, a timeout controller, or the OOM killer can all send that signal. The runtime's OOMKilled reason, an increase in memory.events:oom_kill in the same window, and kernel logs are what narrow the cause to OOM.

The second signal is whether the candidate identifies the resource domain. memory.max is a cgroup hard limit. If usage reaches it and reclaim cannot reduce the charge, the kernel can invoke OOM inside that cgroup. The node can still have 18 GiB available. A global OOM instead arises from node-wide allocation pressure, with a different victim pool and different recovery actions.

The third signal is memory-accounting discipline. Looking only at one process's RSS misses descendants, page cache, tmpfs, shared memory, socket buffers, and kernel data charged to the same cgroup. A strong answer reconciles memory.current, memory.peak, memory.stat, per-process measurements, and application profiles instead of treating VSZ, RSS, and the cgroup charge as equivalent.

Finally, the interviewer is evaluating incident judgment. Temporarily raising a limit may restore service, or it may turn a continuing leak into a node-wide failure. The answer should explain when to shed load, restart, scale, or raise the limit; how to distinguish a normal peak from a leak; and how load, soak, and failure tests validate the policy.

Questions to Clarify Before Answering

• Do the records refer to the same container instance and failure window? An old OOMKilled status, a current

cgroup counter, and a different exit cannot form a causal chain. Align container ID, start and exit times, and path.

• Is the host actually using cgroup v2? v1 and v2 expose different files and semantics. Confirm whether the path

is the container's cgroup or a parent that contains several descendant workloads.

• Did oom_kill or only oom change? oom says the limit was reached and an allocation was about to fail;

oom_kill records a process actually killed by an OOM killer. Cross-check hierarchical counts with memory.events.local.

• Was the node also under global memory pressure? Inspect kernel logs, MemAvailable, swap, PSI, eviction events,

and neighboring workloads. A cgroup OOM and node pressure can occur near each other.

• Does the 4 GiB limit belong to the container, the Pod, or a parent cgroup? A parent limit can constrain a child

before the child's own limit is reached. Walk the hierarchy and inspect every effective boundary.

• What workload correlates with growth? Request concurrency, queue depth, batch size, cache keys, connections,

input size, and deployment version help separate a working set that recedes from retained memory that grows with time.

• How many processes share the cgroup? Sidecars, workers, and spawned children all contribute to the charge.

Profiling only the main process can miss the owner of the growth.

• What are the recovery and data-integrity requirements? Killing one member of a multiprocess workload can leave

inconsistent state. Restart, load shedding, traffic shifting, and group termination depend on idempotency and RTO.

30-Second Answer Framework

“I would treat 137 as a SIGKILL clue, not an OOM conclusion. I would align the container ID and failure time, then check the runtime's OOMKilled reason, the target cgroup's memory.events.local, memory.current, memory.max, and kernel logs. The node has free memory, but the cgroup hit its limit and oom_kill increased, so the evidence points to a cgroup OOM. I would shed or shift load and preserve evidence, raising the limit temporarily only after checking node headroom. Then I would break down anonymous, file, shared, and kernel memory with memory.stat, and correlate per-process profiles with workload metrics to separate a leak, unbounded cache, concurrency spike, or undersized limit. After the fix, I would run representative peak, long soak, and controlled-limit tests and alert on memory.high, pressure, peaks, and OOM events.”

Step-by-Step Deep Dive

Step one: turn the termination into one timeline.

Record the container ID, PID, start and exit times, deployed version, restart count, and cgroup path. Exit 137 commonly corresponds to SIGKILL in shell and container status, but it does not prove OOM: kill -9, a platform timeout, or a node agent can produce the same result. Correlate reason: OOMKilled, the delta in cgroup events, and kernel messages around the same second.

Prefer memory.events.local so a parent's hierarchical counter does not mix in descendant events. If oom_kill changes from 7 to 8 at this exit, memory.current approaches 4 GiB, and the kernel log reports a memory-cgroup OOM, that is a coherent cgroup OOM chain. If there is only 137, with no counter change or OOM runtime reason, investigate manual signals, health-check timeouts, systemd-oomd, eviction, and runtime termination instead.

Step two: identify the OOM resource domain.

memory.max constrains accounted memory for the cgroup and its descendants. When it is reached and direct reclaim cannot satisfy an allocation, the kernel can choose a victim only from that cgroup. The host's 18 GiB of MemAvailable therefore does not protect the container. Walk upward through parent cgroups to find the boundary whose memory.max and event counters were hit.

Global OOM has different evidence: node memory and swap approach exhaustion, memory PSI rises, kernel logs include global memory state and a victim, and other workloads are affected. Under node pressure, the platform may evict a Pod first. Remediating cgroup OOM focuses on workload usage and its limit; global OOM also requires correcting node overcommit, requests and reservations, system daemons, and workload placement.

Step three: explain where the 4 GiB went.

Start with memory.current and memory.peak, then break the charge down with memory.stat. Here, anon rises from 1.2 GiB to 3.5 GiB over six hours while file is around 280 MiB. That prioritizes heaps, anonymous mappings, and the processes that own them, but it does not yet prove a leak. Kernel documentation says cgroup accounting also covers page cache, tmpfs and shared memory, kernel structures, and socket buffers; a parent includes descendants.

Put the total beside each process's RSS, PSS, anonymous mappings, process and thread counts, runtime heap metrics, and business measurements on one timeline. If live objects and the application heap grow together, investigate retained references or an unbounded cache. If the heap is stable but RSS remains high, inspect allocator fragmentation, native libraries, or mmap. Growth in file, shmem, sock, or slab instead points toward tmpfs files, cache, connection backlog, or kernel objects.

VSZ is virtual address space, not physical residency or the cgroup charge. A single snapshot is also insufficient. A healthy batch workload may peak and fall after reclaim; a leak generally raises the baseline over repeated cycles at comparable load. Compare the slope and the post-release steady state at the same throughput.

Step four: make each hypothesis falsifiable.

Keep a short candidate list and state a prediction for each one. With a concurrency peak, memory should track in-flight requests and recede when they finish. With an unbounded cache, entry count and anon rise together and stabilize when capacity is capped. With consumer backlog, queue depth, batch objects, and memory move together. With an undersized limit, the stable representative working set repeatedly approaches the boundary without an upward long-run slope.

Use a heap profile, allocation profile, or object histogram appropriate for the runtime, but account for collection overhead. Start with low-risk production metrics or a traffic-shifted replica; before taking a dump, verify disk space, privacy, and pause cost. Change one factor at a time in version comparisons or traffic replay. Raising the limit, disabling cache, and lowering concurrency simultaneously would hide the cause.

Step five: separate containment from the durable repair.

Protect users and the node first: cap concurrency into the instance, pause memory-heavy batches, shift traffic, or roll back to a known-safe version. If replay is safe and multiprocess state will remain consistent, a restart quickly releases memory. If killing one worker leaves shared state corrupt, consider terminating the workload as a unit. In cgroup v2, memory.oom.group=1 asks the OOM killer to treat the cgroup as indivisible, but its recovery semantics must be tested.

Raise memory.max temporarily only after measuring node headroom, protection for neighbors, and the expected peak. Attach an expiry, monitoring, and rollback threshold. For continuing growth, a larger limit only delays the next OOM. Durable fixes may cap a cache, release references, limit batches and concurrency, repair child-process lifecycles, or resize requests and limits from a measured working set and burst allowance.

Do not “solve” the incident by assigning a normal service oom_score_adj=-1000. That exempts it from OOM selection and may force the kernel to kill more important or more numerous processes; reserve it for essential system tasks after an end-to-end failure-policy review. Swap also changes reclaim and latency behavior. It may absorb a short burst but does not repair unbounded growth.

Step six: validate capacity and create earlier signals.

Replay representative traffic under production-equivalent cgroup accounting and limits. Cover steady load, peaks, large inputs, backlog recovery, and multiprocess behavior. Use a short load test for the peak, a long soak test for the growth slope and post-release baseline, and a controlled lower limit to exercise alerting, termination, restart, and data integrity. Success means defined headroom at target throughput and latency, a receding memory.current, no new oom_kill, and no worsening of node pressure or neighbors.

Use memory.high as a control boundary before the hard limit: exceeding it throttles the cgroup and drives direct reclaim without directly invoking OOM, leaving time to alert or automate containment. Monitor the memory.current/memory.max ratio, memory.peak, the high, max, oom, and oom_kill events, memory PSI, restart rate, application heap, and growth slope. Limits should come from peak and soak measurements and be reviewed when versions, concurrency, or input distributions change.

High-Quality Sample Answer

“I would first prove whether OOM caused this termination. Exit 137 says the process received SIGKILL; manual termination and timeouts can do that too. I would align container ID, exit time, and cgroup path, then examine the runtime's OOMKilled reason, the delta in memory.events.local, and kernel logs. If oom_kill increased at this exit, memory.current touched 4 GiB, and the log identifies a memory cgroup, that supports a cgroup OOM.

That also explains why the node can have 18 GiB available. memory.max is the cgroup's hard boundary, and failed reclaim makes the kernel choose within that resource domain without first exhausting the host. I would still inspect parent cgroups, node PSI, swap, and global OOM logs to exclude concurrent node pressure or platform eviction.

For containment, I would shed or shift memory-heavy traffic and preserve pre-failure metrics and a low-overhead profile. If restart is safe, I would restore a known version. I would raise the limit temporarily only after checking node headroom and neighboring workloads, with an expiry and rollback condition. More memory is not the durable fix.

For diagnosis, I would reconcile the cgroup total using memory.current, memory.peak, and memory.stat, then inspect RSS, PSS, anonymous mappings, and runtime heaps per process. Here anon grew from 1.2 GiB to 3.5 GiB while file memory was about 280 MiB, so I would prioritize heap, anonymous mmap, child processes, and the allocator, but prove the owner with profiles. I would correlate memory with concurrency, queue depth, cache entries, batch size, and version. A rising baseline at comparable load suggests a leak; a stable working set that recedes suggests a normal peak or a low limit.

After the repair, I would run peak load and a long soak under the same cgroup limit, verifying that memory recedes, event counters stop increasing, and latency and throughput meet targets. I would then set memory.high and alerts before the hard boundary, monitor peaks, pressure, OOM events, and growth slope, and size concurrency, cache, and capacity from the measured working set. For a multiprocess service I would also test data integrity after a partial kill before deciding whether OOM should terminate the whole group.”

Common Mistakes

• Declaring OOM from 137 alone → SIGKILL can come from an operator or timeout controller → **Correlate runtime

reason, cgroup events, and kernel logs.**

• Ruling out OOM because the host has free memory → A cgroup can hit its hard limit while the node has headroom →

Identify the resource domain first.

• Looking only at one process's RSS → Descendants, files, shared memory, and kernel charges also count → **Reconcile

the total and break down memory.stat.**

• Treating VSZ as physical usage → Address-space size is not residency or accounted memory → **Cross-check RSS/PSS,

mappings, and cgroup metrics.**

• Calling any anon growth a leak → A normal working set or batch peak can also grow → **Compare slopes and the

post-release baseline at equivalent load.**

• Doubling the limit immediately → This can delay a leak and consume node safety margin → **Require capacity

evidence, an expiry, and rollback thresholds.**

• Making the application immune to OOM → Damage can shift to other tasks or the node → **Change oom_score_adj

only as part of an end-to-end failure policy.**

• Running only a one-minute load test → A short test misses slow leaks and fragmentation → **Run both peak and

long soak tests.**

• Alerting only at the hard limit → Responding at memory.max is usually too late → **Use memory.high, PSI, and

growth slope for earlier action.**

• Assuming recovery after one worker dies → Multiprocess shared state may be inconsistent → **Test group

termination, restart, and data-recovery semantics.**

Follow-Ups and How to Respond

Follow-up 1: Exit code 137 is present, but oom_kill did not increase. What do you inspect next?

First confirm the cgroup path and event time, and read memory.events.local so that you do not compare a parent or a new instance. If OOM evidence is still absent, inspect the runtime termination reason, deployment or health-check timeout, operator audit trail, systemd-oomd, node eviction, and kernel logs. Exit 137 narrows the outcome to SIGKILL; it does not identify the sender.

Follow-up 2: How do you distinguish a memory leak from an undersized limit?

Compare repeated cycles at similar throughput, inputs, and concurrency. A leak generally raises the baseline or live object count and does not recede at low load. An undersized limit is more likely to fail at a repeatable peak and return to a stable working set afterward. Validate with heap or allocation profiles, cache entries, child processes, batch size, and memory.stat; do not rely on one graph. Both conditions can coexist.

Follow-up 3: When is it valid to increase memory.max?

Increase it when representative peak and soak tests show healthy behavior whose required working set exceeds the current limit, while node capacity, reservations, and neighbor protection still leave margin. An incident-time increase needs an expiry, monitoring, and rollback threshold. If usage grows without bound, a higher limit is containment only; also shed load and repair the growth source.

Follow-up 4: How should memory.high and memory.max work together?

memory.high is a throttling and direct-reclaim boundary; crossing it does not directly invoke the OOM killer, so it can provide an observation and automation window. memory.max is the final isolation boundary and can invoke cgroup OOM if reclaim fails. Set both from measured working sets, peaks, latency tolerance, and node margin, and alert separately on high, max, oom, and oom_kill events.

Follow-up 5: Why might a multiprocess service use memory.oom.group?

If killing one worker leaves inconsistent shared state, locks, or unfinished transactions, treating the cgroup as an indivisible workload can make failure and recovery deterministic. Before enabling it, test whole-group restart time, task idempotency, and data recovery. Tasks with oom_score_adj=-1000 are exceptions, so also verify that no partial workload remains.