Linux Interview: How Do Signals and Graceful Shutdown Work?

Prompt and Applicable Context

A Linux HTTP service runs in a Kubernetes container. During a rolling update it receives SIGTERM with a 30-second grace period while 200 requests are in flight, two child workers are running, and a background-job consumer is active. Design the shutdown protocol from signal delivery through process exit. Explain the behavior of SIGTERM and SIGKILL, signal-handler safety, traffic draining, job and child-process handling, deadlines, and verification.

The 30 seconds and 200 requests are interview scenario inputs. Kubernetes commonly uses a 30-second default Pod termination grace period, but production values should come from measured request duration, cleanup time, workload semantics, and availability requirements. In this scenario, use an internal 25-second drain deadline and reserve five seconds for final cleanup and scheduling variance. That split is an engineering choice, not a platform guarantee.

The core problem is a Linux process-lifecycle protocol. Kubernetes and HTTP provide the operating context. A strong answer follows the signal from the kernel to the intended process, turns it into a safe state transition, controls admission and in-flight work, and proves that termination stays within the deadline.

What the Interviewer Evaluates

First, the interviewer checks signal semantics. A process may catch, block, or ignore SIGTERM; its default action is termination. Once an application catches it, the application must eventually terminate itself. SIGKILL cannot be caught, blocked, or ignored, so it provides no cleanup opportunity. Repeated standard signals are not a durable queue: multiple pending instances of the same standard signal may coalesce.

Second, the interviewer checks whether the signal reaches the service. In a container, a shell-form entry point can leave /bin/sh -c as PID 1 and prevent the executable from receiving the expected SIGTERM. The service should normally be the exec-form entry point, or a wrapper should finish with exec. A service that creates children must also forward termination and reap them, or run with an appropriate tiny init when it cannot perform PID 1 duties itself.

Third, the interviewer looks for an async-signal-safe boundary. A raw C signal handler interrupts normal execution at an arbitrary instruction. Calling printf, allocating memory, taking a mutex, closing an application object graph, or flushing a client library can deadlock or corrupt state. The handler should only publish a minimal notification using signal-safe operations; the normal control path owns cleanup.

Fourth, the interviewer evaluates shutdown ordering. The service should become unready and stop accepting new work, then drain or cancel admitted work under a deadline. It must stop job intake without losing lease ownership, keep shared dependencies open until users of them finish, terminate and reap children, perform bounded telemetry flushes, and exit before the orchestrator escalates to SIGKILL.

Finally, the interviewer wants operational proof. A handler unit test alone cannot prove container PID layout, endpoint removal, connection behavior, job redelivery, child reaping, or compliance with the grace period. The answer should include container-level and rollout-level tests with observable pass criteria.

Questions to Clarify Before Answering

• Who sends the signal, and to which PID? Confirm the container runtime, configured stop signal, entry point, wrappers, and whether the application is PID 1.
• What consumes the 30 seconds? A Kubernetes preStop hook runs within the same termination grace period. Its duration reduces the time left for application draining.
• What does “in flight” mean? Separate accepted requests, keep-alive connections with no request, streaming responses, upgraded connections, and queued application work. They need different completion policies.
• Can the service reject new work immediately? Identify the readiness probe, load balancer, listener, service mesh, and any direct callers. Readiness propagation is not instantaneous.
• What are the request-duration and retry contracts? A short idempotent read may finish; a long upload or side-effecting write may need cancellation, handoff, or an idempotency key.
• How are background jobs owned? Clarify acknowledgement timing, visibility timeout or lease, heartbeat behavior, redelivery, and idempotency. “Stop the worker” is unsafe without this contract.
• Who owns the child workers? Determine whether the parent can signal a process group, whether children have their own shutdown protocol, and who calls waitpid.
• Which runtime handles signals? A dedicated sigwait thread, an event-loop callback, and a raw C handler have different safety boundaries. State the actual runtime behavior.
• What outcome defines success? Set limits for new admissions, completed and canceled work, duplicate side effects, zombie processes, exit time, and forced-termination rate.

30-Second Answer Framework

“I verify that an exec-form entry point delivers SIGTERM to the application. The raw handler only wakes the normal control path. That path fails readiness, stops request and job admission, and drains accepted work to a 25-second internal deadline. It then cancels leftovers safely, terminates and reaps both children, performs bounded final flushes, and exits before the 30-second grace period ends. SIGKILL cannot run cleanup. I test the built container under concurrent requests and jobs, checking the admission cutoff, retry behavior, child reaping, and exit time.”

This framework establishes the control flow. The detailed answer must also cover multithreaded signal delivery, interrupted system calls, repeated shutdown requests, and the difference between endpoint removal and listener admission.

Step-by-Step Deep Dive

Begin with the delivery path. Use the executable form of ENTRYPOINT or CMD so the application receives the runtime’s stop signal directly. If a wrapper is needed for setup, end it with exec "$@". Inspect the running container rather than trusting the Dockerfile: verify PID 1, its command line, parent-child relationships, and the configured stop signal. A rollout test that sends SIGTERM to the container is the decisive check.

Initialize the wake-up mechanism before installing the handler, and install the handler before advertising readiness. A C-shaped sketch can use a nonblocking self-pipe:

static volatile sig_atomic_t stop_requested = 0;
static int wake_fd; /* initialized as nonblocking before sigaction */

static void on_term(int signo) {
  int saved_errno = errno;
  stop_requested = 1;
  const unsigned char byte = 1;
  (void)write(wake_fd, &byte, sizeof byte);
  errno = saved_errno;
}

write is async-signal-safe. The nonblocking descriptor prevents the handler from waiting if notifications already fill the pipe; the flag preserves the state even if the wake-up write cannot add another byte. The handler does not log, allocate, lock, wait for children, or call application clients. The ordinary event loop drains the pipe and advances an idempotent shutdown state machine.

An alternative for a multithreaded service is to block termination signals before creating worker threads, then let one dedicated thread call sigwait or use signalfd on Linux. Signal disposition is process-wide, while each thread has its own signal mask. A process-directed signal may be delivered to any thread that does not block it. Centralized synchronous signal handling removes the asynchronous handler from application code, provided signal masks are established consistently before threads start.

Do not rely only on an interrupted system call to wake the service. Depending on the interface and SA_RESTART, a blocking call may resume automatically or return EINTR. The self-pipe, event descriptor, runtime signal channel, or dedicated signal thread creates an explicit wake-up path. Every blocking wait in shutdown should also have a deadline.

Drive the service through explicit states:

RUNNING
  --SIGTERM--> QUIESCING
  --admission closed--> DRAINING
  --work finished or 25 s reached--> FINALIZING
  --children reaped and bounded flush complete--> EXITED

The transition from RUNNING must be atomic and idempotent. The first SIGTERM records the start time and deadline. A second SIGTERM should not start another cleanup graph or close the same resource twice. The team can choose whether it only records a repeat or shortens the drain, but the behavior must be documented and tested.

At QUIESCING, make readiness fail and immediately stop new application admission. Kubernetes marks a terminating endpoint unready, but control-plane and proxy propagation takes time. Close the listening socket, disable accepts, or have the admission layer return a retryable response for requests that have not crossed the accepted-work boundary. Existing accepted connections may remain open for draining. Handle HTTP keep-alive explicitly so an idle old connection cannot submit unlimited new requests after shutdown starts.

Stop the background consumer from fetching new jobs at the same boundary. For an already leased job, continue only when it can finish safely before the deadline. Otherwise stop heartbeats or release/nack the lease using the queue’s contract so another worker can retry. Acknowledgement must follow durable completion. Side effects need idempotency keys or transactional state transitions because forced termination can occur after an external write and before acknowledgement.

At DRAINING, track admitted work with a counter or registry. Allow requests to finish while their dependencies remain available. Do not close the database pool, cache client, or telemetry exporter while request handlers still use them. At the 25-second internal deadline, cancel remaining work according to the protocol: stop streaming, propagate cancellation, return a defined response where possible, and leave retriable state consistent. Reserve the remaining five seconds for cancellation callbacks, child reaping, final state writes, and runtime scheduling variance.

For the two child workers, stop their input first, send their agreed termination signal, and wait with a deadline. If the parent owns a dedicated process group, it can signal that group while avoiding unrelated processes. Reap every exited child with waitpid so no zombie remains. A tiny init can provide signal forwarding and reaping for a container whose application cannot do so, but it does not define the application’s job or request semantics.

At FINALIZING, emit the final shutdown metrics and flush logs or traces under a strict time budget. Observability helps explain forced exits, yet an unavailable telemetry backend must never consume the entire grace period. Close remaining resources in dependency order and return exit status zero for a completed graceful shutdown. If the process misses the platform deadline, Kubernetes ultimately asks the runtime to send SIGKILL; no handler, deferred block, or shutdown hook runs after that point.

Use signals as control notifications, not work messages. Standard signals can coalesce, contain little payload, and may arrive in surprising code locations. Put work, retries, and durable ownership in queues or state stores. The signal only starts or escalates the local lifecycle transition.

Verification should exercise the same boundary used in production:

1. Build the real image, inspect PID 1, start the service, and send the container SIGTERM rather than invoking an internal shutdown endpoint.
2. Hold 200 mixed-duration requests, including work that finishes within 25 seconds and work that must be canceled. Assert that no request is newly admitted after the cutoff and every admitted request has a recorded terminal outcome.
3. Confirm readiness becomes false and the old Pod receives no new rollout traffic after endpoint propagation. Also test a direct connection so listener admission is verified independently.
4. Run leased background jobs across termination. Verify completed work is acknowledged once, unfinished work becomes eligible for retry, and duplicate delivery cannot duplicate the business effect.
5. Verify both child workers receive termination, exit by their deadline, and are reaped. Inspect the process table for zombies.
6. Send a second SIGTERM and prove cleanup remains idempotent. Separately send SIGKILL to prove no cleanup is assumed and recovery contracts still protect durable work.
7. Record shutdown_started, admission state, in-flight count, drain-deadline breaches, child status, exit time, and forced termination. Fail the test if graceful exit reaches 30 seconds.

High-Quality Sample Answer

“I would begin at the container boundary. I would use an exec-form entry point and inspect the image at runtime to verify that the HTTP service is PID 1 or sits behind an init that forwards signals. A shell wrapper would end with exec, so SIGTERM cannot stop at the shell.

Before readiness becomes true, the service would install its signal path. In a raw C handler I would only set a sig_atomic_t flag and write to a nonblocking self-pipe. Logging, mutexes, memory allocation, database calls, and child waits stay out of that handler. In a multithreaded implementation I would prefer blocking termination signals before creating workers and consuming them from one sigwait thread. Either design wakes the normal control loop explicitly instead of depending on EINTR.

The first SIGTERM atomically moves the service from running to quiescing and fixes an internal deadline 25 seconds later. The service immediately fails readiness, closes or disables new admission, prevents keep-alive connections from starting more requests, and stops fetching background jobs. Readiness removal and closing admission are both necessary because endpoint propagation is asynchronous.

The 200 accepted requests can continue while their database and cache clients remain open. I track them directly. Requests that complete before 25 seconds return normally. At the internal deadline, I cancel the rest through the application protocol and preserve retriable state. For the job consumer, I acknowledge only durable completion; unfinished leased jobs are released or allowed to expire according to the queue contract, and their side effects use idempotency keys.

I then terminate the two child workers through their defined signal path and reap them with a bounded wait. Only after request and child users are gone do I close shared clients. Logs and traces receive a small bounded flush budget. A successful path exits zero before 30 seconds. If that deadline is missed, SIGKILL can terminate the process and no cleanup code will run, so durable correctness cannot depend on the final hook.

For verification, I run the built container with 200 concurrent mixed-duration requests and active leased jobs, then send it SIGTERM. I assert the actual process receives the signal, readiness flips, no new work crosses the admission boundary, admitted work completes or is explicitly canceled by 25 seconds, unfinished jobs can retry without duplicate effects, both children are reaped, and the process exits before 30 seconds. I also test repeated SIGTERM for idempotency and SIGKILL for recovery behavior. The rollout dashboard should expose shutdown duration, in-flight work, deadline breaches, and forced exits.”

Common Mistakes

• Doing cleanup inside the raw handler → The signal may interrupt code while a library lock or allocator state is active → Publish a minimal signal-safe notification and clean up on the normal control path.
• Assuming the application receives SIGTERM → A shell-form entry point may keep the shell as PID 1 → Use exec form or a wrapper ending in exec, then test the built container.
• Treating readiness failure as admission closure → Endpoint updates take time and direct or existing connections may still send work → Fail readiness and enforce an application listener/admission cutoff.
• Closing shared clients first → In-flight handlers can fail after admission even though they had time to finish → Drain users before closing the resources they need.
• Stopping a job consumer without checking leases → Work may remain invisible, be acknowledged too early, or repeat side effects → Follow acknowledgement, lease, retry, and idempotency contracts explicitly.
• Waiting forever for a perfect drain → The orchestrator eventually sends SIGKILL and removes all cleanup opportunity → Use an internal deadline with time reserved for finalization.
• Forgetting child ownership → Children can outlive the parent briefly or become zombies when not reaped → Forward termination deliberately and use a bounded waitpid loop.
• Starting cleanup twice on repeated signals → Duplicate close and flush operations can race or crash → Make the state transition atomic and cleanup idempotent.
• Using standard signals as a command queue → Identical pending standard signals may coalesce and carry no durable ownership → Store work and retries in a queue; use the signal only for lifecycle control.
• Testing only an internal shutdown method → It bypasses PID layout, runtime signal delivery, and orchestration behavior → Send real signals to the production image under realistic concurrent work.

Follow-Up Questions and Responses

Follow-up 1: Why can’t the signal handler call the normal shutdown function?

The handler can interrupt the program while another thread or the interrupted thread holds allocator, stdio, logging, or application locks. Most application cleanup functions are not async-signal-safe. Calling them can deadlock or corrupt internal state. The handler should set a flag and use a signal-safe wake-up operation; the event loop or a dedicated signal thread invokes normal shutdown code afterward.

Follow-up 2: How do signals behave in a multithreaded process?

Signal disposition is shared by the process, but each thread has its own signal mask. A process-directed signal can be delivered to any eligible unblocked thread. One robust pattern blocks termination signals before workers are created and lets one thread receive them synchronously with sigwait or signalfd. Another uses a minimal process-wide handler that only posts a safe notification. Mixed and inconsistent masks make behavior harder to reason about.

Follow-up 3: What is the practical difference between SIGTERM and SIGKILL here?

SIGTERM requests termination and gives the application a chance to run its protocol because it can be caught. Its default action still terminates the process. SIGKILL is kernel-enforced termination: it cannot be caught, blocked, or ignored, and no cleanup runs. The grace period is valuable only if the SIGTERM path is reachable, safe, and bounded.

Follow-up 4: Why fail readiness and also close admission?

Readiness changes tell Kubernetes and its proxies to stop routing, but endpoint updates and connection draining are asynchronous. Existing keep-alive or direct connections may still reach the process. The application cutoff defines the exact point after which new work cannot enter, while readiness removes the Pod from normal routing. Verification should observe both layers.

Follow-up 5: What should happen to a job that is halfway complete?

Use the job’s ownership contract. Continue only if it can finish safely before the internal deadline. Otherwise stop or release its lease so it can be retried, and avoid acknowledging it before durable completion. External side effects require an idempotency key or transactional state transition because termination can occur between the side effect and acknowledgement.

Follow-up 6: What changes when a preStop hook exists?

The hook consumes the same Pod termination grace period. Measure its worst-case duration and subtract it from the application budget. Keep the hook bounded and avoid duplicating application cleanup in two competing paths. The application must still handle SIGTERM, because hooks can fail and processes can receive signals outside a normal rollout.

Follow-up 7: How would you investigate forced terminations in production?

Correlate Pod termination reason and timestamps with application shutdown-start time, in-flight count, job leases, child status, and the last completed shutdown state. Separate missing signal delivery from a slow drain, a stuck child, or a blocked final flush. Track graceful shutdown duration and forced-exit rate by version so a regression appears during a canary rollout.