Question and scope
You are designing a request path in which service A calls B and B calls C. The client allows 2 seconds from acceptance to a response. A may spend 500 ms before calling B; B may call C and a non-critical audit service. Explain the contract for deadlines, cancellation, retries, and observability.
Assume each hop can fail independently, clocks are not perfectly synchronized, and a timed-out read-only RPC may be retried only when the operation is idempotent. The goal is to stop doomed work while preserving enough budget for response serialization and transport.
What the interviewer is testing
InterviewStack's staff-level system-design rubric expects candidates to clarify latency and failure constraints, identify bottlenecks, and justify trade-offs. A strong answer turns those signals into an explicit budget contract:
- One absolute request deadline is established at the edge; every hop derives its remaining budget instead of inventing a fresh timeout.
- Cancellation follows the call tree, including hedged or parallel branches, so a completed response stops work that can no longer help.
- Retries consume the same remaining budget and are restricted by idempotency and an attempt cap.
- Metrics distinguish deadline exhaustion, caller cancellation, queue delay, downstream rejection, and useful work completed after cancellation.
Clarifying questions before answering
- Is the 2-second limit a user-visible SLO or a hard correctness deadline? A hard deadline makes late results invalid; an SLO permits a carefully bounded background completion path.
- Which calls are critical to the response? Non-critical audit or recommendation calls can be detached, sampled, or returned with a degraded response; billing authorization cannot.
- Are writes idempotent? A retry budget is safe for a read or idempotency-keyed command, but not for an unprotected side effect.
- Does the RPC stack propagate context automatically? If not, define an interceptor or middleware contract and test it at every language boundary.
- Do we need work to finish after cancellation, such as a checkpointed export? That exception needs a durable job contract rather than silently ignoring cancellation.
30-second answer framework
“I set one deadline at the ingress and carry the remaining budget through the RPC context. Each service checks cancellation before queueing and during long work, then passes a shorter child budget that leaves response and network slack. Retries use only the remaining time, are capped, and require idempotency. When the caller gets a result or cancels, the cancellation signal fans out to every branch. I measure deadline-exceeded rate, remaining budget at each hop, queue time, cancellation latency, and late work, then load-test the chain and inject slow or cancelled dependencies.”
Step-by-step deep answer
1. Establish and represent the budget
The edge records an absolute deadline, for example t0 + 2s. Treating it as a point in time avoids each hop silently adding another 2 seconds. A service computes remaining = deadline - now and reserves a small local margin for serialization and network transit. The margin is a policy, not a universal constant; validate it with percentile latency data.
The gRPC guidance distinguishes a deadline from a timeout and recommends explicit client deadlines. It also converts propagated deadlines into remaining time so clock skew does not make a downstream server wait past the original budget. The wire contract should therefore carry deadline context through the RPC library, not a free-form application header that callers can forget.
2. Spend budget by critical path
Suppose A spent 500 ms. It forwards at most 1.5 seconds to B. If B needs 400 ms of local work and calls C, B forwards the lesser of its remaining budget minus a 100 ms response margin and C's service-specific upper bound. Parallel calls share the same parent deadline; they do not each receive a full 1.5 seconds.
Queue admission checks the remaining budget. If the estimated queue delay already exceeds it, reject fast or return a documented degraded response. This prevents a request from occupying a worker after success is impossible.
3. Propagate cancellation, not just expiry
Deadline expiry is one cancellation reason; the caller closing the connection or another branch winning a hedge is another. Every child RPC receives the parent cancellation context. Workers check it before expensive steps and at bounded intervals during loops, then release permits and close streams.
Google SRE warns that deadline propagation alone can still leak work when a deeper call fails early. Cancellation must travel back up and fan out to siblings. A checkpointed, durable job may finish a safe checkpoint, but that is an explicit asynchronous workflow with its own status API, not a synchronous RPC that ignores cancellation.
4. Make retries budget-aware
Compute attempt_deadline = min(parent_remaining - response_margin, per-attempt_cap). Stop when the next attempt cannot finish before the parent deadline. Retry only transient failures and only operations that are naturally idempotent or protected by an idempotency key. Hedged requests use the same parent budget and cancel losing attempts as soon as one acceptable response arrives.
Do not combine a long retry timeout with a circuit breaker that has no knowledge of remaining budget. Record the attempt number and original deadline so a downstream service can distinguish a fresh request from a nearly expired retry.
5. Define observable failure semantics
Return a stable status for deadline exhaustion, such as gRPC DEADLINE_EXCEEDED, and preserve the original cancellation cause in logs. Trace fields should include original deadline, remaining budget on entry and exit, queue delay, attempt count, and cancellation-to-stop latency. Alert on budget spent in queues and on useful-work-after-cancel, not only end-to-end latency.
Test the contract with a three-hop integration harness: delay C beyond the remaining budget, cancel at A while B is queued, make the first hedge succeed, and inject clock offsets. Assert that no child runs beyond the parent deadline except an explicitly declared checkpoint, that retries stop, and that permits are released.
High-quality sample answer
“I would make the ingress the sole owner of the 2-second deadline. A receives it in the RPC context, and after 500 ms it forwards the remaining budget to B. B subtracts its own response margin before calling C; parallel branches still share the same parent deadline. Every child observes cancellation before queueing and while doing bounded work, and the cancellation fans out when the client disconnects or a hedge wins. Retries are limited to the remaining budget and require idempotency. A late audit call can be detached into a durable job, but the critical response cannot pretend that late work is useful. I would prove the contract with injected slow dependencies, queue delays, cancellation races, clock skew, and traces showing budget at each hop.”
Common mistakes
- Give every hop a fresh timeout → the call tree can run for N times the user budget → propagate one deadline and compute remaining time.
- Propagate expiry but ignore caller cancellation → hedged and abandoned requests keep consuming workers → fan out the cancellation context and measure stop latency.
- Retry every timeout → non-idempotent writes may duplicate side effects and expired retries amplify load → require idempotency, classify errors, and cap attempts by remaining budget.
- Queue before checking budget → a doomed request occupies scarce capacity → reject or degrade when queue delay cannot fit the remaining time.
- Use a fixed margin without measurement → large payloads fail consistently or latency slack is wasted → derive margins from transport and serialization percentiles.
- Hide late work as success → dashboards show healthy responses while resources leak → separate synchronous deadlines from explicit checkpointed jobs.
Follow-ups and responses
What if clocks differ between services?
Do not compare raw wall-clock timestamps from independent machines. Use the RPC library's remaining-time conversion or propagate a timeout after subtracting elapsed time. Add clock-offset tests to the harness.
Should a deadline ever be extended downstream?
Only by changing the product contract to an asynchronous job. Extending a synchronous child deadline cannot make the parent's late response valid; it only increases wasted work.
How do you reserve budget for a slow payload?
Measure serialization and network tails separately, reserve a bounded margin, and cap payload size or use streaming. If the margin repeatedly consumes the budget, change the SLO or response shape instead of silently increasing every timeout.
What survives cancellation?
Only explicitly durable, idempotent work such as a checkpoint write or audit event. It must be observable under a job identifier and must not hold the synchronous request's worker or connection.