Prompt and context
An idempotent read API has a high p99 while p50 is healthy; slow calls appear to come from occasional queueing or host hiccups. Design request hedging: send a primary, issue a copy to another replica after a delay, return the first acceptable result, and cancel the other. Explain the cost and failure-amplification boundaries.
What the interviewer is testing
Understand the goal
Hedging targets the tail; it does not make every request faster. It fits safely repeatable reads and should not blindly duplicate side-effecting writes.
Control the cost
Duplicating every request can nearly double backend load. A strong design uses delayed triggers, a maximum attempt count, throttling, cancellation, and metrics based on the original request.
Handle correlated failures
Sending both attempts to one overloaded host has little value. Cross-instance, zone, or failure-domain routing plus queue and error protection determines whether hedging is safe.
Questions to clarify first
- Is the API idempotent, and can multiple concurrent reads be accepted?
- What are the p50, p95, p99, p99.9 baselines and SLO?
- Are slow calls isolated stragglers or a queueing problem shared by every replica?
- How are replicas selected across instances, zones, and versions?
- Does cancellation actually release downstream threads, connections, and compute?
- Which error rate, queue depth, or hedge fire rate disables the policy?
A 30-second answer
“I would enable hedging only for idempotent reads. Send the primary first; use a dynamic p95-style threshold per request class, then send one hedge to a healthy replica in another failure domain only when the threshold expires and a budget allows it. Share one deadline, return the first success, and cancel the loser. Bound attempts, use throttling and queue/error guards, and monitor p99, fire rate, extra requests, cancellation success, and original latency. Disable the policy when it amplifies load.”
Step-by-step deep answer
Define the request state machine
States are primary_sent, hedge_waiting, hedge_sent, winner_selected, and deadline_exceeded. Send the primary first; finish if it returns within the threshold, otherwise create one hedge. The first successful response wins and all other attempts receive cancellation.
Choose the threshold and scope
Maintain latency histograms bucketed by method, tenant, request size, or prompt length. Use a dynamic p95-style threshold with bounds and a cold-start fallback. Do not use only a global mean or immediately duplicate every request.
Route to an independent replica
Avoid the primary's host, zone, or failure domain. Prefer a healthy node with a short queue. If every candidate is overloaded, hedging adds congestion; wait, degrade, or fail fast instead.
Handle cancellation and the deadline
Clients and proxies must propagate a cancellation token. Downstream work must stop and release connections, threads, GPUs, or caches. Both attempts share the total deadline so duplication cannot extend user-visible waiting.
Protect downstream capacity
Set maxAttempts, a minimum hedge delay, an in-flight budget, and a per-service token bucket. gRPC caps maxAttempts at 5 and offers retry throttling; a production design still needs its own capacity and error budget.
Handle errors and non-idempotent calls
Continue only for retryable, idempotent cases. Return deterministic validation or authentication errors immediately. Writes need idempotency keys and deduplication, or should use one request plus compensation.
Pseudocode
~~~text send(primary) timer = hedgeThreshold(request_class) if primary unfinished at timer and budget_allows(): send(hedge, differentfailuredomain) winner = firstsuccessbefore_deadline() cancel(allotherattempts) record(primarylatency, hedgefired, winner, cancel_result) ~~~
Complexity, observability, and rollback
The worst attempt count is bounded by maxAttempts; extra request volume depends on fire rate and cancellation latency. Track p50/p95/p99, hedge fire rate, extra QPS, downstream queues, errors, cancellation success, and original primary latency. Turn the policy off by service, tenant, or region when congestion or errors rise.
| Control | Purpose | Failure mode if misconfigured |
|---|---|---|
| hedge delay | Duplicate only slow calls | Too short amplifies QPS |
| maxAttempts | Bound concurrent copies | Too high creates a request storm |
| cancellation | Free loser resources | Failure keeps consuming capacity |
| throttle budget | Tighten during congestion | Missing metrics hides overload |
Model answer
“I would first confirm that this is an idempotent read and build a per-request-class latency histogram. After sending the primary, I would send one hedge only after a p95-style threshold, a healthy independent replica, and a concurrency budget all permit it. Both attempts share a deadline; the first success wins and the proxy propagates cancellation while recording whether resources were actually freed. Max attempts, token throttling, queue depth, and error rate protect the backend, while deterministic errors are never duplicated. I would canary the policy and compare p99, fire rate, extra QPS, cancellation latency, and unhedged primary latency; a hedge storm disables it automatically.”
Common mistakes
Duplicating every request immediately
That turns hedging into unconditional replication, increasing normal load and cost without showing that stragglers caused the tail.
Hedging side-effecting writes
Two attempts can create two records or charge twice. Without idempotency keys, deduplication, and transaction semantics, do not hedge.
Using the same failure domain
A shared host, rack, or zone failure slows both attempts and adds load to the overloaded location.
Looking only at user-visible p99
Hedging can hide a worsening primary path and delay scaling signals. Record unhedged primary latency and queue depth too.
Ignoring cancellation
Stopping the wait is not stopping downstream work. Verify propagation, resource release, and cancellation latency.
Having no kill switch
High errors, queue buildup, or an abnormal hedge fire rate require a fast service, tenant, or region-level disable switch.
Follow-up questions and responses
How does hedging differ from retry?
Retry usually waits for failure before sending again. Hedging sends a copy after a latency threshold while the first attempt is still running. Both need idempotency, deadlines, and throttling.
Why start with p95?
It leaves most normal requests untouched while covering a small tail slice. The threshold should be tuned per request class, budget, and experiment results.
What if every replica is congested?
Stop hedging and use rate limits, queues, degradation, or fast failure. Hedging cannot repair insufficient capacity.
How do you prove cancellation works?
Record downstream cancellation, work completion, connection occupancy, and release latency. Inject a slow loser and confirm it stops after a winner is selected.
How does streaming change the design?
Use time to first byte or token as a trigger, but duplicating after output begins can create duplicate data. Define stream merging, cancellation, and client visibility first.
What are key gRPC knobs?
maxAttempts, hedgingDelay, and non-fatal status codes control when copies are sent. gRPC caps attempts at 5 and provides retry throttling and server pushback; the service still needs its own capacity budget.