System design interview: How do you isolate dependency concurrency to prevent cascading overload?

Prompt and context

A service calls a database, payment provider, and recommendation system. Recommendations slow from 10 ms to 1 second. In-flight requests grow until threads and connections are consumed, and APIs that do not need recommendations fail too. Design bulkheads so the slow dependency affects only related functionality.

AWS Builders’ Library describes this as latency-driven concurrency overload: a timeout is not a concurrency limit. Resources must be isolated around each dependency or client, protecting both the service and downstream systems.

What the interviewer evaluates

• You explain the Little’s Law intuition that higher service time raises in-flight work at the same arrival rate.
• You allocate concurrency by dependency, API, tenant, or resource pool so one slow dependency cannot consume global capacity.
• You compare hard, soft, and dynamic quotas with their fairness and utilization trade-offs.
• You design fast rejection, degradation, deadlines, circuit behavior, recovery, and observability.
• You prove unrelated APIs remain healthy without unbounded queues or starvation.

Questions to clarify first

• Which APIs call recommendations and which are critical? Can any path degrade?
• What are thread, connection, memory, and queue limits and current concurrency distribution?
• Does the dependency support cancellation, idempotency, batching, or caching? How is the caller deadline propagated?
• Is budget isolated by API, dependency, tenant, or Availability Zone? Is borrowing required?
• What user experience, error contract, and recovery objective apply during failure?

A 30-second answer

“I create a bounded concurrency bulkhead and queue per dependency, with separate pools for critical and degradable APIs. Requests carry a deadline; over-budget work fails fast or serves cache rather than waiting forever. I use soft quotas for utilization but hard global and per-class limits, with bounded borrowing. Metrics cover in-flight work, rejects, wait, timeout, degradation, and recovery by dependency and API. A slow-dependency test must leave unrelated APIs healthy.”

Step-by-step solution

Step 1: Quantify latency-driven concurrency

Measure arrival rate, service time, in-flight requests, and timeouts per dependency. If service time rises from 10 ms to 1 s, in-flight work can grow about 100x at the same arrival rate. Find the constrained resources before sizing bulkheads.

Step 2: Partition resource pools

Give each dependency its own connection pool, semaphore, and bounded queue; split critical and degradable APIs for the same dependency. Verify that isolation is real: threads, connections, and state must not remain shared behind separate counters.

Step 3: Choose hard, soft, and dynamic quotas

Hard quotas protect an API but waste capacity under skew. Soft quotas borrow idle capacity under a global cap. Dynamic quotas adapt to load but require minimum guarantees, maximum bounds, and a safe change rate. Add tenant partitions when fairness matters.

global_limit = 500
payments = hard 150
recommendations = soft 200, borrow <= 100
other_apis = reserved 50

Step 4: Reject and degrade deliberately

If no permit is available, return an explicit error or cache result instead of entering an unbounded queue. Respect the caller deadline and cancel work that can be cancelled. Retries need a budget, backoff, and idempotency; critical writes must not silently degrade.

Step 5: Recover without oscillation

After recovery, increase permits gradually and use half-open probes to avoid a burst. Retain time series for rejects, timeouts, and queue watermarks, segmented by dependency, API, tenant, and cell. Version configuration and keep rollback.

Step 6: Verify the isolation boundary

Inject latency, errors, and connection exhaustion into one dependency. Observe recommendation rejects, payment success, global thread/connection use, and tail latency. Test bursts, tenant skew, configuration changes, and cell failure; unrelated API SLOs and queue bounds are acceptance gates.

A strong sample answer

“I use arrival rate, service time, and in-flight work to show why a slow recommendation dependency multiplies concurrency. Each dependency has its own semaphore, pool, and bounded queue; critical and degradable APIs are separated. Payments get a hard reservation; recommendations use a soft quota with bounded borrowing. Every call propagates a deadline and fails fast or returns cache when permits are exhausted.”

“Recovery uses half-open probes and gradual permit increases. Metrics include in-flight work, rejects, wait, timeouts, degradation hits, downstream errors, and recovery time. A drill slows only recommendations and checks payment and unrelated API tails, pools, and threads stay within bounds.”

Common mistakes

• Only increase timeouts → in-flight work grows → set dependency-level concurrency limits.
• Share one thread and connection pool → one slow dependency takes down the service → partition by dependency and criticality.
• Use an unbounded queue → memory and latency run away → bound it and reject fast.
• Static hard quotas everywhere → skew leaves capacity idle → allow bounded soft borrowing.
• Retry without a budget → downstream overload is amplified → propagate deadlines, limit attempts, and require idempotency.
• Test only the failed dependency → isolation is unproven → verify unrelated API SLOs simultaneously.

Follow-up questions and answers

Why is a timeout not a concurrency limit?

It bounds one wait, but the waiting request still occupies a thread, connection, and memory. Higher dependency latency puts more requests in flight, so permits must be bounded.

How does soft borrowing prevent one API from taking everything?

Use a global cap, per-API minimum reservation, maximum borrow, and reclaim rate. Once the borrower reaches its bound, it fails fast instead of growing forever.

When is returning cache safe?

For reads where bounded staleness is acceptable, return a timestamped cache value. Payment, authorization, and write results cannot be silently replaced by stale data.

How do you prove isolation works?

Inject slowness and errors into one dependency, then inspect unrelated API in-flight work, tails, thread/connection use, and success. Synchronous degradation means a shared resource or queue boundary remains.