Problem and scope
The gateway fronts several APIs. Traffic can rise 20x in a minute, while databases, search, and third-party dependencies have different capacity. Design request classification, admission, queues, load shedding, degraded responses, and recovery. Business authorization, a complete WAF, and adding capacity to a broken dependency are out of scope.
What the interviewer is evaluating
Distinguish rate limiting from overload protection. Rate limiting usually caps identity traffic over a window; admission control decides whether work may consume real resources using concurrency, queue age, cost, and dependency health. Strong answers explain what is dropped, why, how low-priority work avoids starvation, and how recovery avoids a traffic surge.
Clarifying questions
- Which requests are critical, and which may be delayed, cached, or approximated?
- Are you protecting the gateway, one dependency, one tenant, or every boundary?
- How long may work wait, and should clients retry, poll, or accept an empty result?
- Must capacity be fair across tenants, regions, APIs, or cost classes?
- Which status codes, headers, and decision metrics are visible to callers?
A 30-second answer framework
“I would classify requests by trusted route metadata, tenant, and cost, then perform hard concurrency and budget checks before placing accepted work in bounded queues partitioned by priority and tenant. A controller adjusts target concurrency from in-flight work, queue age, error rates, and dependency signals. During overload it rejects retryable or low-value work first and reserves capacity for critical requests. Cacheable or approximate operations use a degraded path, while rejections include retry timing. Recovery uses gradual ramp-up, leases, and controlled probes so retries cannot create a second overload.”
Step-by-step deep design
The gateway validates request size, timeout budget, and tenant quota, then maps the request to criticality, cost, retryability, and dependency labels. Trusted route configuration supplies labels; clients cannot self-declare priority. Health and management traffic use a reserved pool so data traffic cannot exhaust the control plane.
Maintain in-flight caps, queue caps, and timeout budgets per API and dependency. Semaphores or leases protect real resources; queues are bounded rather than hiding overload in infinite backlog. Consume budget only after admission, and release leases on cancellation or timeout. Streaming requests need separate connection and byte budgets so one long stream cannot occupy every slot.
The scheduler selects work by priority, tenant weight, and age. Critical work receives a minimum concurrency reservation; low-priority work may be dropped or delayed, with aging to prevent starvation. A tenant burst cannot borrow another tenant’s reservation. Across regions, local fast decisions can accept bounded counter error instead of adding a fragile global coordination dependency.
Each short window, a controller adjusts target concurrency: reduce it when p95 latency, queue age, or downstream errors cross thresholds; increase it slowly when stable. Smooth signals and hysteresis prevent threshold oscillation. Client retries are not a health signal. Track retry amplification and bound return traffic with Retry-After, jitter, and retry budgets.
Choose shedding by request semantics. Recommendations, analytics, and previews may return cached or approximate results; writes, payments, and permission changes usually fail fast and require safe retry. A degraded response carries version, timestamp, and freshness instead of pretending to be complete. The gateway should not drop an unrepeatable side effect it does not understand.
When a dependency times out or errors, an isolation pool limits its connections, concurrency, and retry budget; a circuit breaker permits only controlled probes. Cached and static responses use the isolated pool. On success, ramp traffic gradually. Record admission decision, policy version, rejection reason, queue wait, and dependency signals so a pile of 429 responses remains diagnosable.
Monitor admission rate, rejection rate by priority and tenant, oldest queue age, in-flight work, p95/p99 latency, downstream errors, retry amplification, degraded freshness, and recovery slope. Reconcile resource budgets with actual connections, threads, database connections, and queue jobs. Inject traffic spikes, slow dependencies, bad policy configuration, controller loss, regional partitions, and replay storms.
High-quality sample answer
“At the gateway I would attach trusted criticality, cost, retryability, and dependency labels. Every API and dependency has bounded concurrency, bounded queues, and reserved capacity; the scheduler uses priority, tenant weight, and aging. A controller adjusts target concurrency from p95 latency, queue age, and downstream errors with hysteresis. Under overload it rejects low-value or retryable work first and preserves capacity for critical writes; previews can return cache data with freshness.
Every rejection includes a stable reason, Retry-After, and request ID. Isolation pools, retry budgets, and controlled probes prevent cascades; recovery ramps gradually. Metrics cover fairness, retry amplification, freshness, and recovery slope, while fault injection tests controller loss, regional partitions, and replay storms. The contract is predictable critical-path latency under overload, not infinite waiting for every request.”
Common mistakes
- Add a fixed QPS limiter only → the bottleneck may be connections, CPU, or a dependency → combine concurrency, queue, and health signals.
- Use an unbounded queue → latency explodes and capacity gaps stay hidden → bound the queue and reject explicitly.
- Give every request equal priority → low-value work crowds out critical paths → reserve semantic capacity and age work.
- Let clients self-report priority → attackers bypass protection → classify from trusted route policy.
- Retry without a budget during overload → amplification collapses the dependency → use budgets, jitter, and
Retry-After. - Return stale data without freshness → users mistake it for current truth → include version, timestamp, and source.
- Release all traffic immediately → replay storms overload recovery → ramp gradually with controlled probes.
- Require exact global counters → the protection path gains more failure dependencies → make fast local decisions with bounded error.
Follow-up questions and answers
Follow-up 1: How is admission control different from rate limiting?
Rate limiting constrains identity traffic over a window. Admission control decides whether work can consume real resources using concurrency, queue wait, cost, and dependency health. They can coexist, but a fixed QPS cap is not resource admission.
Follow-up 2: How do you prevent low-priority tenants from crowding out critical tenants?
Reserve a pool or minimum quota for critical tenants, then schedule shared capacity by weight. Give every tenant a maximum budget so bursts cannot borrow reservations indefinitely.
Follow-up 3: Why not queue everything?
After the business deadline, waiting creates timeouts and retries rather than useful work. A bounded queue makes excess demand explicit and gives callers feedback.
Follow-up 4: Which signals drive the controller?
At minimum p95/p99 latency, in-flight work, oldest queue age, dependency error rate, and resource utilization. Smooth them, add hysteresis, and separate client-retry traffic from healthy demand.
Follow-up 5: Can a critical write degrade?
Only if business semantics permit durable enqueue followed by asynchronous completion. Payments, permissions, and inventory cannot return false success; they fail safely and retry with idempotency.
Follow-up 6: How do you verify recovery will not overload again?
Inject dependency recovery, backlog, and client retries, then observe ramp rate, reservations, queue age, and errors. Set a maximum increase, cooldown, and manual pause switch.