Prompt and context
The platform has hundreds of gateway instances across multiple regions. Product teams need requests-per-second, concurrency, and daily quotas by tenant, application, endpoint, and plan, with changes effective within minutes. One tenant must not consume shared capacity, yet core APIs should remain available during a brief quota-service outage.
The interview tests control-plane/data-plane separation, counting models, distribution consistency, cross-region trade-offs, and failure behavior. Envoy distinguishes local from global rate limiting, while RFC 9331 defines HTTP RateLimit fields; connect the protocol, product policy, and runtime decision.
What the interviewer evaluates
Cover policy shape, versions and approval, descriptor keys, token bucket or sliding windows, hot tenants, cross-region counting, configuration distribution, caching, fail-open versus fail-closed, quota headers, audit, and cost.
Clarifying questions to ask
- Are quotas hard limits, soft alerts, or both, and may traffic burst or borrow future capacity?
- Which dimensions require global precision and which allow bounded regional approximation?
- What are propagation and rollback targets, and how long can a gateway be disconnected before a policy is stale?
- Must an over-limit response expose retry timing, remaining quota, and billing events?
- Which core endpoints must continue during a limiter outage?
A 30-second answer
“The control plane stores approved, versioned policies, compiles them into gateway descriptors, and distributes deltas. The data plane makes local fast decisions; only dimensions requiring global precision call a shared limiter. Token buckets absorb bursts and counters are isolated by tenant and endpoint. Policies carry TTL, checksum, and rollback version. Choose fail-open or fail-closed by endpoint risk, return standard RateLimit information, and audit the decision reason.”
Step-by-step deep dive
Step 1: Define policy and release workflow
A policy contains tenant, application, endpoint, window, rate, capacity, concurrency, daily quota, region scope, and priority. Gateways should not interpret arbitrary product fields; the control plane compiles them into stable descriptors.
policy v42:
subject: tenant:acme / app:billing
route: POST /invoices
rate: 200 requests/second
burst: 400
scope: globalRelease requires approval, static conflict checks, traffic simulation, and a signed version. Keep author, reason, expected impact, and rollback pointer; never overwrite an audited version.
Step 2: Choose counting and sharding
Use a local token bucket at the gateway for short bursts. A shared limiter can count global dimensions by descriptor. For daily quotas, use atomic window counters or sharded quota, with explicit window boundaries and a clock source.
Shard by tenant, application, and endpoint so one hot key does not become the bottleneck. A hot tenant may receive hierarchical tokens, preallocated capacity, or a dedicated shard, with explicit treatment of short overage and final billing.
Step 3: Distribute policy with consistency
Stream compiled policies with version and checksum to gateways. Accept only monotonically increasing, validly signed versions; load the last valid snapshot on startup. Missing updates mark the gateway stale after its TTL and emit an alert.
Prefer local distribution within a region. Cross-region policy uses a global version and explicit effective time. A rollback is another version, never history rewriting; gateways report version coverage after acknowledgement.
Step 4: Handle consistency, bursts, and fairness
Global exact counting adds network latency and shared-state cost. Reserve strong coordination for high-risk or contractual dimensions and allow bounded error elsewhere. Align burst capacity with backend concurrency; increasing only the gateway bucket can still overload the service.
Tenant fairness must protect shared connection pools. Couple quota decisions with concurrency bulkheads, queue length, and priority. Record whether traffic was rejected or queued and why, so customers can diagnose more than a single number.
Step 5: Define failure and degradation
If the limiter is unreachable, low-risk read-only endpoints can use the latest local snapshot with bounded fail-open; writes, billing, and expensive endpoints use fail-closed or a stricter local cap. Give every degraded decision an expiry and reconcile or mark approximate counts after recovery.
Exercise gateway restart, clock drift, duplicate messages, and distribution interruption. Reject a corrupt snapshot and retain the last valid version; an empty configuration must never mean unlimited quota.
Step 6: Protocol, audit, and observability
Return stable 429 semantics on over-limit responses and expose interpretable limit, remaining, and reset information according to RFC 9331. For tenants that should not see exact numbers, use a tiered message. Internally record policy version, descriptor, region, counter source, degraded state, and request trace.
Monitor version coverage, decision latency, rejection rate, hot keys, count error, degradation duration, and rollback. Evaluate new policies in shadow mode and compare rejection changes before release.
A strong sample answer
I would compile approved policies into versioned descriptors and distribute deltas from the control plane. Gateways use local token buckets for low latency, calling a shared service only for dimensions that need global precision. Policies carry signatures, TTL, checksums, and rollback versions, isolated by tenant, application, endpoint, and region. Degrade by endpoint risk, return stable 429 and RateLimit information, and record version, counter source, and approximation state.
Common mistakes
- Sending every request to a central counter → latency and failure domain grow → make local decisions and coordinate globally by risk.
- Overwriting configuration → no audit or rollback → use approval, signatures, and monotonic versions.
- Setting rate without burst or concurrency → the backend can still be flooded → couple tokens, bulkheads, and queues.
- Always failing open → writes and billing become unbounded → degrade by endpoint risk.
- Returning a vague error → customers cannot adjust traffic → provide stable status, reset timing, and an auditable reason.
Follow-up questions and responses
Follow-up 1: Why not require strong consistency everywhere?
Strong consistency requires shared state and network round trips, reducing availability and increasing cost. Reserve it for contractual or security-critical dimensions and publish the bounded-error model elsewhere.
Follow-up 2: How do you handle a cross-region burst?
Preallocate regional tokens under a global cap. High-value traffic may borrow briefly, but record borrowed capacity and repayment or billing rules so one region cannot dominate.
Follow-up 3: Who owns propagation delay?
The gateway keeps its last valid version and marks itself stale while the control plane tracks coverage. After TTL, tighten or pause by endpoint risk; never silently become unlimited.
Follow-up 4: How do you prove a new policy does not harm customers?
Run shadow evaluation and historical replay, compare rejection, latency, and tenant distributions, set automated gates, then canary narrowly with a one-click rollback version.