Question and When It Applies
A control-plane API serves health checks, controller writes, tenant queries, and long-running watches. A traffic spike or one noisy tenant can slow every request. Design priority and fairness controls, and explain how classification, concurrency limits, queues, timeouts, and degradation work together.
What the Interviewer Evaluates
- Turning “important” into auditable request classes instead of permanent team privilege.
- Protecting the control plane with seats, queues, and quotas rather than only an ingress rate limiter.
- Explaining fairness granularity, long-request occupancy, tenant isolation, and failure degradation.
- Validating the design with latency, rejection rate, queue age, and controller recovery time.
Clarifying Questions Before You Answer
- Which requests are safety-critical writes, and which reads or watches can be delayed?
- Is fairness computed by tenant, identity, workload, or resource type? Is an emergency admin path allowed?
- Are you protecting the API server, downstream storage, or both?
- Do long requests consume multiple seats, and how are seats released on disconnect?
- Can overload return 429, or must the system preserve a minimum success rate and retry signal?
30-Second Answer Framework
I would map requests by identity, method, and resource into bounded priority flows, then give each flow concurrency seats and a fair queue. High priority gets bounded baseline capacity; low priority queues or rejects under pressure. Long requests are classified separately and charged for continuous occupancy. Every rejection carries an actionable retry signal. I would gate rollout on p99 latency, queue age, 429 rate, downstream saturation, and recovery time for critical controllers.
Step-by-Step Deep Dive
Step 1: Build explainable request classes
Classify from verifiable identity, HTTP method, target resource, and long-running status. Do not let clients submit an arbitrary “high priority” field. Split controller writes, node heartbeats, admin queries, and bulk lists into distinct flows, and log the matching rule.
Step 2: Model capacity as seats
Seats represent simultaneous service capacity rather than only request count. A fast read may consume one seat; a slow query or watch may need more. Release seats on completion or cancellation. Give each priority a nominal limit while keeping a global seat cap so baselines cannot exceed safe capacity.
Step 3: Apply fair queuing within each priority
Within a priority, use a weighted fair queue keyed by tenant or flow identity so one tenant cannot fill the class. Dispatch a non-empty flow that is below its current limit. Aging may raise the weight of a repeatedly delayed request, but it must not bypass the global seat cap.
Step 4: Handle long requests and dependencies
Give watches and log tails separate budgets, maximum durations, and heartbeat cancellation. Put independent concurrency bulkheads around storage, caches, and external services; an ingress queue must not transfer unlimited pressure downstream. Retry only retryable failures, with exponential backoff and jitter.
Step 5: Define overload actions
As pressure rises, pause new low-priority work, then restrict bulk lists and expensive filters, and finally return 429 when a request cannot queue. Include a clear wait hint in the response; clients still need retry caps, jitter, and deadlines. If a critical write cannot be dropped, place it in a durable queue and return a queryable operation ID.
Step 6: Observe fairness, not just average latency
Monitor p50, p99, queue age, seat occupancy, 429s, cancellations, and downstream errors by priority, tenant, and request type. Alert on maximum wait, critical-write success, recovery time, and service disparity between tenants. Load tests should include one-tenant bursts, seat exhaustion by long requests, misclassification, and controller recovery.
Step 7: Evolve and roll back safely
Record new classification matches in observe-only mode before enforcing limits. Version and audit configuration, and keep a constrained operational path. When seats or priorities change, compare downstream capacity and historical queue distributions; do not rely only on API-layer metrics.
High-Quality Sample Answer
I would model control-plane capacity as a global seat pool. Requests map to priority flows using identity, method, resource, and long-running attributes. Controller writes and node heartbeats get baseline seats, but the sum of baselines stays below safe capacity. Ordinary tenant reads share a weighted fair queue. Watches are charged separately and have maximum durations; cancellation immediately releases seats. During overload, pause bulk reads, restrict expensive filters, and return 429 with a wait hint when work cannot queue. Writes that must complete go to a durable queue. Before enforcement, I would measure classification matches and tenant queue age; after rollout, critical-write success, p99, 429s, downstream saturation, and recovery time become rollback gates.
Common Mistakes
- Setting only a global QPS limit, allowing long requests to exhaust concurrency.
- Giving an admin or tenant unlimited priority and creating unauditable starvation.
- Charging only by request count without modeling reads, lists, and watches differently.
- Making every client retry 429 immediately and creating a synchronized retry storm.
- Looking only at average latency and missing the maximum wait of low-priority queues.
- Changing limits without a canary, version, or rollback path.
Follow-Up Questions and Responses
Follow-up 1: Why not use only a token bucket?
A token bucket limits arrival rate but does not express different concurrency costs, long-request occupancy, or tenant fairness. It can be one ingress layer, but seats and queues are still required.
Follow-up 2: Can high priority starve low priority?
Yes, unless high priority also has limits and the system enforces reserved shares, maximum consecutive service, or aging. An emergency path must remain audited and capacity-bounded.
Follow-up 3: How many seats should a watch consume?
There is no universal constant. Measure connections, event rate, serialization cost, and downstream query pressure, choose a conservative baseline, and calibrate with load tests. Timeouts and disconnects must release seats.
Follow-up 4: Who decides the 429 retry time?
The server supplies a minimum wait hint based on expected recovery; the client adds exponential backoff, jitter, and a deadline. The hint is not a capacity guarantee, so clients still cap attempts.
Follow-up 5: How do you prove fairness?
Define tenant-level objectives such as seat share, maximum queue age, and completion rate within a priority. Compare distributions during a single-tenant burst and mixed load instead of reporting only a global mean.
Follow-up 6: What if a rule misclassifies a critical write as low priority?
Keep rule-match logs and human review, and ship configuration as a versioned artifact. If critical-write success falls, roll back the classification version immediately and use a constrained safety path for queued work.