Prompt and scope
Design a worker pool for a backend service that accepts jobs, runs at most a fixed number concurrently, and shuts down without silently dropping accepted work. Explain queue capacity, admission or rejection, cancellation, retry ownership, result correlation, metrics, and graceful shutdown.
Assume jobs are independent and may call a downstream API. The pool is an in-process component; durable queues belong in a separate design when jobs must survive process loss.
What the interviewer is testing
They want to see a clear concurrency limit, a bounded memory model, and a policy for overload. They also test whether cancellation reaches workers, whether retries can multiply load, and whether shutdown distinguishes queued, running, completed, and rejected jobs.
Questions to clarify before answering
- Is losing work on process crash acceptable? If no, place the job in a durable broker first.
- Are jobs idempotent, and can they be retried safely?
- What are downstream rate limits, average runtime, and tail latency targets?
- Should submit block briefly, return
429, or shed low-priority work when the queue is full? - Does cancellation mean “stop before start” only, or can the job cooperatively interrupt I/O?
A 30-second answer framework
“I would expose a bounded submit API backed by a fixed worker count and a finite queue. Admission returns a clear overload result when capacity is exhausted, while a context or cancellation token lets queued jobs leave before execution and lets running jobs stop cooperatively. Each accepted job has an ID and terminal state; retries are capped, jittered, and owned by one layer. Metrics cover queue depth, age, active workers, rejection, runtime, and cancellation. Shutdown stops admission, cancels queued work according to policy, drains accepted work until a deadline, and reports anything that could not finish.”
Step-by-step deep dive
Step 1: Define the state machine
Use submitted → queued → running → succeeded|failed|cancelled. A full queue yields rejected rather than an invisible wait. Persisting these states outside the process is required when callers need recovery after a crash.
Step 2: Bound both workers and queue
Choose W workers and queue capacity Q; memory is bounded by Q job payloads plus worker stacks. Do not create one thread or goroutine per request. Python’s ThreadPoolExecutor exposes max_workers, but an application-level bounded admission policy is still needed when submission volume is unbounded.
Step 3: Choose admission and backpressure
Offer non-blocking submit, bounded waiting, or priority-aware rejection. Return a stable overload code and Retry-After only when retrying is safe. A producer that waits forever on a full queue can deadlock a request path, while an unbounded queue converts overload into latency and memory growth.
submit(job, deadline):
if stopping or deadline expired: return REJECTED
if queue.try_push(job): return ACCEPTED(job.id)
if policy == WAIT and wait_until(deadline) and queue.try_push(job):
return ACCEPTED(job.id)
return OVERLOADEDStep 4: Make cancellation cooperative
Pass a cancellation context to each job. Remove cancelled jobs that have not started; running jobs must check cancellation at safe points and pass it to downstream clients. A pool cannot safely kill arbitrary threads, so document what “cancelled” means for non-interruptible I/O.
Step 5: Keep retry ownership single-layered
Choose either the pool, the job handler, or the durable queue to schedule retries. Cap attempts, add exponential backoff with jitter, and classify errors into retryable and permanent. Otherwise a timeout at three layers can create a retry storm against the same dependency.
Step 6: Correlate results and errors
Return a job ID or future, never a shared mutable result slot. Record the original error, attempt count, and terminal state. If callers poll, make the result store’s retention and authorization explicit; if callers await, define how disconnects affect work.
Step 7: Design graceful shutdown
On shutdown, stop admission first, then mark queued jobs according to policy, and let running jobs drain until a deadline. The Go pipeline guidance propagates cancellation through a done signal; the same principle applies here. After the deadline, report unfinished jobs for replay or mark them abandoned only with an explicit contract.
Step 8: Instrument the bottleneck
Track queue depth and age, active workers, utilization, accepted/rejected/cancelled counts, execution duration, retry count, and downstream errors. Alert on sustained queue age and rejection, not only CPU. Size W from the downstream bottleneck: connection pool, external rate limit, or CPU capacity.
Trade-offs and boundaries
Trade-off 1: Fixed workers or dynamic scaling
Fixed workers make concurrency predictable and protect dependencies. Dynamic scaling can improve throughput but must enforce a hard global limit and account for every replica; otherwise each instance scales independently and overwhelms the dependency.
Trade-off 2: Reject or wait when full
Rejecting gives callers fast feedback and preserves latency. Waiting can smooth short bursts, but the wait must have a deadline and must not hold scarce request threads indefinitely.
Trade-off 3: In-process or durable queue
An in-process pool is low latency and simple. A durable broker adds recovery, replay, and operational cost. Use the durable option when accepted work must survive deploys, crashes, or multi-instance routing.
Failure drills and evolution plan
Drill 1: Downstream outage
Make every job fail slowly and verify queue age, rejection, timeout, and capped retries. Confirm the pool does not create more concurrency while the dependency is unhealthy.
Drill 2: Cancellation storm
Submit 10,000 jobs, cancel half before execution, and stop the service during the drain. Verify queued cancellations do not run and accepted running jobs reach a reported terminal state.
Drill 3: Replica rollout
Run two versions during a rolling deploy. Check that each instance enforces its local limit and that a durable queue or global limiter enforces the system-wide limit when required.
Common mistakes and follow-ups
Mistake 1: Unbounded buffering
An unbounded queue hides overload until memory or latency collapses. Make capacity and the rejection policy observable.
Mistake 2: Duplicate retries
If both the HTTP client and job handler retry, attempts multiply. Assign one retry owner and propagate attempt metadata.
Mistake 3: Treating cancellation as thread killing
Most runtimes cannot safely kill arbitrary work. Use cooperative checks, cancellable I/O, and a clear abandoned-work policy.
Mistake 4: Draining without stopping admission
New jobs can keep the queue non-empty forever. Close admission before waiting for the drain deadline.
Mistake 5: Missing per-replica limits
A pool of 20 workers on 10 replicas is 200 concurrent calls. State whether the limit is local, sharded, or globally coordinated.
Mistake 6: No result retention policy
Futures and polling records need expiry, authorization, and a failure path. Otherwise accepted jobs become unbounded storage.