Prompt and scope
A cluster enables a scheduling plugin that calls the API server. When API latency rises, synchronous calls occupy scheduling cycles and unscheduled Pods accumulate. Design the asynchronous approach described by KEP-5229, covering a priority queue, request coalescing, retries, cancellation, fairness, and rollback.
What the interviewer is testing
- Whether you separate serial scheduling-cycle semantics from asynchronous API side effects.
- Whether a bounded priority queue and coalescing prevent starvation and duplicate writes.
- Whether you define idempotency keys, deadlines, cancellation, stale results, and permanent errors.
- Whether you cover priority fairness, backpressure, metrics, and feature-gated rollout.
Clarifying questions to ask
- Is the call a read, an idempotent write, or external-resource creation? Side effects define retry limits.
- Can the plugin accept eventual external state, or must it confirm state before binding?
- Does failure requeue the Pod as unschedulable, or only retry the API operation? Keep those loops separate.
- What are the cluster API QPS, concurrent scheduling level, and queue budget?
30-second answer framework
Move slow API operations from scheduling threads to a bounded priority queue. The scheduler submits a task with an idempotency key and continues with other Pods. Serve by Pod priority and wait time, coalescing identical keys. Completion only triggers a safe re-evaluation; stale results cannot bind a Pod. Classify errors into retryable and permanent, with deadlines, cancellation, backoff, and concurrency limits. Roll out behind a feature gate using queue depth, wait time, success rate, and scheduling latency; disable the asynchronous path and fall back when thresholds fail.
Step-by-step deep dive
1. Define the synchronous boundary
The scheduling cycle selects a node and owns the scheduling context; an asynchronous task performs work that may be delayed. If the result is a hard binding prerequisite, model it as pending and prevent binding until confirmation. Only work that cannot compromise the current cycle may be made asynchronous.
2. Build a bounded priority queue
Each task carries a Pod UID, operation type, idempotency key, deadline, and cancellation context. The queue has a capacity limit. When full, return an explicit backpressure signal instead of accumulating unbounded memory. Serve high-priority work first, while aging or quotas prevent permanent starvation.
submit(key, priority, deadline, operation)
if same key is pending: coalesce(operation)
else if queue is full: return Backpressure
else enqueue(operation)3. Coalescing and idempotency
Coalescing merges concurrent requests for one logical operation; it does not provide API-server idempotency. Writes need stable resource keys, conditional updates, or server-side idempotency. Check the prior result before retrying so an external resource is not created twice. Requests for different versions or targets must not be merged because their strings look similar.
4. Completion, cancellation, and stale results
Publish a completion event that asks related Pods to re-evaluate; do not assume the scheduling state is still valid. Cancel a task when the Pod is deleted, preempted, or enters a new scheduling context. Drop results past their deadline and record why. A successful API call from an expired context must not perform an old bind.
5. Retry, backpressure, and fairness
Retry timeouts, transient network errors, and 429 responses with exponential backoff. Authentication failures, invalid parameters, and conflicts need permanent-failure handling or a fresh calculation. Bound worker count, per-plugin quota, and API QPS. Count Pod scheduling retries separately from API-task retries, or one slow operation can amplify the queue.
6. Compatibility, rollout, and rollback
Preserve plugin APIs and scheduling semantics while putting the asynchronous path behind a feature gate. Start with low-risk plugins and low-concurrency clusters. Compare scheduling P99, queue wait, API latency, task failures, duplicate requests, and unschedulable retries. If backlog, binding errors, or API pressure exceed baseline, stop new submissions, drain or cancel tasks, and return to the synchronous path.
Model high-quality answer
First classify which calls may be delayed and which are binding prerequisites. Put the former in a bounded priority queue and represent the latter as an explicit pending state. Include Pod UID, operation type, idempotency key, deadline, and cancellation context; coalesce equal keys and apply backpressure when full. Workers execute only idempotent or safely retryable operations, backing off transient errors and handing permanent errors to plugin failure handling. Completion triggers re-evaluation, never an unconditional bind. Roll out behind a feature gate and measure scheduling P99, queue depth, API QPS, duplicate rate, failure rate, and binding correctness. On a breach, stop submissions, clean up tasks, and fall back.
Common mistakes
- Send every call to the background → a binding prerequisite may be bypassed → draw the scheduling state machine first.
- Use one global FIFO → high-priority Pods wait behind low-priority work → add priority, aging, and quotas.
- Deduplicate by request text → versions or side effects may be merged incorrectly → use resource keys and explicit idempotency.
- Retry forever → an API outage becomes a queue storm → use deadlines, backoff, attempt limits, and a circuit breaker.
- Measure throughput only → latency and correctness regressions stay hidden → observe queue, API, retries, and binding.
Follow-up questions and responses
Can a read and write for the same Pod be coalesced?
Not from the Pod UID alone. A read may coalesce by resource version; a write needs operation type, target, and idempotency key, while preserving required ordering.
Which task should be dropped when the queue is full?
Use deadline, priority, and reconstructability. A non-droppable binding prerequisite should create backpressure. Any dropped task must lead to an explainable retry or failure state, never silent deletion.
What if the API call succeeds after the Pod was preempted?
Use cancellation and object-version checks to prevent the old context from writing further. Keep the success for audit, then let the new scheduling context recalculate instead of reusing a stale bind intent.