Prompt and context
You operate a Kubernetes GPU platform. Training and batch-inference jobs must wait for quota, node resources, maintenance windows, and security policy before Pods are created. Design admission with Kueue Workloads, ClusterQueues, ResourceFlavors, and AdmissionChecks, while preventing starvation and unsafe release.
What the interviewer is testing
Separate queueing from admission: a Workload entering a queue does not mean Pods may be created. A ClusterQueue chooses flavors from resource groups and nominal quota, while AdmissionChecks let internal or external controllers influence release. Cover state consistency, revocation, partial admission, tenant fairness, backoff, and auditability.
Clarifying questions to ask first
Resources and priority
Clarify GPU model, memory, CPU, topology, preemption, tenant priority, queue fairness, and maximum wait. Counting GPUs alone can create false capacity when memory or topology does not fit.
External admission dependencies
Identify controllers for maintenance windows, image scanning, budget approval, and data access. Decide whether results are replayable and whether timeout means reject or wait. Every external check needs an owner and TTL.
Failure and revocation policy
Define what happens when nodes change, quota is reclaimed, or policy is revoked after admission. Admission records, Pod creation, and external checks need an idempotent correlation to prevent duplicate release.
30-second answer framework
“A Workload enters a LocalQueue, and its ClusterQueue evaluates resource groups, flavors, and cohort quota. All required AdmissionCheckStates must be Ready before admission; Pending remains queued, while Rejected or timeout follows a bounded retry or termination policy. Persist a traceable workload state, re-check resources and policy before creating Pods, and measure quota use, wait time, check latency, rejection, and revocation to validate safety and fairness.”
Step-by-step deep answer
Step 1: Define Workload and queue boundaries
Convert the user Job into a schedulable Workload with PodSets, resource requests, priority, and tenant queue. A LocalQueue orders and waits; it must not create Pods or bypass ClusterQueue resource constraints.
Step 2: Select ResourceFlavors through the ClusterQueue
Describe CPU, memory, and GPU resources with ResourceGroups, then choose a flavor combination with available nominal quota. Put GPU model, region, node labels, and topology in the flavor constraints so “enough GPUs” does not become a false admission.
Step 3: Orchestrate the AdmissionCheck state machine
Expose observable Pending, Ready, Rejected, and terminal states. Security, maintenance, and budget controllers update state idempotently using the Workload UID. Kueue admits only when every required check is Ready; Pending must not be mislabeled as failure, and Rejected cannot be silently ignored.
Step 4: Handle partial admission and resource changes
If partial admission is allowed, define reducible PodSet parallelism, minimum size, and later expansion rules. Re-evaluate flavors and checks after quota release or node failure; never reuse an expired admission snapshot. Permanently infeasible requests need an explanatory rejection rather than infinite retries.
Step 5: Preserve fairness and prevent starvation
Define cohort sharing and borrowing boundaries across tenants, queues, and priorities. Prevent high-priority jobs from occupying scarce GPUs indefinitely; add wait or aging rules for lower priorities. Observe quota allocation, check waiting, and actual Pod startup separately so a fast admission with slow startup is visible.
Step 6: Design revocation, retry, and audit
Use jittered backoff and retry limits for controller timeout or failure. Revocation must safely propagate to the Workload and PodSets, not merely flip one field. Record actor, time, reason, flavor, quota version, and external-check version so decisions are replayable.
Step 7: Validate observability and failure drills
Monitor queue wait, admission latency, each check’s Pending duration, rejection, revocation, quota utilization, flavor selection, idle GPUs, and Pod startup. Drill controller outage, duplicate callbacks, quota reclamation, node failure, and network partition to prove the system neither releases unsafely nor remains stuck forever.
High-quality sample answer
I would place the Workload in a LocalQueue, let the ClusterQueue select a feasible resource combination from flavors and cohort quota, and use AdmissionChecks for security, maintenance, and budget conditions. Admit only when every required check is Ready and the resource snapshot is still valid; Pending stays queued and Rejected or timeout uses bounded backoff. GPU model, topology, and region belong in flavors. Tenant quota and aging protect fairness, while idempotent callbacks and revocation propagate safely to Workload and PodSets.
Common mistakes
- Mistake: Treating queue entry as permission to create Pods. → Why it fails: Queueing and admission are separate states. → Fix: Require ClusterQueue allocation and all checks Ready.
- Mistake: Selecting resources by GPU count only. → Why it fails: Model, memory, topology, or region may not fit. → Fix: Express hardware constraints with ResourceFlavors.
- Mistake: Retrying a Pending check forever. → Why it fails: Permanent infeasibility or controller failure stays hidden. → Fix: Distinguish Pending, Rejected, and reasons with limits and backoff.
- Mistake: Revoking only a Workload field. → Why it fails: Existing Pods may keep consuming resources. → Fix: Define PodSet action, audit, and rollback behavior.
Follow-up questions and answers
Follow-up 1: How does AdmissionCheck differ from a Kubernetes Admission Webhook?
AdmissionCheck is the Kueue business state for whether a Workload may start. A webhook extends the API request path. They can cooperate, but a webhook passing does not mean resources have been allocated.
Follow-up 2: Why are ResourceFlavors needed?
The same GPU count may represent different models, regions, or topologies. A flavor binds those schedulable attributes to resource-group quota, making selection explainable and safe.
Follow-up 3: How do you prevent duplicate callbacks from moving state backward?
Use Workload UID, check name, and version as an idempotency key. Reject stale versions and ensure repeated Ready updates cannot trigger a second Pod creation.
Follow-up 4: When should a request be rejected instead of kept Pending?
Reject when its resource flavor, policy, or budget can never be satisfied and provide the reason. Temporary node shortage, a maintenance window, or controller retry can remain Pending, but needs a maximum wait and alert.