Prompt and scope
A multi-tenant cluster has GPU memory and virtual-NIC bandwidth that cannot be allocated only as whole devices. The platform wants several Pods to share one device, while every request has a minimum, step, upper bound, and traceable allocation identity. Design the resource model and end-to-end flow with Kubernetes DRA consumable capacity.
Assume Kubernetes 1.34 and a driver that can enforce capacity limits. The scheduler must not oversubscribe a device. Kubernetes 1.34 made core DRA APIs generally available while consumable capacity is an alpha capability; a strong answer marks the boundary between stable APIs, feature gates, and driver-specific guarantees.
What the interviewer evaluates
The answer should assign clear responsibilities to DeviceClass, ResourceSlice, ResourceClaim, DeviceRequest, the scheduler, and the driver. It should state the invariant that allocated capacity for one device never exceeds total capacity, and explain allowMultipleAllocations, RequestPolicy, ShareID, and DistinctAttribute.
The interviewer also checks whether you confuse a successful scheduling decision with application-level throttling. A production design needs driver enforcement, status reporting, namespace authorization, node-failure recovery, and a reversible rollout.
Clarifying questions
Can the resource be isolated?
Confirm whether hardware or the driver can enforce isolation. If it cannot, the platform can offer soft quotas or whole-device allocation, but it cannot promise QoS from an API field.
What are sharing and authorization boundaries?
Ask whether namespaces may share a device, whether admin access is allowed, and which device attributes or capacities tenants may read. The answers change selectors, admission checks, and audit scope.
What must survive a failure?
Clarify what happens after scheduling failure, driver allocation failure, node loss, or Pod recreation: release, lease retention, or requeue. Transient retries and permanent hardware failures need distinct states.
A 30-second answer framework
“I define the capacity invariant and tenant boundary first. DeviceClass describes eligible devices, ResourceSlice publishes capacity and request policy, and ResourceClaim expresses a Pod’s need. The scheduler selects a device without exceeding capacity; the driver uses ShareID to enforce the actual memory or bandwidth limit and reports status. Cross-namespace sharing requires authorization and audit. Every release is idempotent. I roll out one device class at a time and compare allocated capacity, actual limits, scheduling latency, rejection rate, and recovery time.”
Step-by-step deep dive
Step one: define the resource model
DeviceClass describes a device type and CEL selectors. ResourceSlice publishes each device, its attributes, capacity, and whether multiple allocations are allowed. ResourceClaim uses a DeviceRequest to state the class and capacity. Keep total device capacity separate from request capacity; a device count of one is not a capacity value.
Step two: state the capacity invariant
For each device, track allocated capacity, units, and the request policy. If a device has 40 GiB and requests must be at least 5 GiB in 5 GiB steps, every request must satisfy those bounds and all active ShareIDs must sum to at most 40 GiB. Release and retry operations use an allocation version so duplicate callbacks cannot subtract twice.
Step three: describe scheduling and driver data flow
The scheduler reads ResourceSlice, filters selectors, capacity ranges, and allowMultipleAllocations, reserves a candidate, and binds the ResourceClaim. The driver receives the allocation, creates an independent limit keyed by ShareID, applies it to the device, and reports dynamic data in ResourceClaim status. If scheduling succeeds but the driver rejects the allocation, the controller exposes a retryable or terminal state; it must not make the Pod appear Ready.
Step four: handle namespaces and duplicate devices
A shared device does not erase the ResourceClaim namespace boundary. Admission should restrict DeviceClass use, admin access, and driver configuration. DistinctAttribute prevents one claim from selecting the same underlying device twice, such as when two network interfaces must reach different subnets. Audit tenant, claim, ShareID, capacity, and policy version rather than only the final device name.
Step five: design failure and reclamation
After a driver restart, recover ShareIDs and actual limits from durable state. After node loss, mark allocations unknown and do not immediately give the capacity to another Pod until a lease, device fence, or driver confirmation proves the old allocation is gone. Pod deletion, claim expiry, and scheduling rollback must be repeatable. Existing allocations keep their policy snapshot; new claims use a changed policy.
Step six: rollout, SLOs, and capacity math
Assume 100 devices with 40 GiB each and a 70 percent target average utilization. The logical schedulable capacity is about 2,800 GiB, not a throughput promise; reserve driver overhead, fragmentation, and failure headroom. Set SLOs for scheduling p99, driver-configuration p99, capacity rejection rate, and status convergence. Enable the feature gate per device class. Compare throughput, tail latency, fragmentation, and recovery with whole-device allocation.
Step seven: compare an alternative
If hardware supports only fixed slices, MIG or pre-partitioned DeviceClasses are simpler and easier to prove, at the cost of fragmentation and less elasticity. If the driver cannot enforce fine-grained limits, fall back to whole-device allocation or capacity expansion; do not pretend that a capacity field provides isolation.
High-quality sample answer
I first verify that the device can enforce capacity isolation. If it cannot, the design is declarative selection only and has no QoS promise. If it can, DeviceClass describes eligible devices, ResourceSlice publishes capacity and request policy, and ResourceClaim carries the tenant request. The scheduler binds only when selectors match and the capacity sum stays below the device limit; the driver then creates a ShareID-scoped limit and reports status.
I make capacity sum, policy version, and idempotent release explicit invariants. Cross-namespace sharing uses admission, admin authorization, and audit; DistinctAttribute prevents duplicate underlying-device choices within one claim. On driver or node failure, I retain an unknown allocation until fencing or confirmation before reclaiming it. Rollout starts with one class and measures scheduling p99, driver p99, rejection, actual enforcement, and recovery. Fixed-slice hardware uses a pre-partitioned design.
Common mistakes
- Symptom → put a capacity value only in ResourceClaim → Why it fails → the driver may not enforce a real limit → Fix → prove the enforcement and status path.
- Symptom → use device count as the capacity sum → Why it fails → concurrent requests oversubscribe or waste fragments → Fix → track allocated capacity per device and version.
- Symptom → treat shared devices as unconditional cross-tenant sharing → Why it fails → namespace and authorization boundaries disappear → Fix → add admission, admin access control, and audit.
- Symptom → release capacity immediately after node loss → Why it fails → the old driver may still enforce the allocation, creating double assignment → Fix → fence, confirm a lease, or keep an explicit unknown state.
- Symptom → promise stable SLOs for an alpha feature gate → Why it fails → API, driver, and upgrade behavior vary → Fix → mark version boundaries and validate through a canary.
Follow-ups and strong responses
Follow-up one: Two namespaces request the last 10 GiB simultaneously. How do you avoid a race?
Use a ResourceSlice version or equivalent optimistic concurrency check when reserving capacity. If binding fails, reread current state and retry. A driver callback cannot bypass the scheduler’s invariant; the control plane confirms the final allocation.
Follow-up two: The driver reports a ShareID, but the Pod never starts. What happens?
Separate allocation state from Pod readiness. After a timeout, the controller performs an idempotent release or enters an operator-visible state while retaining the ShareID, claim version, and reason. Do not discard capacity or audit data merely because the Pod is not Running.
Follow-up three: The policy changes from a 5 GiB minimum to 10 GiB. Do existing claims change?
Completed allocations keep the policy snapshot under which they were granted; new claims use the new policy. Rebalancing requires an explicit migration flow, driver support for resizing, and a reversible release-and-reallocate sequence.
Follow-up four: How do you prove shared bandwidth is actually enforced?
Run a single-tenant baseline and a multi-ShareID load test with fixed traffic, nodes, and driver versions. Compare per-tenant throughput, p99, drops, and the enforcement ceiling. ResourceClaim status alone does not prove hardware QoS.