Prompt and scope
A multi-tenant machine-learning platform uses GPUs, FPGAs, and high-speed NICs. Its current design allocates whole devices through node labels, extended resources, and device plugins, making device attributes, sharing, and tenant isolation difficult to express. The team wants to adopt Dynamic Resource Allocation (DRA), whose core resource.k8s.io/v1 APIs became stable in Kubernetes 1.34.
Design the migration and explain the responsibilities of ResourceClaim, DeviceClass, ResourceClaimTemplate, ResourceSlice, the driver, and the scheduler. The cluster must migrate gradually without restarting every workload at once.
What the interviewer evaluates
The interviewer wants to see a separation between declaring a device need and allocating a concrete device. Explain the data flow among the control-plane APIs, scheduler, node driver, and kubelet, and distinguish stable DRA APIs from capabilities that remain experimental or driver-specific.
Strong answers cover capacity sharing, cross-namespace references, driver failure, Pod replacement, coexistence with old workloads, least privilege, and rollback signals instead of merely listing resource kinds.
Clarifying questions before answering
- Is the request for a whole device, a sliceable capacity, or any device matching attributes?
- Does the driver support DRA, and can the old device plugin run during migration?
- Should workloads create Claims directly, or should platform templates create them?
- May tenants reuse Claims across namespaces, and which objects may the platform controller write?
- During migration, is the priority job continuity, scheduling throughput, or utilization?
A 30-second answer framework
“I would model device needs as Claims, let the driver discover, allocate, and configure devices, and let the scheduler make feasibility decisions from Claims and ResourceSlices. I would canary a small device class in an isolated namespace. The old plugin and DRA may coexist by pool, but they must never own the same device. Each phase measures Claim state, Pod binding, node visibility, driver errors, allocation latency, and tenant boundaries. If allocation fails, I stop new Claims, preserve running jobs, and route new workloads back to the old path.”
Step-by-step deep answer
Draw the DRA data flow first
A workload references a Claim through the Pod's resourceClaims field. A Claim may be created directly or generated for a Job by a ResourceClaimTemplate. A DeviceClass describes a selectable class of devices, while a driver publishes node devices and attributes through ResourceSlice. The scheduler combines the Pod, Claim, DeviceClass, and ResourceSlice to select a node; the node driver then performs the concrete allocation and configuration.
This separates device selection from application YAML and avoids encoding hardware topology in label conventions.
Design Claim templates and tenant boundaries
Expose only tenant-facing parameters such as memory tier, interconnect type, or network bandwidth. Do not expose driver-internal identifiers. A platform controller applies namespace admission policy, quota, and lifecycle cleanup. Cross-namespace Claim use requires explicit reference permissions and audit events. During tenant deletion, stop new Claims, wait for Pods to release devices, and then reclaim device state.
Handle whole devices and shareable capacity
Whole-device allocation can map one DeviceRequest to one device. Sharing memory, bandwidth, or time slices requires a driver to publish capacity and support the relevant DRA consumable-capacity behavior. Do not simulate sharing with scheduler labels; scheduling can succeed while the node oversells the real capacity. Define units, concurrency limits, release timing, and fragmentation policy.
Coexist with device plugins
Partition migration by device class or node pool so that the old plugin and DRA never advertise ownership of the same hardware. Existing workloads keep extended resources; new workloads reference DRA Claims. Pool labels are a migration boundary, not the final source of device attributes. Every pool needs explicit ownership and a disable switch to prevent duplicate driver initialization.
Handle scheduling, preemption, and failure
When a Claim waits or fails, the Pod should expose an explainable Pending reason. Selecting a node does not mean the device is configured; the driver may fail during node setup. A controller should classify failures as retryable, non-retryable, or requiring cleanup and send each class to the right alert. Preemption must consider Pod priority, occupied Claim capacity, and release latency, not only CPU or memory.
Observe and plan capacity
Measure Claim creation to binding, binding to node allocation, and allocation to Pod Ready as separate stages. Track idle, allocated, unavailable, fragmented capacity, and driver errors. Reconcile capacity advertised by ResourceSlices with what nodes actually expose; stop the canary expansion when they drift. Training platforms should also record retries, checkpoint recovery, and device-health events.
Canary, rollback, and consistency
Start with one driver and one node pool using retryable jobs. Expand to multiple tenants and templates next; migrate long-running, non-interruptible jobs last. Rollback does not mean deleting bound Claims. Stop new Claim creation, let running Pods finish or migrate, freeze driver changes, and route new jobs to the old plugin. Retain Claim, Pod, and driver events so the incident remains diagnosable.
Validate APIs and security boundaries
Before deployment, run contract tests against the resource.k8s.io/v1 APIs for Claim, Template, DeviceClass, and ResourceSlice versions, state transitions, and deletion order. Inject driver restarts, node loss, duplicate allocation, leaked Claims, and cross-namespace access. Check admission, RBAC, audit, and node-agent permissions. Expand the device pool only after functionality, isolation, and recovery metrics pass.
High-quality model answer
“I would split the migration into resource model, driver, scheduling, and runtime layers. Workloads declare Claims; Templates generate them; DeviceClass expresses selection; the driver publishes ResourceSlices and allocates on the node; the scheduler handles feasibility and node choice. Old plugins and DRA coexist by pool or device class, never with double ownership. For sharing, I require a driver capacity model and release semantics rather than label-based overselling. I start with retryable jobs, measure Claim-to-Ready latency, allocation failures, fragmentation, driver health, and tenant violations, then stop new Claims and preserve state before routing new jobs back during rollback.”
Common mistakes
- Treating DRA as new device labels → scheduling succeeds without node-level fulfillment → have the driver publish structured devices and capacity.
- Letting the plugin and DRA own one device → duplicate initialization or allocation → partition ownership by pool or device class.
- Designing Claims without reclamation → device and quota leaks accumulate → define release, timeout, deletion, and reconciliation.
- Simulating sharing with labels → the scheduler cannot see real remaining capacity → make the driver publish capacity and concurrency.
- Equating node selection with allocation success → driver errors leave Pods Pending indefinitely → separate scheduling, node allocation, and Ready metrics.
- Deleting every Claim during rollback → running jobs and evidence disappear → freeze new allocation and preserve state.
Follow-up questions and responses
Follow-up 1: Why not keep registering GPUs as extended resources?
Extended resources fit simple quantity requests but do not express device attributes, alternatives, shared capacity, or complex configuration well. If the requirement is only whole-device counting and the driver is mature, the old path can remain. DRA should be introduced when device selection and lifecycle complexity justify it.
Follow-up 2: How do you prevent overselling when several Pods reference one Claim?
Reuse semantics must be explicit in the driver and platform policy, not assumed from YAML. For shared capacity, the driver records allocated capacity, concurrency limits, and release state; admission limits references that violate tenant policy.
Follow-up 3: Should the scheduler retry a driver failure during node setup?
Classify transient node failure, device-health failure, and invalid Claim parameters first. Retry transient failures with backoff; mark invalid parameters non-retryable; isolate unhealthy nodes so the scheduler does not repeatedly send work to the same failure domain.
Follow-up 4: How do you prove migration did not reduce utilization?
Compare allocation success, wait time, fragmentation, useful compute time, and job retry rate for similar workloads in the same device pool. Split DRA and the old plugin by device class; a cluster-wide average GPU utilization is insufficient.