Prompt and context
A platform team wants to change CPU and memory without recreating Pods. Design an in-place resize flow, explain when a container must restart, how to handle insufficient capacity, and how to observe and roll back.
Kubernetes 1.33 graduated container in-place resize to Beta; Pod-level resource resize is Beta in 1.36. A running Pod can receive new CPU and memory resources, while kubelet uses resizePolicy, node capacity, and cgroup state to apply or defer the change. “Change the spec” does not mean “never restart.”
What the interviewer is testing
Cover Pod-level bounds versus container resources, CPU and memory semantics of resizePolicy, Pending and InProgress conditions, infeasible and deferred requests, scheduler versus kubelet responsibilities, QoS and priority, and canary and rollback boundaries.
30-second answer framework
“I would make resize a declarative request plus a state machine. A controller updates the resource spec, kubelet checks feasibility, and reports PodResizePending or PodResizeInProgress; CPU often applies without restart, while memory restart behavior follows each container’s resizePolicy. An infeasible request keeps its reason and retries under limits. The platform observes observedGeneration, actual cgroups, restarts, and QoS changes, using small canaries and a reverse patch for rollback.”
Step-by-step deep dive
Step 1: Define the resource model
Container resources set requests and limits for each container; supported versions also expose Pod-level aggregate bounds. A Pod limit caps aggregate use, but a container limit cannot exceed it. The API must state whether it edits spec.containers[*].resources or spec.resources.
Step 2: Build a declarative resize flow
The platform accepts a target and reason, writes resource fields, and stores the old value, actor, and generation. The API never edits node cgroups directly. Kubelet observes the new generation, checks and applies it, then writes status.
Step 3: Handle restart policy
A container can choose a policy per resource:
resizePolicy:
- resourceName: cpu
restartPolicy: NotRequired
- resourceName: memory
restartPolicy: RestartContainerNotRequired attempts an online update; RestartContainer permits a restart to apply the new value. For a Pod with restartPolicy: Never, every container resource must use NotRequired, or the request is invalid.
Step 4: Model feasibility and deferred status
When the node cannot provide the target, kubelet reports PodResizePending with reasons such as Infeasible or Deferred; a processed request can enter PodResizeInProgress. Controllers should read status instead of only the API spec and use observedGeneration to associate status with the requested generation.
Step 5: Treat CPU and memory differently
CPU can usually update cgroup quotas without restarting the application. Memory downsizing can be constrained by the current working set, and some runtimes need a restart to apply it safely. Do not infer memory behavior from CPU; read the per-resource policy and emit an explicit event.
Step 6: Coordinate scheduling and QoS
An in-place update is not a reschedule. Increasing a request may require node capacity, so deferred requests should retry by PriorityClass, QoS class, and wait time. Recalculate quotas, QoS, and alerts after the update so a namespace or Guaranteed workload is not silently under-accounted.
Step 7: Observe the effective value
Collect desired spec, status conditions, observedGeneration, actual container cgroups, restarts, OOMs, CPU throttling, and memory working set. If status says complete but cgroups did not change, treat it as failure and block another patch instead of stacking changes.
Step 8: Canary, rate-limit, and roll back
Resize in batches by workload and node pool, limit concurrent operations, and set Pending timeouts and retry caps. On failure restore the saved resources; if memory policy restarts a container, drain traffic and verify readiness before expanding. Every request needs an idempotency key to prevent controller retries from duplicating work.
Trade-offs and boundaries
Restart-free continuity versus resource certainty
In-place changes reduce interruption, but capacity and deferred state make completion asynchronous. Use small steps for latency-sensitive services; batch jobs may accept restarts for deterministic results.
Pod bounds versus container precision
Pod resources express a shared ceiling while container resources protect critical containers. When both exist, validate that container limits stay within the Pod bound and show the effective boundary in UI and audit records.
Automatic retry versus approval
Temporary capacity shortage merits bounded retry. Memory restarts, QoS changes, or production peaks may require approval or a maintenance window; infinite retries are unsafe.
Failure drills and evolution plan
The node lacks capacity
Submit an expansion above node capacity, confirm Infeasible or Deferred, and verify retries do not alter the old effective value.
A memory change restarts the container
Set RestartContainer for memory and observe traffic drain, restart, readiness recovery, and event ordering. Ensure the controller does not report the restart as a business failure.
Generations race
Submit two targets quickly. The old generation must not overwrite the new target, and final observedGeneration must match the cgroup value.
Common mistakes and follow-ups
Mistake 1: Assuming in-place resize never restarts
Follow-up: What causes a restart? A container’s resizePolicy can require it, especially for memory; Pod-level resize has no separate restart policy, but container policy still applies.
Mistake 2: Treating a spec update as completion
Follow-up: How do you confirm success? Check conditions, observedGeneration, cgroups, container events, and restart count rather than desired fields alone.
Mistake 3: Patching repeatedly when capacity is missing
Follow-up: What is correct? Keep the Pending reason, retry within priority and time limits, and offer migration, a smaller target, or approval.
Extended follow-ups and model answers
Why design CPU and memory separately?
CPU quotas can usually change online; memory downsizing is constrained by working set and runtime behavior and may restart. A shared API still needs per-resource policy and state.
How do you stop a duplicate resize from overwriting a newer target?
Use resource version or generation for conditional updates, accept only the newest target, and reconcile status and cgroups through observedGeneration.
When should you avoid in-place resize?
Use a rolling replacement when strong isolation is required, capacity is unstable, the application cannot tolerate a restart, or resource changes violate runtime assumptions. Keep in-place resize as a controlled optimization.