System Design Interview: How Would You Design a Kubernetes Pod-Level Resize Controller?

Prompt and context

A multi-container Pod has an API proxy, cache sidecar, and batch worker. The team wants to adjust Pod-level CPU and memory budgets from latency and queue length without deleting the Pod. Design the controller, including observation, decisions, /resize writes, status tracking, concurrency protection, infeasible requests, and rollback.

What the interviewer tests

• Whether you distinguish desired resources, actual resources, and container restart policy.
• Whether you design idempotent reconciliation, status conditions, and deferred/infeasible retries.
• Whether Pod budgets and container requests have explicit boundaries and safety rails.
• Whether metrics, permissions, recovery, and progressive rollout are part of the design.

Questions to clarify

1. Does the cluster support Pod-level resize, and which feature gates and kubectl version are installed?
2. Does the controller change CPU, memory, or both Pod and container resources?
3. Which containers may restart, and which have non-interruptible connections or state?
4. Is the goal cost reduction, an SLO, or absorbing a bursty queue?

30-second answer

I would build an event-driven idempotent reconciler: read metrics and Pod state, calculate a target within budgets, SLOs, node capacity, and a cooldown, then submit a small change through the /resize subresource. Status records observedGeneration, PodResizePending, PodResizeInProgress, and Infeasible or Deferred reasons; retries use backoff, priority, and a maximum count. CPU and memory are evaluated separately, container resizePolicy is respected, and the last verified budget is restored on memory risk or SLO regression. Every write has an owner, audit trail, and RBAC boundary.

Step-by-step deep dive

Define the resource model and safety boundary

Pod-level spec.resources is an aggregate budget; container requests and limits still affect guarantees and restart behavior. Keep per-workload minimum, maximum, step, cooldown, and SLO guardrails, rejecting requests beyond namespace quota or node capacity.

Collect metrics and calculate a target

Use latency, queue length, CPU throttling, working set, and OOM events. Apply windows and hysteresis to avoid reacting to one spike; the target must satisfy both the Pod budget and the sum-of-container-request constraint.

Submit an idempotent resize-subresource update

The controller updates desired state with a resource version and owner. Example request:

spec:
  resources:
    requests:
      cpu: "300m"
      memory: "512Mi"
    limits:
      cpu: "1"
      memory: "1Gi"

Call the /resize subresource and check resourceVersion. On a conflict, reread and reconcile instead of overwriting a user or another controller.

Track conditions and retry priority

Read conditions such as PodResizePending and PodResizeInProgress plus observedGeneration. Infeasible means current constraints cannot satisfy the request; Deferred means it is postponed. Persist reason, next attempt, and count. Schedule retries by workload priority, QoS, and wait time so low-priority work cannot starve forever.

Handle container restart policy

Pod-level changes can trigger container-level resizePolicy. CPU may apply without restart while memory can require one; inspect each container’s policy, connections, and state. Put non-restartable requests behind a safety queue and never report business continuity from Pod-level success alone.

Observe, roll back, and stay highly available

Record target and actual values, condition transitions, restart count, latency SLO, memory peak, and failure reason. Run controller replicas with leader election and deduplicate the queue by Pod key. On a bad budget, OOM, or SLO regression, restore the last stable target and pause automation for human review.

Model answer

I would build an idempotent reconciler that reads latency, queue, throttling, working set, and OOM metrics, then calculates a target within minimum/maximum budgets, step, cooldown, node capacity, and namespace quota. Submit small resource-versioned changes through /resize; reread on conflicts. Record observedGeneration, Pending, InProgress, Infeasible, Deferred reason, and retry time, scheduling retries by priority and wait. Inspect every container’s resize policy before a memory change can restart it. Use leader election, metrics, and audit logs; restore the last stable budget and pause automation on SLO regression or OOM.

Common mistakes

• Editing Pod spec directly and assuming kubelet applies it, ignoring the /resize subresource.
• Reading desired resources only and ignoring actual resources and conditions.
• Omitting cooldown and hysteresis, causing oscillation and frequent restarts.
• Dropping Deferred requests forever or retrying Infeasible requests without a limit.
• Ignoring container resizePolicy and memory-restart risk.
• Lacking resourceVersion, owner, and RBAC, allowing controllers to overwrite each other.

Follow-up questions

How do you stop two controllers overwriting each other?

Use explicit ownership, resourceVersion, field management, and one write owner; reread and merge intent on conflicts instead of unconditional overwrite.

How do you distinguish Infeasible from Deferred?

Infeasible means current constraints cannot satisfy the target and require a new target or capacity; Deferred is temporary and should retry with its reason and priority preserved.

When should automation pause?

Pause on OOM, sustained SLO regression, unchanged conditions, excessive restarts, or invalid observations, while keeping a manual recovery path.

How do you prove no interruption occurred?

Correlate resize conditions, container restartCount, connection errors, latency, and queue metrics by policy; Pod phase remaining Running is insufficient.