System design interview: how would you design topology-aware gang scheduling for Kubernetes workloads?

Problem and applicable scenarios

You own a distributed-training platform on Kubernetes. A workload is represented by a PodGroup, and its Pods should share a rack or zone to reduce communication latency. If one topology domain cannot fit at least minCount Pods, the workload should remain unschedulable.

This fits platform-engineering, SRE, and system-design interviews. Assume Kubernetes v1.36 alpha Topology-Aware Scheduling, disabled by default, with one topology constraint per PodGroup.

What the interviewer is evaluating

• Whether you distinguish cross-domain balancing with topology spread from same-domain placement for a gang.
• Whether PodGroup, node labels, candidate placements, resource feasibility, and atomic binding form one data flow.
• Whether you explain minCount, capacity shortage, scale-up, and the absence of topology-triggered preemption.
• Whether you identify alpha status, heterogeneous groups, and rollback risk before calling the design production-ready.

Clarifying questions before answering

1. Must Pods share a rack, zone, or NUMA domain? Each topology key changes latency and capacity assumptions.
2. Must every Pod start together, or is minCount enough to make progress? This determines Gang policy and elasticity.
3. Are Pods homogeneous? Different resource requests affect whether a candidate placement can be found.
4. May the cluster scale or preempt? v1.36 TAS does not preempt merely to satisfy topology.

A 30-second answer framework

“I define the goal as group-level feasibility inside one topology domain, not replica spreading. A Workload template creates a PodGroup with gang and minCount; node labels provide the topology key. The scheduler generates candidate node subsets, checks the entire group, and scores feasible placements. If none fits, the group stays unschedulable. The controller and autoscaler reason about the whole resource shape, and preemption is handled separately because v1.36 TAS does not trigger it. I test domain capacity, node loss, and heterogeneous requests behind an alpha feature gate.”

Step-by-step deep dive

1. State co-location, not spreading

topologySpreadConstraints use maxSkew to balance Pods across domains; this question requires all members to share one topology-label value. Mixing the semantics produces the opposite placement, so write the invariant as “one domain must fit at least minCount.”

2. Define the PodGroup contract

The template declares Gang policy, minCount, and one topology key. An illustrative configuration is:

apiVersion: scheduling.k8s.io/v1alpha2
kind: PodGroup
spec:
  schedulingPolicy:
    gang:
      minCount: 4
  schedulingConstraints:
    topology:
      - key: topology.example.com/rack

Nodes need stable, governed label values. PodGroup expresses placement; the Workload controller still owns member creation, versions, and lifecycle.

3. Explain candidate placement

The scheduler first generates candidate node subsets using resources, taints, affinity, and the topology key. It then checks whether the full PodGroup fits within each subset and scores feasible placements. This avoids binding early Pods across domains and leaving the remaining members unable to satisfy minCount.

4. Handle capacity, scale-up, and preemption

If no candidate domain has enough capacity, the whole group remains unschedulable. The autoscaler should consume the complete CPU, memory, GPU, and domain shape rather than scaling for the first Pending Pod. v1.36 TAS does not trigger Pod or Workload preemption to satisfy topology, so capacity reservations, queues, or an explicit upstream preemption policy are separate requirements.

5. Handle member recreation and domain stickiness

When members already run in a domain, recreated Pods are forced into that same domain; they remain Pending if it lacks capacity even when another domain has room. Expose this as a domain-sticky wait and decide whether to wait, restore a checkpoint, or create a new PodGroup generation with a different constraint.

6. Evaluate alpha and heterogeneous limits

The v1.36 API is alpha and disabled by default. The official release notes state that heterogeneous PodGroups or inter-Pod dependencies are not guaranteed to find a placement even when one exists. Production gates should cover version, feature gate, scheduler plugin, autoscaler, and rollback; one successful schedule is not a universal guarantee.

7. Build a verification and rollback loop

Run the same training workload with just-enough domain capacity, one missing node, missing topology labels, different GPU types, and member recreation. Record candidate domains, feasible-member count, Pending reasons, wait time, cross-domain traffic, and training throughput. If canary tests fail, disable the feature gate or return to ordinary Gang scheduling while preserving PodGroup events for audit.

High-quality sample answer

I would define the invariant as “at least minCount members fit simultaneously in one topology domain,” rather than using spread constraints for balance. A Workload controller creates a PodGroup with Gang, minCount, and one topology key, then creates Pods that reference it. The scheduler generates candidate domains from resources and node labels, checks the complete group, and scores only feasible domains; if none works, the group stays Pending. The autoscaler scales from the group shape, while preemption is separate because v1.36 TAS does not trigger it. Recreated members keep domain stickiness; if that domain is lost, create a new generation. Finally, exercise node loss, missing labels, heterogeneous GPUs, and rollback behind the alpha gate while measuring wait time, cross-domain traffic, and training throughput.

Common mistakes

• Symptom: Explain same-domain placement with maxSkew. Why it fails: spread targets distribution, opposite to Gang co-location. Fix: state the PodGroup single-domain invariant first.
• Symptom: Bind Pods one by one. Why it fails: early bindings consume capacity in several domains and leave no way to reach minCount. Fix: evaluate the complete candidate placement first.
• Symptom: Assume TAS automatically preempts. Why it fails: v1.36 explicitly says topology-aware scheduling does not trigger preemption. Fix: design reservations, queues, or an upstream preemption controller.
• Symptom: Ignore heterogeneous groups and alpha status. Why it fails: placement is not guaranteed and the feature is disabled by default. Fix: add version, gate, plugin, and rollback checks.

Follow-up questions and answers

What if one rack cannot fit minCount, but two racks together can?

The same-domain invariant is unmet, so keep the group Pending or lower minCount only if the application permits elasticity. Do not silently fall back to cross-rack placement because the communication assumption changed.

How should the autoscaler scale correctly?

Treat the full resource request, topology key, and minCount as one scale unit. Predict the node types and count required in a target domain, regenerate candidates after scale-up, and enforce a wait deadline.

Why can a recreated member not move to another domain?

The documented behavior keeps new members in the domain where existing members run; moving one changes latency and shared-resource assumptions. If the domain is permanently lost, create a new PodGroup generation instead of silently mutating the old one.

How can this coexist with topology spread constraints?

Use TAS to co-locate members inside one training job; use spread to distribute replicas across multiple jobs. Before putting both on one Pod, test whether their hard constraints have a non-empty intersection or Pods may remain Pending.

When would ordinary Gang scheduling be better than TAS?

Choose ordinary Gang when cross-domain communication is cheap, domain capacity is frequently tight, the alpha feature is not ready for your rollout, or the workload does not need shared racks. Enable TAS only when throughput gains justify capacity and operational cost.