How would you design Kubernetes Topology Aware Routing across zones?

Question and context

A three-zone Kubernetes cluster runs a service with expensive cross-zone traffic. The team wants requests to prefer Pods in the source zone while remaining available when a zone lacks capacity, traffic is skewed, or nodes fail. Design enablement, endpoint distribution, fallback, observability, and rollback.

What the interviewer evaluates

• Whether you separate EndpointSlice hints, kube-proxy consumption, and Service traffic policy.
• Whether you identify prerequisites such as endpoint count and balanced source traffic instead of assuming same-zone is always better.
• Whether you design global fallback, failure safeguards, and hot-spot monitoring.
• Whether you quantify cross-zone cost, latency, endpoint load, and rollout risk.

Clarifying questions first

Traffic and goals

Is source traffic evenly distributed across zones? Is the goal lower p95 latency, lower cross-zone cost, or data residency? Is cross-zone access preferable to returning an error?

Endpoints and scaling

How many ready Pods exist in each zone? Could scaling, rolling release, or a PodDisruptionBudget temporarily reduce the safe endpoint count? Does the Service also require node-local traffic?

Failure and observability

Is the failure domain a zone, node, or network? How will you detect an overloaded zone, missing hints, or kube-proxy fallback? Can the feature be enabled per Service?

A 30-second answer

I would first verify that source traffic and endpoint distribution satisfy the routing prerequisites, then enable a zone preference through EndpointSlice hints or the Service's supported traffic-distribution setting. The control plane and kube-proxy must fall back to cluster-wide endpoints when capacity or safety conditions fail. Monitor cross-zone bytes, p95 latency, per-zone requests, and endpoint load; canary the change and disable the preference if a hot spot or failure appears.

Deep-dive solution

1. Separate control and data planes

The EndpointSlice controller uses endpoint topology to produce hints, and kube-proxy or another node data-plane component consumes them. Topology Aware Routing prefers endpoints in the source zone; it does not guarantee strict same-zone routing. Observe control-plane calculation, EndpointSlice propagation, and node behavior separately.

2. Check prerequisites

Kubernetes documentation recommends at least three endpoints per zone; in a three-zone cluster that commonly means at least nine endpoints. With too few endpoints, the controller may emit no hints. A source distribution concentrated in one zone can also overload that zone, so validate the assumption with traffic history and capacity data.

3. Choose the configuration

Use the topology mode or supported trafficDistribution capability for the target Kubernetes version. For example:

apiVersion: v1
kind: Service
metadata:
  name: checkout
  annotations:
    service.kubernetes.io/topology-mode: "Auto"
spec:
  selector:
    app: checkout
  ports:
    - port: 443
      targetPort: 8443

Do not mix a legacy topology-aware-hints annotation, the current topology mode, and version-specific fields without checking the API version, feature gates, and kube-proxy behavior.

4. Design safe fallback

The control plane or kube-proxy should use cluster-wide endpoints when endpoints are insufficient, hints are invalid, or allocation is unsafe. Make fallback observable; otherwise cross-zone traffic may be misread as a successful optimization. Test a whole-zone outage, delayed EndpointSlice propagation, incorrect node labels, and mixed kube-proxy versions.

5. Prevent hot spots

Same-zone preference narrows the endpoint set. Track per-zone requests, connections, CPU, queue depth, and errors. If one zone receives a disproportionate source share or has too few endpoints, reduce the preference or temporarily use global routing. Scaling and rolling releases must preserve ready endpoints and PDB headroom in every zone.

6. Evaluate cost and latency

Record cross-zone bytes, request p50/p95/p99, connection reuse, endpoint load, and failure rate together. Compare the same traffic window before and after enablement; cache-hit or client-retry changes can otherwise be misattributed to topology routing. Cost reduction must not buy worse error rate or tail latency.

7. Roll out and roll back gradually

Enable the feature on a non-critical Service or one-zone canary, verify EndpointSlice hints, kube-proxy choices, and fallback events, then expand to similar services. Rollback removes the preference and verifies global endpoints; retain configuration versions, metric windows, and failure-drill evidence.

Example of a strong answer

I treat topology preference as a reversible performance optimization. First validate endpoint counts and source distribution, then let the EndpointSlice controller create hints for the node data plane. Insufficient endpoints, zone imbalance, or incompatible components trigger global fallback. Monitor cross-zone bytes, per-zone load, latency, errors, and fallback count; canary before expansion. Any overloaded or failed zone can recover availability by disabling the preference.

Common mistakes

• Assuming hints guarantee strict same-zone routing.
• Ignoring endpoint count and balanced-source prerequisites.
• Watching cross-zone cost while missing endpoint overload and tail latency.
• Treating legacy annotations, version fields, and feature gates as one universal API.
• Having no global fallback or failure drill when hints disappear.
• Letting a rollout or scale-down drop a zone below its safe ready-endpoint margin.

Follow-up questions and answers

Why fall back when endpoints are scarce?

Topology preference narrows the candidate set; too few endpoints increase overload and failure risk. Global endpoints preserve availability first.

Is three endpoints a hard rule?

It is an applicability recommendation in the Kubernetes documentation to improve zone allocation, not an absolute guarantee for every workload. Validate it against traffic, capacity, and failure objectives.

What if all requests originate in one zone?

Same-zone preference can concentrate load there. Reduce the preference, add endpoints in that zone, or fall back globally, then confirm with load and latency metrics.

How do you prove kube-proxy consumed the hints?

Inspect EndpointSlice hints, node component versions, and actual request distribution. Use cross-zone bytes and per-zone endpoint hit rate for end-to-end evidence rather than checking only control-plane objects.

Does rollback require rebuilding the Service?

Usually remove or adjust the topology preference and verify that EndpointSlice and node data-plane behavior return to global selection. Keep a rollback drill and metric evidence.