Prompt and context
An HA Kubernetes cluster is rolling its control plane from v1.35 to v1.36. During the window, different kube-apiserver instances may know different resource versions, so a client routed to an older server can receive a misleading 404 for a resource that exists on a newer peer. Design and explain how Mixed Version Proxy should route requests, constrain trust, provide observability, and support rollback. Distinguish Kubernetes v1.36 Beta behavior from the normal version-skew policy.
What the interviewer evaluates
- Separating version skew, discovery caches, and API visibility into distinct failure domains.
- Knowing that the proxy routes resource requests; it does not replace API conversion or storage migration.
- Explaining peer discovery, forwarding, identity propagation, and loop prevention.
- Covering upgrade safety, auditability, timeouts, retries, and rollback.
Clarifying questions to ask
- Is this a rolling replacement inside one cluster or a cross-cluster migration? Assume one cluster with several peer API servers.
- Are clients using built-in Kubernetes resources only, or also aggregated API servers and CRDs? Confirm the proxy's coverage for unknown resource versions.
- Are a shared load balancer, mTLS, centralized audit, and metrics already available? They determine identity propagation and diagnosis.
- What are the budgets for extra latency, unavailability, and rollback time?
30-second answer framework
Start with the failure: during a rolling upgrade, an old API server can mistake a new resource version for a missing resource. Then give the path: the entry API server performs discovery and a local decision; when the version is unknown locally, it forwards to a compatible peer, returning the peer response, status, and audit identity to the caller. Close with boundaries: follow the version-skew policy, forbid proxy loops, constrain proxy identity, return explicit forwarding failures, avoid retries that hide control-plane faults, and roll back through metrics and a staged feature switch.
Step-by-step deep dive
1. Classify the request and decide whether to forward
The API server parses group, version, resource, and verb. If the resource is registered locally, it follows local authentication, authorization, admission, and storage. If the requested version is unknown in local discovery, the server checks which peers advertise that resource version. It forwards only after a peer explicitly confirms support; otherwise it returns a distinguishable error. The proxy must not guess a version or turn every ordinary business 404 into a forwarding trigger.
2. Discover peers, preserve identity, and stop loops
Peers should come from a controlled control-plane membership list or a trusted internal discovery mechanism. Connections use the existing control-plane service identity and encrypted channel. The forwarded request carries the original user identity, groups, authorization headers, and audit context, while the proxy still enforces its own authorization boundary. Add a hop marker or equivalent internal metadata; a server that sees the marker must not forward again, preventing an A-to-B-to-A loop.
3. Handle consistency and failures
A read can observe briefly different discovery caches across API servers, so clients should rely on the server's resource version and resourceVersion. Forwarding occurs only when the target is reachable and version-compatible; timeouts, rejection, or a version-mismatched response fail fast with a recorded reason. Do not apply unbounded retries to writes, which could duplicate side effects. If retry is required, use an idempotency key or an explicit client retry policy.
4. Roll out, observe, and roll back
Upgrade one API server off-peak, then watch proxy hit rate, forwarding latency, unknown-version errors, 5xx responses, audit completeness, and discovery convergence before expanding the batch. Mixed Version Proxy is Beta and enabled by default in v1.36, but that does not remove the need to validate the distribution's version-skew matrix. If signals degrade, stop replacement, keep traffic on known-compatible peers, and, where supported, disable the feature gate and roll back the control plane. Preserve the forwarding chain and original identity in audit records.
Model answer
I would treat Mixed Version Proxy as a temporary control-plane compatibility layer, not as a new API gateway. The entry server authenticates, authorizes, and classifies the request. A resource it knows locally follows the local path; a resource version unknown locally is forwarded only when a peer explicitly advertises it. The connection uses the existing control-plane identity, carries the original user and audit context, and adds a single-hop marker to prevent loops. Peer status and response semantics are preserved. Timeouts fail fast, and writes do not receive unbounded automatic retries.
I would validate the path with a staged upgrade: record local hits, proxy hits, target version, latency, error class, and an audit correlation ID. Pause the batch when hit rate or 5xx exceeds a threshold. Beta-by-default in v1.36 is not a substitute for checking the distribution's skew policy or the coverage of aggregated APIs and CRDs. Rollback means stopping replacement, restoring the control-plane member set, disabling the feature gate when supported, and checking discovery caches so clients do not continue using stale capabilities.
Common mistakes
- Describing the proxy as an arbitrary API version converter instead of a route for unknown resource versions.
- Saying only “forward to a newer node” without peer discovery, identity propagation, or loop protection.
- Forwarding every 404, turning genuine missing resources into control-plane traffic.
- Retrying writes unconditionally and creating duplicate side effects.
- Watching only average latency while missing discovery convergence, audit integrity, or skew constraints.
Follow-up questions and responses
Follow-up 1: Can aggregated API servers or CRDs be proxied automatically?
Confirm whether that resource is inside the implemented Mixed Version Proxy scope. Aggregated API servers have independent discovery, authentication, and availability boundaries; support for built-in resources does not imply support for them. Give a capability matrix and an explicit fast-fail path for unsupported resources.
Follow-up 2: How do you prevent forwarding from amplifying a control-plane outage?
Use a single-hop marker, deadlines, concurrency limits, and a circuit breaker. Do not retry indefinitely at the source. Alert on proxy hit rate, target errors, and queue depth, and let the upgrade controller pause a batch when those signals cross thresholds.
Follow-up 3: What if the client's discovery cache is stale?
The client still follows Kubernetes discovery and version-skew rules. Server-side forwarding can reduce false 404s during a rolling upgrade; it cannot permanently cache new resources for clients or replace API version migration. Have the client rediscover and use an explicit API version when necessary.