Backend interview: How would you safely migrate Kubernetes Service externalIPs after v1.36 deprecation?

Question and scope

You own a bare-metal Kubernetes cluster where teams use Service.spec.externalIPs to route public addresses directly to Services. After upgrading to v1.36, deprecation warnings appear and security reviewers worry that a tenant can claim an arbitrary address and intercept traffic. Design a migration: discover real dependencies, choose a LoadBalancer controller or Gateway API per entry point, block new usage, and prove rollback.

Kubernetes documents that externalIPs are not allocated or ownership-checked by Kubernetes. In v1.36 the field is deprecated; later releases are expected to disable kube-proxy behavior and eventually remove it. Distinguish the current compatibility window from the long-term target; deprecation is not immediate deletion.

Context and boundaries

Focus on Service networking, RBAC, admission policy, migration ordering, and observability. Cloud load balancers, bare-metal network devices, and DNS are platform dependencies; state their contracts, address-ownership proof, dual-entry window, and failure fallback.

What the interviewer tests

• Whether you distinguish the Service externalIPs spec field from a Node ExternalIP address and a displayed LoadBalancer address.
• Whether you can trace an end-to-end migration through audit, traffic, permissions, controllers, and DNS.
• Whether DenyServiceExternalIPs can block new writes without deleting existing business traffic.
• Whether you compare the ownership and rollback boundaries of LoadBalancer controllers, MetalLB-like controllers, and Gateway API.
• Whether metrics and staged cutover prove there is no traffic black hole or IP hijack.

30-second answer

“I would inventory Service specs, audit logs, DNS, node routing, and real traffic, mapping each externalIP to a tenant, port, and backend. During migration I would enable DenyServiceExternalIPs to block only new values, leaving existing objects removable. For each entry point I would choose an administrator-controlled LoadBalancer controller or Gateway API, validate health checks, source addresses, and connection draining through a parallel path, then switch DNS or the upstream. After old traffic reaches zero, remove externalIPs, retain a rollback snapshot, and gate rollout with audit, 5xx, connection-failure, and address-conflict metrics.”

Step-by-step solution

1. Build the inventory. List every Service using externalIPs, its addresses, ports, namespace, owner, DNS TTL, protocol, EndpointSlices, external traffic policy, and recent traffic. Separate the field from Node .status.addresses entries named ExternalIP.

1. Model the security risk. A user writes externalIPs into Service spec while Kubernetes does not allocate the address or guarantee uniqueness. A low-trust tenant can claim another tenant’s address and intercept traffic. Audit creates, updates, deletes, and kube-proxy rules, correlating anomalies by address and tenant.

1. Block new use first. Enable DenyServiceExternalIPs. It rejects new Services using the field and new values added to existing Services, while removal of existing values remains possible. Test in audit or a low-risk cluster before enforcement; migrate existing objects separately.

1. Choose the replacement. In cloud environments prefer an administrator-controlled type: LoadBalancer. On bare metal use a controller with an address pool and conflict checks. When role separation or shared routing matters, use Gateway API: the platform owns the Gateway and applications manage constrained HTTPRoute objects.

kind: Gateway
apiVersion: gateway.networking.k8s.io/v1
spec:
  gatewayClassName: platform-public
  addresses:
  - type: IPAddress
    value: 192.0.2.4

This illustrates administrator-owned addressing only. Verify the GatewayClass, controller capabilities, certificates, and IP pool for the deployment before applying it.

1. Run parallel validation. Put the new entry point behind a separate hostname or low-TTL DNS and compare connection success, TLS, source address, long connections, health checks, and EndpointSlice convergence. If reusing an IP, record ownership in the new controller before switching upstreams so two implementations never announce it concurrently.

1. Drain and remove. Observe no new connections on the old entry point and wait for the maximum connection lifetime. Remove externalIPs while retaining audit evidence and a rollback snapshot. The order must preserve at least one verifiable path among DNS, load balancer, Service, and backend.

1. Plan versions and rollback. v1.36 emits deprecation warnings; the published timeline expects kube-proxy behavior to be disabled no earlier than v1.40 and full removal no earlier than v1.43. Rollback restores only an ownership-validated old entry point; it never reopens arbitrary tenant writes. If the replacement fails, temporarily revert DNS or controller configuration while continuing to block new risk.

Model answer

I would treat every externalIP as a migration asset, not as a field to replace blindly. API inventory, audit, DNS, node rules, and traffic logs establish address, tenant, port, connection lifetime, and ownership. The field is user-writable Service spec and Kubernetes does not allocate or conflict-check it, which creates the CVE-2020-8554 class of interception risk.

During migration I would enable DenyServiceExternalIPs to reject additions while preserving removal of existing values. The replacement depends on the entry point: a cloud LoadBalancer, a bare-metal controller with an administrator address pool, or Gateway API with platform-owned Gateways and application-owned constrained routes. A parallel hostname or low-TTL cutover compares connection, TLS, source-address, and health-check behavior before draining old connections and removing the field.

The plan records v1.36 warnings, the earliest expected v1.40 kube-proxy behavior change, and the earliest v1.43 full removal. Rollback restores only a validated administrator-owned entry point. I would track address conflicts, 5xx, connection failures, DNS propagation delay, and remaining Services.

Common mistakes

• Mistake: Treating deprecation as immediate deletion → Why it fails: the compatibility window is misread and causes unplanned downtime → Fix: plan warnings, behavior disablement, and removal as separate milestones.
• Mistake: Replacing externalIPs directly with loadBalancerIP → Why it fails: address pools and conflict checks may still be bypassed → Fix: let an administrator-controlled controller allocate and publish status.
• Mistake: Deleting every old field when admission enforcement starts → Why it fails: there is no traffic or connection-drain evidence → Fix: block additions, canary the replacement, drain, then remove by asset.
• Mistake: Checking only Services, not DNS, node rules, and traffic → Why it fails: hidden entry points or black holes remain → Fix: maintain an end-to-end migration inventory and observation window.
• Mistake: Letting tenants own Gateway addresses → Why it fails: the replacement recreates the permission flaw → Fix: platform-owned Gateways and constrained application routes.

Follow-up questions and answers

Does DenyServiceExternalIPs affect existing Services?

It rejects new Services using the field and new values added to existing Services; removing existing values remains possible. Verify the behavior in the target version and observe rejection events. It is a guard, not an automatic migration tool.

Why not hand-fill a LoadBalancer IP on bare metal?

Manual assignment still lacks address pools, conflict detection, health state, and audit. A controller with an administrator-owned pool makes allocation, release, and uniqueness a platform responsibility; fixed addresses still need explicit approval.

How can Gateway API preserve application self-service?

The platform creates the Gateway and GatewayClass and restricts addresses, listeners, and cross-namespace references. Application teams submit HTTPRoutes subject to reference policies, policy objects, and audit controls.

How do you handle long-lived connections and WebSockets?

Measure connection lifetime before switching DNS or the entry point. Stop new connections on the old path but allow existing ones to drain, using dual-entry retry behavior where appropriate. Separate new-connection failures from natural closure in metrics.

When can the rollback switch be removed?

After old traffic is zero, every Service has removed the field, address-conflict checks pass, the DNS TTL window ends, and the new path meets its observation SLO. Then remove the snapshot while retaining audit evidence; a calendar date cannot replace evidence.

References

• Kubernetes v1.36 Service ExternalIPs deprecation (Kubernetes Blog)
• Service documentation (Kubernetes Documentation)
• Admission Control documentation (Kubernetes Documentation)
• Gateway API documentation (Kubernetes Documentation)

Interview checklist

Separate field semantics from the security risk, then give the order: inventory, block additions, replacement entry point, parallel validation, drain, removal, and rollback.

One-sentence takeaway

ExternalIPs migration replaces a user-writable entry point with an administrator-owned, auditable, verifiable address-allocation path.

Keep practicing

If a cluster uses Gateway API, MetalLB, and a cloud LoadBalancer together, design one entry-point catalog, address-ownership model, and cross-environment rollback protocol.