Prompt and Scope
A platform team runs a shared Kubernetes cluster while application teams deploy gRPC services. The platform requires one entry point, TLS termination, routing by service/method/header, and canary releases; each application team may edit only its own Route and cannot control infrastructure or another namespace. Design the resources, request path, authorization boundary, observability, and failure rollback.
Assume a Gateway API controller is installed, backends speak HTTP/2 gRPC, and clients have explicit timeout and retry policies. Focus on cross-team control-plane boundaries rather than one vendor implementation.
What the Interviewer Evaluates
- Whether you separate GatewayClass, Gateway, GRPCRoute, and Service responsibilities.
- Whether you understand GRPCRoute matching granularity instead of producing generic Ingress YAML.
- Whether you handle cross-namespace ReferenceGrant, RBAC, default deny, and audit.
- Whether you explain weights, long-lived streams, and retry hazards.
- Whether you define status signals, rollback gates, and controller/data-plane failures.
Clarifying Questions Before Answering
- Is the Gateway centrally owned or team-owned? This changes Route attachment and approval.
- Is routing by service, method, header, or hostname? Matching precedence changes conflict handling.
- Are calls unary or long-lived streaming? Streaming usually should not be retried or cut over per request.
- Is canary selection by percentage, tenant, header, or client version? Define weight, stickiness, and rollback signals.
- Can a backend Service be cross-namespace or cross-cluster? That adds ReferenceGrant and health-check concerns.
30-Second Answer Framework
I separate the platform-owned Gateway from application-owned GRPCRoutes, with Services as the backend-discovery boundary. A Route declares gRPC method and metadata matches; cross-namespace references require explicit authorization. The controller exposes accepted and reference-resolution status, while the data plane routes by weight and avoids blind retries for streams. I start canarying with one tenant or an explicit header, use method errors, tail latency, and business health as rollback gates, and return to the stable backend when a Route or backend is invalid.
Step-by-Step Deep Dive
1. Resource responsibilities and data flow
The platform creates a GatewayClass selecting the controller, then listeners, addresses, and TLS policy. An application creates a GRPCRoute in its namespace, attaches it to an allowed Gateway with parentRefs, and targets its Service. The controller turns accepted rules into data-plane configuration, matching hostname, gRPC service, method, and headers before selecting backendRefs.
2. Matching and conflict handling
Define precedence between exact and fallback matches: a full service/method match can beat service-only, which can beat a header fallback. Do not allow two teams to implicitly overwrite the same parent and match. Surface conflicts through conditions such as Accepted and ResolvedRefs, and make CI check them. For an unknown method, choose explicit UNIMPLEMENTED or a stable fallback; never silently route to an arbitrary service.
3. Multi-tenant authorization
An application can write its Route and Service only. allowedRoutes limits namespaces that may attach to a Gateway; a cross-namespace backend reference requires a ReferenceGrant approved by the destination namespace owner. RBAC, admission policy, and Git review prevent application teams from changing platform TLS, listeners, or another team’s grant. Audit who changed a parent, backend reference, or weight.
4. Canary, streams, and retries
Unary RPCs can send new requests to stable and canary by weight; a streaming RPC keeps its route after connection establishment. Do not automatically retry non-idempotent calls, and do not stack Gateway retries on an unclear client timeout. Use an explicit header or tenant list as the canary key so one tenant does not randomly drift during debugging. Change weights in small commits and record activation time.
5. Health, status, and failure rollback
The data plane needs gRPC health checks, connection pools, and an explicit per-route timeout. If the controller is unavailable, keep serving the last accepted configuration but reject unverified changes. If a Gateway is not accepted, a reference cannot resolve, or a backend has no endpoints, expose status that blocks the release. Rollback can set the canary weight to zero, restore the previous Route, or switch to a standby Gateway; make it idempotent.
6. Observability and capacity boundaries
Record requests, errors, P50/P95/P99 latency, active streams, retries, and connection time by route, service, method, status, tenant, and version. Do not put raw high-cardinality metadata into metric labels; sample and redact it in logs. Capacity tests must cover connections, concurrent streams, TLS CPU, controller propagation delay, and backend connection limits for both unary and streaming traffic.
High-Quality Sample Answer
I give the platform team ownership of GatewayClass, Gateway, and listeners, while application teams own only namespace-scoped GRPCRoutes and Services. A GRPCRoute matches service/method and controlled headers; a cross-namespace backend needs a destination-owned ReferenceGrant, and allowedRoutes plus RBAC constrain attachment.
For release, I weight new unary requests between stable and canary. Streaming connections choose once at establishment, so I do not cut them over mid-stream. I avoid automatic retries for non-idempotent calls and coordinate timeout and retry budgets between client and Gateway. The controller exposes Accepted, ResolvedRefs, and backend health; CI blocks invalid references or conflicts. Canary starts with one tenant or explicit header and rolls back on method-level errors, tail latency, active streams, or business success rate, restoring the previous weight with an audit record.
Common Mistakes
Treating GRPCRoute as ordinary Ingress
Ignoring service/method and HTTP/2 semantics makes matches too broad. Define precedence and unknown-method behavior first.
Letting applications edit the Gateway
Shared listeners and TLS become mutable by tenants. Use allowedRoutes, RBAC, ReferenceGrant, and admission policy as separate gates.
Blindly retrying or cutting over streams
This can duplicate side effects or truncate long connections. Separate unary and streaming policy by idempotency and connection lifetime.
Checking only controller deployment success
A healthy controller does not prove an accepted Route or usable backend. Check Accepted, ResolvedRefs, health, and data-plane signals.
Follow-Ups and Responses
Follow-up 1: Two Routes match the same method. What happens?
Do not depend on an implementation’s incidental order. Constrain ownership with namespace and parent-attachment policy, check conflicts in CI, and treat unresolved status as a release failure.
Follow-up 2: How can canary target one tenant safely?
Use an explicit tenant or header match instead of random weights, and record version and tenant error rates. Rollback first removes that match.
Follow-up 3: Does a dead Gateway controller interrupt traffic?
It depends on whether the data plane retains the last accepted configuration. Continue serving it while freezing changes, monitor configuration age, and fail over to a standby before the safety window expires.
Follow-up 4: Why require authorization for a cross-namespace backendRef?
It lets one team send traffic to another team’s Service. A destination-owned ReferenceGrant makes consent explicit, while RBAC and audit prevent hidden privilege escalation.