Prompt and scope
You operate hundreds of Kubernetes clusters. In v1.36, core components expose /statusz, returning human-readable text by default and a structured API through explicit content negotiation. Design collection, permissions, version compatibility, alerting, and fault isolation. Some clusters still run older releases, and control-plane endpoints cannot be public.
What the interviewer is testing
The interviewer wants you to separate health probes, component state, and business SLOs while designing authentication, rate limits, version parsing, and degradation for many clusters. A strong answer handles sensitive fields, alert storms, collector failures, and an unreachable control plane.
Questions to clarify first
- Do you need liveness, dependency state, or versioned diagnostic fields?
- Does a collector run inside every cluster, or does a central plane pull data?
- Which roles may read statusz, and can responses expose internal topology or versions?
- Is detection expected in seconds, or are minute-level capacity and trends enough?
30-second answer
“I would treat statusz as a diagnostic signal, not a business-health signal. An in-cluster collector accesses endpoints over the private network with least privilege, requests structured output when available, and keeps text for human diagnosis; older clusters use a compatibility probe. The central platform aggregates by cluster, component, and version with rate limits, caching, and alert deduplication. An unreachable control plane is a collection-path failure, not automatically an internal component failure.”
Step-by-step solution
1. Define signals and version contracts
Record component identity, version, response format, checks, and timestamps. Parse structured fields by version, preserving unknown fields without depending on them; use text for display and evidence. Keep statusz separate from /livez, /readyz, and business metrics.
2. Design the collection topology
Run a lightweight collector inside each cluster and access the API server, scheduler, and controller manager over local networking. The center receives redacted state events and aggregates, avoiding public control-plane endpoints. Give the collector a queue, backoff, and local cache.
3. Enforce authentication and authorization
Create a read-only collector identity with the smallest resource scope, limiting component endpoints and network paths. Audit reader, frequency, and response size; never reuse an administrator credential. Filter version or topology fields by tenant and operations role when necessary.
4. Control load and failure
Set per-component concurrency, timeouts, cache duration, and maximum response size. Back off when the control plane is busy or unreachable instead of amplifying the incident with retries. Model component-reported failure, HTTP failure, network unreachability, and collector backlog separately.
5. Aggregate and alert
Build time series by cluster, component, version, and failure type, using event fingerprints for deduplication. Require a continuous window and impact scope, such as several components failing or a concentration in one version. A single timeout is a low-priority diagnostic signal.
6. Canary and roll back
Enable structured collection on a small set of v1.36 clusters first. Compare API load, field stability, alert precision, and collection latency before expanding. If fields or load regress, disable the new parser, return to the compatibility probe, and retain raw responses and audit records.
Model answer
I would layer statusz as control-plane diagnostics beside liveness probes and business SLOs. Each cluster runs a least-privilege read-only collector over private networking; the center receives redacted state and aggregates, while older releases use a compatibility probe. Parse structured responses by version, tolerate unknown fields, and retain text for operators. Bound concurrency, timeout, cache, and response size, separating component failure, network failure, and collector backlog. Deduplicate alerts by fingerprint and window. Canary a few v1.36 clusters and disable only the new parser when needed.
Common mistakes
- Treating statusz as a business SLO → user experience is misclassified → layer it with business metrics.
- Centrally querying public endpoints → the attack surface grows → collect in-cluster and send state centrally.
- Reusing an administrator credential → reads are over-privileged → create a minimal read-only identity.
- Failing the whole pipeline on an unknown field → upgrades interrupt collection → parse leniently by version.
- Alerting on every timeout → alert storms → separate path failure from component failure and deduplicate.
Follow-up questions and responses
Why retain the text response?
Structured fields are for machine aggregation; text helps an operator read and preserve incident evidence quickly. They come from one endpoint but serve different purposes.
How do you avoid a false positive when the control plane is unreachable?
Model network unreachability, authentication failure, collector backlog, and component-reported failure separately. Escalate a component fault only with sufficient evidence.
How do you support older clusters?
Probe endpoint and version first, then choose structured parsing, text parsing, or a compatibility probe. Keep a capability matrix instead of assuming every cluster supports the Beta interface.
How do you prevent collection from slowing the API server?
Bound frequency and concurrency, use cache and backoff, cap response size, and automatically lower collection frequency when API-server load rises.