Prompt and context
A platform team must let kube-apiserver read node metrics without granting the same identity access to Kubelet debug or exec endpoints. The cluster is upgrading from coarse authorization behavior to Kubernetes v1.36, with no monitoring outage, no node-level privilege expansion, and auditable evidence. Explain the authentication path, authorization attributes, migration, rollback, and validation metrics.
What the interviewer evaluates
- Separating Kubelet HTTPS endpoint authentication from authorization instead of discussing only RBAC.
- Expressing least privilege with verb, resource, subresource, node, and namespace attributes.
- Understanding that the API Server's Kubelet client identity needs explicit authorization rules.
- Designing compatibility migration, fail-closed behavior, auditability, and rollback.
Clarifying questions to ask
- Which callers need the Kubelet API: only the API Server, monitoring agents, or operators connecting directly to nodes?
- Which capabilities are required: metrics, logs, running-container state, command execution, or port forwarding?
- Does Kubelet use client certificates or another authentication method, and is the API Server authorization callback available?
- Is a short read-only degradation acceptable during rollout, and what is the monitoring outage budget?
30-second answer framework
Split the path into three layers: TLS or another authenticator establishes caller identity, the Kubelet authorizer maps the request to attributes, and an authorization service makes the decision. Apply least-privilege rules to only the node endpoints that are needed, isolating debug, exec, and proxy capabilities. Migrate by inventorying calls, observing denials, tightening rules in batches, and switching to enforced denial only after metrics and a rollback switch are ready.
Step-by-step deep dive
1. Authentication, authorization, and request attributes
The Kubelet HTTPS endpoint first validates a client certificate or the configured authenticator and obtains a username and groups. Authorization maps the HTTP request to Kubernetes attributes such as verb, resource, subresource, namespace, name, and node. Authentication success does not imply authorization; every endpoint needs a decision. When the API Server calls Kubelet, the --kubelet-client-certificate and matching key identify a controlled principal that must have explicit permissions in the authorization system.
2. Express least privilege with subresources
Treat metrics, logs, container state, and debug execution as separate capabilities. Grant only the required read-only node resources or subresources and constrain the node scope. Do not grant wildcard resources or broad nodes/proxy access merely to make monitoring work. Give command execution, port forwarding, and debug endpoints separate high-risk roles that are not bound to the API Server's ordinary service identity.
3. Migration and compatibility
Build a call matrix from current audit and access logs: identity, path, verb, target node, result, and caller version. After enabling fine-grained authorization, run an observation phase that records would-be denials without breaking collection, fill only the necessary rules, and switch nodes in batches. KubeletFineGrainedAuthz is GA and enabled by default in Kubernetes v1.36, but the rollout must still verify distribution defaults, API Server client credentials, and every monitoring component's paths.
4. Observability, rollback, and defense in depth
Record audit events for both allows and denials with identity, node, resource, and reason. Monitor authorization latency, denial rate, metric collection success, and unexpected endpoint access. If the rollout breaks monitoring, first roll back client permissions or temporarily restore a compatibility rule while keeping high-risk endpoints closed. Network controls must still limit Kubelet port reachability; authorization does not replace TLS, network isolation, or node identity protection.
Model answer
I would verify the Kubelet endpoint authenticator first, then map each request to standard authorization attributes. The API Server uses a controlled client certificate, and the authorization system evaluates verb, resource, subresource, node, and namespace for least privilege. Metrics collection receives only the required read-only capability. exec, port forwarding, and debug endpoints use separate high-risk roles; wildcard permissions and broad nodes/proxy access are not attached to the ordinary API Server identity.
The migration has four stages: inventory the call matrix; observe denials without interrupting collection; add minimal rules in node and component batches; then enforce denial with a rollback switch. KubeletFineGrainedAuthz is GA and enabled by default in v1.36, but I would verify distribution settings, API Server credentials, and monitoring versions. Audit records retain identity, node, resource, result, and reason, while network policy continues to restrict Kubelet ports so authorization, transport, and network controls converge.
Common mistakes
- Configuring TLS authentication and assuming a client certificate grants every Kubelet API.
- Using wildcard resources or broad
nodes/proxypermissions to fix monitoring. - Enforcing denial without a call matrix, causing silent collection failures after upgrade.
- Reusing one high-privilege role for the API Server and operator debugging.
- Relying only on authorization rules while ignoring Kubelet port exposure and audit alerts.
Follow-up questions and responses
Follow-up 1: Why not grant the monitoring identity nodes/proxy directly?
nodes/proxy allows access to node endpoints through the API Server and can cover much more than metric reads. Confirm the real request paths and grant concrete resource or subresource permissions. If an implementation constraint requires proxy access, pair a dedicated identity with node scoping and audit alerts.
Follow-up 2: How do you prove the upgrade did not expand privilege?
Compare the call matrix and authorization decisions before and after the change, focusing on new verbs, resources, subresources, and node scope. Analyze denial deltas, then use synthetic requests to prove metrics remain readable while exec remains denied. Make those checks release gates.
Follow-up 3: What happens if the authorization service is unavailable?
Choose an explicit fail-closed policy so a callback failure cannot become an allow. Depending on the business budget, keep a short read-only cache or pause collection, but do not open high-risk endpoints to restore metrics. Alert and revalidate after the callback recovers.