Prompt and context
In a multi-tenant Kubernetes cluster, a service must use mTLS to call an internal API. The application may read files, but it must not call the Kubernetes API or carry a long-lived private key in its image. Design the complete PodCertificateRequest path for requesting, signing, projecting, rotating, revoking, and auditing certificates.
What the interviewer evaluates
- Whether you separate PodCertificateRequest, the traditional CertificateSigningRequest, and signer responsibilities.
- Whether you can explain how kubelet-projected volumes isolate keys, certificates, and trust roots.
- Whether you handle signer authorization, tenant boundaries, rotation windows, Pod deletion, and node compromise.
- Whether you recognize that PodCertificateRequest is beta and disabled by default in v1.35, with explicit enablement required in v1.36.
Clarifying questions to ask
- Is the certificate for client authentication, server authentication, or mutual mTLS?
- Which SANs, lifetime, trust roots, and revocation semantics are required?
- Do the cluster version, feature gate, runtime configuration, and kubelet support Pod certificate projection?
- Can the application reopen files or watch directory changes before its certificate expires?
30-second answer framework
I would let kubelet request a certificate on behalf of the Pod. The signer would accept only an authorized signer and constrained identity fields, then deliver the key, certificate, and trust root through a read-only projected volume. The application would read files rather than call the API and reload them during rotation. Requests, issuance, projection, and refresh would be auditable; a failure would retain a still-valid old certificate or block new connections. Because PodCertificateRequest is beta and disabled by default in v1.35, and needs a feature gate and runtime configuration in v1.36, I would start with capability detection and a node-pool canary.
Step-by-step deep dive
1. Define identity and authorization boundaries
The Pod declares a purpose and target signer, but cannot choose an arbitrary CA. Admission policy binds namespace, service account, Pod labels, and allowed signers; the signer validates request origin, SAN templates, and key algorithms, rejecting cross-tenant subjects. API access remains with kubelet and control-plane components while the application receives file-read access only.
2. Design the request and projection path
Kubelet creates the Pod request and obtains the signed result. The private key is generated on the node so it never enters the image or a business API. Certificate, key, and trust bundle are mounted separately with least-privilege permissions. Projection uses atomic replacement so the application cannot observe a partially written file. Signer name, lifetime, and SAN are recorded as audit attributes.
3. Handle rotation, revocation, and failure
Refresh before expiry and cover kubelet restarts, network loss, and temporary CA outages. Keep the old certificate until the new chain validates; on load failure, reject new connections and expose a health signal. Remove projections after Pod deletion, service-account changes, or node recycling. Revocation depends on the CA contract, such as CRL, OCSP, or short lifetimes; Kubernetes does not provide one universal revocation behavior for every signer.
4. Plan versions and observability
Verify the v1.35 beta default-off state, the v1.36 feature gate and runtime configuration, and the signer-kubelet compatibility matrix before deployment. Monitor request latency, issuance failures, remaining lifetime, rotation success, permission errors, and unexpected SANs. Logs should contain request UID, namespace, Pod, signer, and outcome, never private keys or full certificate bodies. Keep a short-lived legacy projection path and rollback switch during the canary.
High-quality sample answer
I would treat a Pod certificate as a short-lived identity credential brokered by kubelet. Admission binds the namespace and service account to an allowed signer; the signer validates SANs, algorithms, lifetime, and tenant boundaries. The key is generated on the node, while certificate, key, and trust root are delivered as separate read-only projected files. The application never calls the API: it watches the directory or reopens files per request and atomically switches only after validating the new chain. Rotation starts early; a temporary outage keeps the still-valid old certificate, while expiry or identity change blocks new connections and raises an alert. Pod deletion removes the projection, and revocation follows the signer contract for short lifetimes or CRL/OCSP. Before rollout, I verify the v1.35 beta default-off state and v1.36 feature gate and runtime configuration, then canary by node pool while recording request UID, signer, latency, failure rate, and remaining lifetime.
Common mistakes
- Giving the application a Kubernetes API token that can create arbitrary CSRs.
- Letting requesters choose any signer, SAN, or cross-namespace subject.
- Putting private keys in images, ConfigMaps, or long-lived Secrets while ignoring node permissions.
- Reading the certificate only at startup and continuing to use an expired file descriptor after rotation.
- Treating the beta capability as enabled and stable on every cluster version.
- Logging full PEM data, private keys, or tenant-sensitive audit fields.
Follow-up questions and answers
Why not use a long-lived Secret?
A long-lived Secret increases the exposure window and is difficult to bind to Pod identity and deletion. Short-lived certificates with kubelet projection and rotation reduce credential lifetime, but the signer’s revocation and recovery contract must remain explicit.
How do you limit a compromised node?
Restrict node access to Pod directories and container users, use short lifetimes and minimal SANs, and place high-value identities behind an isolated or hardware-protected signer. Auditing and anomalous-connection detection reduce response time.
What if the feature gate is disabled?
Have the deployment controller detect the API resource, runtime configuration, and kubelet capability. Pause the workload or switch to an audited legacy path when prerequisites are absent; never silently create files that appear valid but will not rotate.