Prompt and scope
This is an identity-infrastructure and backend reliability question. Kubernetes ServiceAccounts use signed JWTs to access the API server or other systems that trust the identity. In v1.36, external ServiceAccount token signing is stable, allowing signing requests to leave the API server through a local Unix domain socket. The design should reduce private-key exposure while preserving TokenRequest, public-key verification, rotation, and high availability semantics.
What the interviewer evaluates
- Whether you separate signing, issuance, verification, and authorization instead of saying only “connect a KMS.”
- Whether you design UDS/gRPC connection behavior, deadlines, concurrency, retries, and fail-closed handling.
- Whether you handle overlapping keys, JWT
kid, caches, and verifier refresh. - Whether you rotate and roll back without revoking valid short-lived tokens prematurely.
- Whether you define audit, latency, error, and key-usage metrics.
Clarifying questions to ask
- What are token TTL, audiences, issuance QPS, and the number of API server replicas?
- Does the external signer provide HSM guarantees, health probes, versioned keys, and idempotency IDs?
- How do verifiers obtain JWKS, and what are refresh delays and cache TTLs?
- When the signer is unavailable, may new tokens be rejected, or is there a controlled old-key fallback?
- Do offline jobs or out-of-cluster systems depend on these JWTs during rotation?
A 30-second answer
“I would split the path into TokenRequest authorization, API-server calls to the external signer, public-key distribution, and RBAC authorization. The API server calls the signer over a protected Unix socket with a deadline, request ID, and bounded retries; the private key stays inside the KMS or HSM. During rotation, publish the new public key and let verifiers refresh with overlap, then issue with the new kid; retire the old key only after old token TTL expires. If the signer is unavailable, reject new issuance and alert rather than silently creating unprotected tokens. Measure signing latency, rejection rate, socket errors, key IDs, and JWKS age, and keep the old signer and key for rollback.”
Step-by-step deep dive
Step 1: Define trust and data flow
TokenRequest authorization determines who may request a token for which ServiceAccount and audience. After validating the request, the API server sends the JWT payload to the external signer; the signer returns a signature and the API server returns the token. The API server still owns issuer, audience, expiry, and RBAC semantics. The external system only performs controlled signing and must not grant extra permissions.
Step 2: Design the local signer interface
The documented configuration points --service-account-signing-endpoint at a Unix domain socket where a versioned signing protocol can run. Restrict socket permissions, directory, process identity, and SELinux or AppArmor policy to the API server. Include key version, algorithm, digest or signing bytes, request ID, and deadline in the request; return signature, kid, and an audit correlation ID. Never put private keys or complete tokens in ordinary logs.
Step 3: Handle deadlines, retries, and idempotency
Signing is a latency-sensitive path, so use a short deadline and bounded concurrency. A transient network or HSM failure may receive limited retries, but infinite retries would amplify the API-server queue. Request IDs correlate signer and API-server audits; if the protocol supports idempotent caching, duplicate accounting can be avoided. After timeout, return an explicit error and reject new issuance. Whether existing tokens continue to verify depends on public keys and expiry policy.
Step 4: Design key rotation and JWKS overlap
Create a new key and allow the signer to use its new kid, then publish the new public key in JWKS. After verifiers refresh, issue new tokens with that key. Keep the old public key for at least the longest token TTL plus cache and clock-skew windows. Do not remove it merely because the issuer has switched. Rotation must be pausable, auditable, and reversible.
Step 5: Preserve replicas and disaster recovery
Every API-server replica must reach a local or highly available signer with consistent key versions and clocks. For a centralized signer, assess cross-node network, failure domains, and blast radius. For per-node sidecars, ensure HSM connectivity and key distribution cannot diverge. Exercise signer restart, missing socket files, HSM throttling, unavailable JWKS, and API-server rolling upgrades.
Step 6: Verify, audit, and roll back
Enable the configuration on a canary API server and verify TokenRequest issuer, audience, kid, expiry, and RBAC behavior. Run integration tests across old and new keys, verifier types, and clock skew. Track signing p50/p95, failure reasons, socket latency, HSM counts, JWKS age, and token rejection rate. Roll back by restoring the old signer and key configuration while retaining the old public key; do not delete it until old tokens and the audit window have ended.
High-quality sample answer
“I would make the external signer a narrow signing boundary. The API server still validates TokenRequest, issuer, audience, expiry, and RBAC, calls the signer over a protected Unix socket, and keeps the private key only in the KMS or HSM. Requests carry version, kid, ID, deadline, and bounded retries; signer failure rejects new tokens and alerts. Rotation publishes new JWKS, waits for verifier refresh, switches issuance to the new kid, and retains the old key through the longest TTL, cache, and clock-skew window. Every replica must have consistent local-socket access, key versions, and time. The canary checks latency, rejection rate, key-ID distribution, JWKS age, and RBAC integration, while rollback preserves the old signer and public key.”
Common mistakes
- Letting the external signer decide RBAC or audience and expanding its responsibility.
- Copying the private key to a sidecar or logs, losing the security benefit.
- Deleting the old public key immediately after rotation and breaking still-valid tokens.
- Retrying signer timeouts without bounds and overwhelming the API-server queue.
- Testing only successful signing instead of JWKS caches, clock skew, and replica consistency.
- Treating silent old-key issuance during signer failure as a reliable fallback without audit boundaries.
Follow-up questions and answers
Can the API server keep using a local old private key if the signer is down?
Only if the explicit threat model, audit, and rollback plan allow it. The default should reject new issuance so secrets do not return to the API server. Existing tokens can continue verifying with the old public key until their TTL ends.
When is it safe to delete the old public key?
Compute a safety window from the longest token TTL, verifier JWKS cache TTL, maximum clock skew, and offline-consumer retention, then verify that old-kid usage is zero. Delete only after the window and retain a recoverable backup first.
Why use a Unix socket instead of HTTP?
A local socket narrows the listening surface and uses file permissions and host policy to control access. It does not solve protocol authentication, process isolation, or availability by itself; versioned APIs, audits, deadlines, and failure drills are still required.