Backend Interview: How Do You Rotate JWT Signing Keys Without Causing Authentication Outages?

Prompt and Applicable Context

An authentication service signs access tokens with RS256. A token may be accepted for at most 15 minutes, with 1 minute of clock skew. Two hundred backend services read a fixed jwks_uri from the issuer's OpenID Discovery document, cache the JWKS for 10 minutes, and verify tokens locally.

Design signing-key rotation that does not produce a wave of 401 responses for valid requests. Cover these cases:

• old and new tokens arrive during a normal rotation;
• the token header contains a kid absent from the local cache;
• many random kid values try to create a JWKS refresh storm;
• the JWKS endpoint times out or fails during rotation;
• the current private key may be compromised and requires emergency action;
• the operator must prove every verifier accepts the new key before retiring the old one.

The 15-minute lifetime, 10-minute cache, 1-minute skew, and 200 services are scenario constraints, not universal recommendations. The core skill is backend authentication protocol design, cache consistency, failure semantics, and security operations, so the category is backend.

What the Interviewer Evaluates

First, can the candidate state the safe order: publish the new public key, let verifiers observe it, begin signing with the new private key, wait for old tokens to expire, and only then remove the old public key? Switching the signer first guarantees that services with an old JWKS cache will reject new tokens.

Second, do they understand that JWKS is a public-key set? RFC 7517 defines a keys array, and kid only selects a key from a trusted set. A private key must never be published in JWKS. A kid is not proof of trust either; it must be bound to the trusted issuer, a pinned algorithm, and successful signature verification.

Third, can they balance caching and security? Fetching JWKS on every request overloads the issuer, while caching forever delays acceptance of a new key and retirement of an old one. An unknown kid may trigger one controlled refresh, but concurrent refreshes must be coalesced, rate-limited, and followed by a short negative cache when the identifier is confirmed absent.

Fourth, do they separate planned rotation from private-key compromise? Planned rotation overlaps old and new public keys for availability. During compromise, keeping the old key trusted lets an attacker mint tokens. The emergency path requires explicit cache invalidation, revocation controls, and a stronger security priority.

Fifth, do they extend signature checking into complete validation? A verifier pins allowed algorithms, then validates the signature, iss, aud, exp, and nbf. It does not follow an untrusted jku or arbitrary key URL from the token, which could enable algorithm confusion or SSRF.

Questions to Clarify Before Answering

• What is the maximum token acceptance time? Use the lifetime of every old token that could still be accepted, not only a nominal configured TTL, and include clock skew.
• What is the maximum JWKS staleness? Browsers, CDNs, proxies, and in-process caches can each add a layer; the rotation needs the real upper bound.
• Can all verifiers be refreshed proactively? Configuration push, version acknowledgements, or canary probes can replace waiting only for natural expiry.
• What happens when JWKS is unavailable? Define whether a known key can use a bounded stale cache and whether an unknown key fails closed.
• Must already-issued tokens be revoked immediately? A self-contained JWT cannot support precise per-token revocation through rotation alone; a denylist, token version, or introspection may be required.
• Who controls issuer and verifiers? One organization can collect refresh acknowledgements; third-party verifiers usually depend on a documented compatibility window.
• Where are private keys held? Generate and use them in a KMS, HSM, or restricted signing service; only public material belongs on the publication plane.
• What does success mean? Planned rotation keeps valid old and new tokens working. Emergency rotation may accept controlled reauthentication to stop trusting a compromised key quickly.

30-Second Answer Framework

"I split rotation into publish, warm, switch, overlap, and retire. I generate a key with a fresh kid, then publish both the new and old public keys in JWKS. I wait for the maximum 10-minute cache window, or proactively refresh all 200 verifiers and collect their new JWKS version. Only after the new public key is verifiable does the issuer begin signing with the new private key.

I keep the old public key until 15 minutes after the last old token was issued, plus 1 minute of clock skew. Verifiers select kid only within a trusted issuer's set, pin RS256, and validate the signature, iss, aud, exp, and nbf. An unknown kid triggers one coalesced, rate-limited refresh; if it is still absent, reject. During a JWKS outage, a known key may use an explicitly bounded stale cache, while an unknown key fails closed.

If the private key is compromised, I stop old-key issuance, publish and switch to a new key, broadcast cache invalidation, and revoke tokens signed by the old key. I do not use the normal overlap window. I prove completion through results by kid, cache age, JWKS fetch volume, and the last observation of the old kid."

Step-by-Step Deep Dive

Step 1: Pin the trust root and verification invariants

A verifier starts from a configured trusted issuer, reads its OpenID Discovery document, and uses the HTTPS jwks_uri from that document. The token header's kid only selects a candidate public key from this trusted JWKS. The token cannot redirect the verifier through jku, and the application must not concatenate kid directly into file, database, or URL lookups.

Every verification keeps these invariants:

1. accept only the configured RS256; the token cannot negotiate its algorithm;
2. require kid to match one signing key uniquely in the trusted issuer's JWKS;
3. after signature verification, require the expected iss, this API's aud, exp, and nbf;
4. reject the entire token if any check fails;
5. publish only public key material while private keys remain inside the controlled signing boundary.

A minimal key set during rotation can look like this:

{
  "keys": [
    { "kty": "RSA", "use": "sig", "alg": "RS256", "kid": "2026-07-a", "n": "...", "e": "AQAB" },
    { "kty": "RSA", "use": "sig", "alg": "RS256", "kid": "2026-07-b", "n": "...", "e": "AQAB" }
  ]
}

Array order is not preference; the verifier selects an exact kid. A new generation gets a new, never-reused kid, because a cache cannot distinguish changed key material hidden behind the same identifier.

Step 2: Execute planned rotation with publish before sign

Model planned rotation as explicit states:

GENERATED
  -> PUBLISHED(old + new)
  -> VERIFIER_READY
  -> SIGNING_WITH_NEW
  -> OLD_TOKEN_DRAINED
  -> OLD_KEY_RETIRED

The protocol is:

1. generate 2026-07-b in the controlled key system, but do not sign with it yet;
2. publish a JWKS containing 2026-07-a and 2026-07-b, with a new ETag or set version;
3. wait for the 10-minute maximum cache-staleness window, or refresh all 200 verifiers and collect acknowledgements that they recognize the new kid;
4. verify signature, issuer, audience, and time claims with canary tokens in each environment;
5. only after verifiers accept the new public key, begin signing with 2026-07-b;
6. record T_last_old, the issuance time of the final old-key token;
7. no earlier than T_last_old + 15 minutes + 1 minute, and after old-kid traffic matches expectations, remove the old public key from JWKS;
8. continue watching for anomalous old-kid traffic, then disable and destroy the old private material.

"Publish before sign" protects new tokens. "Stop old signing, drain old tokens, then remove" protects old tokens. Cache propagation controls the earliest signing switch; old-token acceptance lifetime and clock skew control the earliest public-key removal.

Step 3: Handle an unknown kid with one controlled refresh

An unknown kid may be a legitimate new generation or attacker-supplied noise. The verifier cannot reject every first sighting immediately, and it cannot turn each sighting into unlimited upstream traffic. A suitable flow is:

verify(token):
  header = parse_bounded_header(token)
  require header.alg == "RS256"
  key = trusted_cache.find(header.kid)
  if key is missing:
    refresh trusted_issuer_jwks once through single-flight
    key = trusted_cache.find(header.kid)
  if key is missing:
    short_negative_cache.add(header.kid)
    reject "unknown kid"
  verify signature and require iss, aud, exp, nbf

Concurrent misses for one issuer share a single-flight refresh. Refreshes have a global cooldown and timeout. After a fresh set confirms the kid is absent, a short negative cache prevents repeated random identifiers from reaching the issuer. The negative TTL cannot be long enough to block a real rotation, and the verifier must bound kid length and format to prevent high-cardinality memory attacks.

The refresh accesses only the configured jwks_uri, over TLS, with response-size, connection, and read limits. It may use ETag conditional requests. Attacker-provided jku, x5u, or similar locations never replace the trust root.

Step 4: Define the availability boundary during JWKS failure

Normal requests use the local cache; the JWKS endpoint does not sit in every authentication request's synchronous path. If a refresh fails:

• a matching known kid may continue within a predefined bounded stale window;
• an unknown kid must be rejected, never verified without a signature or against unrelated keys;
• after the bounded stale window expires, failure to refresh raises an alert and follows the security policy, normally rejection;
• the issuer must not switch to a new signing key when it cannot prove that verifiers have the new public key.

A short stale-if-error absorbs a control-plane blip but also extends local trust in a key removed from the current set. That duration belongs in the security model and must be overridden by explicit invalidation during compromise. Availability cannot depend on indefinitely stale keys.

Step 5: Use a separate emergency path for private-key compromise

Compromise changes the objective to stopping acceptance of attacker-minted tokens as quickly as possible. Stop old-key issuance, generate and publish a new public key, force verifiers to refresh, switch the signer, and mark the old key revoked. Do not preserve the normal 16-minute overlap merely for a seamless experience.

Deleting the old public key from central JWKS is insufficient because a verifier can retain its 10-minute cache. Use an exercised control-plane broadcast, cache-version push, service restart, or another invalidation channel. If third-party verifiers cannot be reached, the issuer is limited by their cache upper bound and must state that exposure explicitly.

Rotation also does not precisely recall already-issued self-contained tokens. For immediate revocation, apply a temporary rule by old kid or an iat cutoff, shorten access-token lifetime, or use introspection/session state for high-risk APIs. Emergency response may force users to authenticate again; that is an acceptable business impact when security takes priority.

Step 6: Prove completion with versions, metrics, and audit

The JWKS response should expose an observable set version or ETag. Verifiers report the current version, cache age, refresh outcome, and recognized kid values. The issuer records its active signing kid, without logging token bodies or private material.

Key metrics include verification outcomes by issuer, kid, and error; unknown-kid cardinality; JWKS fetch count, latency, and failure rate; single-flight coalescing; cache age; new-key canary success; and the last legitimate use of the old kid. A rise in unknown-key errors for the new kid after switching should halt or roll back the signing change before users report it.

Audit records answer who generated, published, activated, and retired each key; which JWKS version was current; which verifiers acknowledged readiness; and what last-old-issuance time justified retirement. Dual approval and least privilege make key-lifecycle operations safer.

Step 7: Test transitions and adversarial inputs

Do not test only one static valid token. Cover at least:

1. with only the old public key published, the old token passes and a new-key token fails;
2. during overlap, both token generations pass and repeated checks do not refetch JWKS;
3. with only the old key cached, a new kid passes after exactly one controlled refresh;
4. many identical or random unknown kid values cause bounded refreshes and are rejected;
5. JWKS timeout, 500, oversized response, and invalid JSON preserve the known-key policy while unknown keys fail;
6. wrong algorithm, iss, aud, expired token, and not-yet-valid token all fail;
7. an attacker-supplied jku does not make the verifier contact a different location;
8. after old-key removal, a fresh process rejects the old token, while an old cache accepts only within its stated window;
9. a compromise drill invalidates the old kid across every controlled verifier;
10. the signing gate prevents new-key activation while verifiers are not ready.

Acceptance checks both authorization results and JWKS request counts. A token that passes while every request hits the issuer is not a correct implementation. Neither is normal fetch volume with valid new tokens being rejected.

High-Quality Sample Answer

"I define kid as a selector within a trusted issuer's JWKS, not as a source of trust. Each verifier accepts only configured RS256, gets public keys from the fixed Discovery jwks_uri, selects by kid, and validates the signature, iss, aud, exp, and nbf. JWKS contains public keys only; the private keys remain in a KMS or HSM signing boundary.

For planned rotation, I generate a key with a new kid, publish both old and new public keys, and update the ETag. I then wait for the scenario's maximum 10-minute cache window or proactively refresh all 200 verifiers and collect readiness. Canary tokens prove that the new key works before the signer switches.

I record the final old-token issuance time. The old public key remains for at least the 15-minute acceptance lifetime plus 1 minute of clock skew, and I remove it only after old-kid traffic drains. Both transition gates are explicit: do not sign new tokens until the new public key has propagated, and do not remove the old public key until old tokens are invalid.

On a cache miss, each issuer gets one single-flight refresh with timeout, cooldown, and ETag. If the key remains absent, reject and negative-cache it briefly. Random kid values cannot amplify into an upstream flood. During a temporary JWKS failure, a known key may continue only inside a stated bounded stale window, while an unknown key always fails closed.

If the old private key is compromised, I stop old signing, publish and switch to the new key, broadcast cache invalidation, and revoke tokens in the old kid or old iat range. I do not retain the normal overlap because the attacker may be signing. Finally, I prove completion with JWKS versions, cache age, verification errors by kid, a new-key canary, and the last old-kid observation, and I regularly rehearse transitions and issuer failure."

Common Mistakes

• Switching the signer before publishing the public key → services with stale JWKS reject new tokens → publish, prove propagation, then begin new signing.
• Publishing only the new key → valid old tokens fail immediately → publish both generations during the acceptance window.
• Waiting an arbitrary duration → it may be shorter than real cache or token lifetime → derive gates from maximum staleness, last old issuance, token lifetime, and clock skew.
• Downloading JWKS for every request → an issuer failure breaks all authentication and amplifies load → use local caching, conditional refresh, and a bounded stale policy.
• Refreshing for every unknown kid → random identifiers create a refresh storm → use single-flight, rate limits, negative caching, and input bounds.
• Trying every key for an unknown kid → key selection becomes ambiguous and widens the attack surface → refresh once, then reject without an exact match.
• Trusting alg or jku from the token → algorithm confusion or SSRF may follow → pin algorithms and JWKS location in verifier configuration.
• Checking only the signature → a token for another issuer, audience, or time can be accepted → also validate iss, aud, exp, and nbf.
• Reusing kid with new key material → caches cannot tell that one identifier changed → assign a fresh identifier to every key generation.
• Treating central JWKS deletion as immediate revocation → verifiers may still hold the old cache → provide explicit cache invalidation and token revocation.
• Keeping normal overlap after compromise → an attacker can mint tokens throughout that window → use the emergency path and accept necessary reauthentication.

Follow-Up Questions and Responses

Follow-up 1: Exactly how long should the old public key remain?

Starting at the final old-key token's issuance time, keep it for at least "maximum acceptance lifetime + clock skew." In this scenario, that is 15 + 1 = 16 minutes. Add queue delay, offline issuance, or any longer implicit acceptance window if they exist. Observe old-kid traffic before removal so configuration and runtime reality agree.

Follow-up 2: Why not return 401 immediately for an unknown kid?

After a legitimate rotation, a new token can arrive before this process's natural cache refresh. One controlled refresh closes that gap. It must be coalesced, rate-limited, and restricted to the trusted URL; reject only when the latest set still lacks the kid, balancing compatibility with amplification resistance.

Follow-up 3: Should verification fail open when JWKS is down?

Never skip signature verification. A matching key already trusted in cache may complete full validation within an explicit bounded stale window. Reject unknown keys or requests after that window. This keeps a brief control-plane outage out of every data-plane request while preserving a measurable trust limit.

Follow-up 4: Why can old tokens work after deleting a compromised key?

Verifiers may still have a 10-minute old cache, and self-contained JWTs do not consult a central authority. The emergency path must invalidate caches proactively and apply a temporary rejection rule by old kid, issuance time, or token version. High-risk systems can use introspection or server-side sessions for faster revocation.

Follow-up 5: How do you avoid a refresh storm during rotation?

Prepublication lets most processes obtain the key during natural refresh, and proactive refreshes can be staggered with jitter. For real misses, use one single-flight per issuer, plus cooldown, ETag, timeout, and short negative caching. Monitor JWKS request volume and unknown-kid cardinality; rate-limit anomalies instead of retrying harder.

Follow-up 6: How can the signing gate be automated?

Assign a version to each JWKS. Verifiers periodically report the loaded version and recognized kid. The release controller requires a target readiness level and successful canary tokens on critical paths before activating the private key. Any rise in new-kid verification failures pauses or rolls back signing; leaving the new public key published does not harm old tokens.

Follow-up 7: What if third-party verifiers cannot acknowledge readiness?

Publish a stable overlap contract: expose the new public key at least one maximum cache period early, retain the old key until all old tokens expire, and send appropriate caching headers. The issuer cannot force third parties to refresh, so compatibility windows, change notice, and canary checks replace internal acknowledgements. Emergency compromise can still leave an unavoidable exposure window.