Prompt and setting
The receiver verifies signed webhook requests while a provider changes the secret. Rotation must tolerate in-flight retries and multiple sender replicas without exposing the secret or creating a verification gap.
What the interviewer tests
- Preserving raw-body signature verification and constant-time comparison.
- Designing a bounded overlap between active and retiring secrets.
- Separating rotation state, replay protection, observability, and rollback.
Clarifying questions before answering
- Who controls rotation, and can the sender expose a key identifier or version?
- How long can delivery retries and clock skew last?
- Can the sender coordinate a dual-signing window, or must the receiver accept two secrets?
- What event ID, timestamp, and raw payload are available for replay checks?
30-second answer framework
I would provision a new secret, distribute it to every receiver, and enter a short overlap window. During overlap, verify a key-id-selected signature or try the current and retiring secret with constant-time checks, while enforcing a timestamp tolerance and event-ID deduplication. Metrics must show which version verifies, and rotation completes only after old-version traffic stays at zero for the retry horizon. Rollback keeps the retiring secret until the window closes.
Step-by-step deep dive
1. Preserve the signed bytes
Read the request body once as raw bytes before JSON parsing. Construct the signature input exactly as the provider specifies, including timestamp and delimiter rules. Compare MACs in constant time and reject malformed or oversized payloads early.
2. Model secret versions
Store active and retiring versions with creation time, expiry, provider scope, and a status such as PENDING, OVERLAP, or RETIRED. A key identifier is preferable because it avoids trial verification; if unavailable, bound the two-secret fallback and record which secret succeeded.
3. Roll out safely
Distribute the new secret through the secret manager, reload receivers atomically, and run a signed canary. Ask the sender to dual-sign or switch only after readiness is observed. Keep the old secret available for the maximum retry duration plus clock-skew margin.
4. Block replays
Require a signed timestamp inside a bounded tolerance and store an event ID or digest with a retention period covering retries. Verification must happen before enqueueing; duplicate valid deliveries can acknowledge success without repeating business effects.
5. Observe and retire
Count verification successes by key version, stale timestamps, duplicate IDs, malformed signatures, and queue outcomes without logging secrets or full sensitive payloads. Retire the old version only after the overlap and retry horizons pass; if the new version fails, restore the old one and alert.
High-quality sample answer
“I would stage a new version in the secret manager, reload every receiver, and verify a canary before switching the sender. For a bounded overlap, accept signatures from the current and retiring versions, preferably selected by a key ID, while checking a signed timestamp and deduplicating event IDs. I would track verification by version and wait through the sender’s retry horizon plus clock skew before retirement. If the new version fails, rollback keeps the old secret valid; logs contain counters and IDs, never secrets or raw payloads.”
Common mistakes
- Replace the secret everywhere at once → retries with the old signature fail → use an overlap window.
- Parse JSON before verification → canonical bytes may change → verify the raw body first.
- Accept either secret forever → stale credentials remain valid → set an expiry tied to retry and skew bounds.
- Log the signature or secret → observability becomes credential leakage → log versioned counters and safe identifiers.
Follow-up questions and responses
What if the sender cannot dual-sign?
Preload the new secret and accept both versions at the receiver for a bounded window. Coordinate the sender switch, monitor version-specific verification, and retain the old version through the maximum retry period.
How do you choose the overlap duration?
Use the sender’s documented retry ceiling, queue delays, clock-skew tolerance, and incident margin. Make the deadline explicit and alert on old-version traffic near expiry rather than guessing from average latency.
What if an attacker replays an old valid event?
Reject timestamps outside the tolerance and deduplicate event IDs or signed payload digests. Keep replay records at least as long as the accepted timestamp window and business retry horizon.