Backend Interview: How Do You Securely Receive and Process Webhooks?

Problem and Use Cases

The receiver converts an untrusted HTTP request into a durable internal event. The hard part is the boundary between those states. A fast 200 OK is wrong if the process can crash before saving the event. Running the payment workflow before responding is also wrong because a slow dependency causes provider retries and amplifies load.

Use these interview assumptions:

• Peak traffic is 2,000 requests per second. The average raw body is 10 KiB and the maximum is 1 MiB, so average-body ingress at peak is about 19.5 MiB/s before headers, replication, and storage overhead.
• The provider expects a response within 2 seconds. Our internal target is a 500 ms p99 acknowledgment to preserve headroom.
• Delivery is at least once and unordered. A provider may send the same logical event concurrently, retry it later, or deliver a newer object state first.
• An accepted event must survive a receiver crash and eventually reach a terminal PROCESSED or FAILED state. One logical event must not apply the same business mutation twice.
• Signing secrets rotate without downtime. Raw payloads are encrypted and retained for 30 days for recovery and audit; idempotency records remain for at least the provider's documented redelivery window.

The main use cases are payment-state updates, subscription lifecycle changes, refunds, disputes, and account notifications. The design must work across providers without pretending their headers, signature algorithms, retry identities, or timestamp rules are identical.

What the Interviewer Is Evaluating

First, the interviewer wants a precise acknowledgment contract. 2xx means “this receiver durably accepted the event,” not “every downstream side effect completed.” Returning success before the durable write creates silent loss. Returning success for an already accepted duplicate is correct because the provider can stop retrying.

Second, they want security in the correct order. The receiver limits method, content type, headers, and body size; preserves the exact raw bytes; verifies the provider-specific signature with trusted secret versions; compares MACs in constant time; and checks signed freshness metadata when the provider supplies it. Parsing and reserializing JSON before verification can change whitespace or key order and invalidate a legitimate signature.

Third, they want the candidate to separate replay prevention from retry deduplication. A signed timestamp rejects an old captured request. A stable provider event or delivery ID prevents a valid retry from applying twice. Some providers generate a fresh attempt timestamp and signature for every retry while retaining the logical event ID. One mechanism cannot safely replace the other.

Fourth, they want a durable processing model with no database-and-queue dual-write hole. A database inbox row and an outbox row can be committed together; a relay then publishes work. Alternatively, workers can lease inbox rows directly. The unique key is enforced by storage, not by a read-then-write check vulnerable to concurrent duplicates.

Finally, a strong answer handles out-of-order state, external side effects, secret rotation, poisoned events, backpressure, observability, reconciliation, and crashes at every transaction boundary.

Clarifying Questions Before Answering

• What exactly does 2xx promise? Here it means the signature passed and the event, or its previously accepted duplicate, is durable. It does not promise that email, ledger, or provider API calls have finished.
• What provider identity is stable across retries? Each adapter must document the logical event ID, attempt timestamp, signature format, and whether manual redelivery preserves the same ID. Never derive identity from the payload hash alone.
• Does the provider sign the raw body and metadata? The adapter defines the canonical signed bytes. The HTTP framework must expose the untouched body before JSON middleware runs.
• What ordering information exists? Prefer an authoritative object version or sequence. An event creation time is useful evidence but is not automatically a strict order. When no version exists, fetch current provider state for state-setting events.
• How long can redelivery occur? The idempotency retention and old-secret overlap must cover the provider's documented behavior and the product's manual replay policy. The 30-day raw-event retention in this problem is a product assumption, not a universal vendor rule.
• Which failures should cause a retry? If authenticity cannot be established or durable storage is unavailable, do not acknowledge. After durable acceptance, worker outages should not change the HTTP response.
• What data is sensitive? Encrypt raw bodies, restrict access, redact logs, and define deletion exceptions. A signature verifies authenticity and integrity; it does not encrypt the payload.

30-Second Answer Framework

“I would expose a provider-specific HTTPS endpoint behind body-size and rate limits, capture the exact raw bytes, and verify the signed ID, timestamp, and payload with the current or previous secret using constant-time comparison. Within one database transaction, I would insert an inbox row under a unique provider-event key and an outbox row, then return 2xx; an already accepted duplicate also gets 2xx. A relay and workers process asynchronously with leases and retries. The business mutation and processed marker commit together, while external effects use an outbox and stable idempotency key. For unordered delivery I use object versions or fetch authoritative current state, never arrival order. I would monitor acknowledgment latency, rejection reasons, inbox age, duplicates, and failures, then test concurrent duplicates, secret overlap, stale signatures, out-of-order events, and crashes around every commit.”

Step-by-Step Deep Dive

Give every provider an adapter, but keep one receiver pipeline. The adapter supplies allowed event types, maximum body size, header parsing, canonical signed bytes, algorithms, trusted secret versions, freshness policy, and extraction of the logical event ID. Secrets come from a managed secret store and are cached only for a bounded period. A request must never choose its own verification key through an untrusted header.

The ingress sequence is deliberate:

1. Require HTTPS POST; apply endpoint and provider rate limits.
2. Validate bounded headers and Content-Length when present.
3. Read at most 1 MiB into raw bytes; reject overflow while streaming.
4. Parse signature metadata without parsing the JSON body.
5. Verify current and previous trusted secret versions in constant time.
6. Check the signed attempt timestamp against the provider-specific tolerance.
7. Parse the verified body and validate the event envelope and allowed type.
8. Durably accept under a unique logical-event key, then acknowledge.

Timestamp freshness and deduplication solve different attacks. Suppose an attacker captures a valid signed request. A tight signed timestamp window blocks replay after the window, but the same captured request may still arrive twice inside it. Conversely, an event-ID record blocks a duplicate logical event but cannot prove that an unsigned timestamp is fresh. Providers also differ: retries can carry a new signed attempt timestamp while retaining the same event ID. Preserve both checks and make their exact semantics adapter-owned.

Use an inbox as the source of truth:

WebhookInbox(
  inbox_id, provider, endpoint_id, provider_event_id,
  event_type, object_id, object_version, provider_created_at,
  received_at, raw_payload_ref, payload_hash, matched_secret_version,
  status, attempt_count, next_attempt_at, lease_until, last_error
)

WebhookOutbox(outbox_id, inbox_id, topic, created_at, published_at)

UNIQUE(provider, endpoint_id, provider_event_id)

Inside one database transaction, insert the inbox row and its outbox notification. If the unique key already exists, read its acceptance state and return 2xx without creating more work. This is an atomic insert, not “query then insert.” Commit before acknowledging. If the database is unavailable or the commit outcome is unknown, return a retryable non-2xx; a later duplicate will converge on the unique row if the first commit actually succeeded.

The database-plus-outbox design closes the gap between saving and queueing. A relay repeatedly publishes unpublished outbox rows and marks them published. Publication may happen twice, so queue consumers still deduplicate by inbox_id. A simpler implementation can skip the broker and let workers claim due inbox rows with leases, such as FOR UPDATE SKIP LOCKED. Choose based on throughput and operational needs, but keep the inbox as the durable acceptance and audit boundary.

Workers claim a short lease, parse the versioned event, and route only supported types. For a mutation in the same database, update the business row, record the processed event, and mark the inbox PROCESSED in one transaction. The processed-event table has the same stable provider key, so a worker retry becomes a no-op. For calls to another service, write a local outbox entry with inbox_id as its idempotency key. Exactly-once delivery across arbitrary networks remains impossible; stable identity and idempotent receivers make at-least-once execution safe.

Arrival order cannot define business order. If events carry an authoritative object version, update with a condition such as incoming_version > stored_version; stale events become processed no-ops. If only state-setting notifications exist, fetch the provider's current resource and converge local state. If the event represents a non-repeatable delta, require a sequence, buffer a bounded gap, and reconcile missing versions. A timestamp alone may tie, skew, or describe creation rather than commit order.

Failures divide at the durable boundary. Before acceptance, invalid signatures, stale signed timestamps, oversized bodies, unavailable secrets, and unavailable storage produce rejection or a retryable response according to policy. Do not persist unverified sensitive bodies merely to debug them. After acceptance, queue or worker outages still get 2xx; the inbox records accumulate and recovery drains them. Transient worker failures back off with jitter. Schema errors and exhausted retries enter FAILED, preserve a redacted diagnostic, and trigger an operator-visible recovery path.

Secret rotation keeps current and previous versions trusted for a bounded overlap derived from provider redelivery behavior. Record which version matched, but never log the secret or signature. New secrets are configured at both ends, observed in production, and old versions are retired explicitly. Emergency compromise may require immediate retirement and replay from the provider, so the rotation runbook must distinguish planned overlap from incident response.

At 2,000 requests/s and 10 KiB average, the ingress sees about 19.5 MiB/s of raw bodies. Capacity planning includes TLS and HMAC CPU, database transaction rate, replication, queue amplification, and burst duration. Scale stateless ingress horizontally, partition inbox indexes by provider and time if required, keep the unique key globally enforceable within its ownership boundary, and put raw encrypted bodies in object storage when database rows would become too large.

Measure accepted, duplicate, invalid-signature, stale, oversized, and unsupported-event rates separately. Track acknowledgment p50/p95/p99, database commit latency, oldest unprocessed inbox age, worker success and retry rates, FAILED counts, outbox relay lag, and matched secret versions. Alerts should use lag and durable state, not queue depth alone. A reconciliation job compares accepted inbox rows with processed records and outbox publication, then re-enqueues safe missing work.

Test from the raw HTTP bytes inward. Use provider-published signature vectors, then change one byte, whitespace, ID, or timestamp. Test missing and duplicate headers, body-size boundaries, clock skew, current/previous secrets, and retirement. Send hundreds of concurrent copies of one event and prove one inbox row and one business mutation. Crash after inbox commit but before response, after queue publish but before marking the outbox, and after business commit but before worker acknowledgment. Deliver versions 3, 1, and 2; saturate workers; restore them; and prove eventual convergence with bounded acknowledgment latency.

High-Quality Sample Answer

“I define 2xx as durable acceptance. The ingress is stateless and provider-specific only at the adapter boundary. It accepts HTTPS POST, limits the body to 1 MiB, preserves the exact raw bytes, and verifies the provider's canonical signed ID, attempt timestamp, and payload against trusted current or previous secrets. HMAC comparisons are constant time. I then parse the verified envelope and allow only supported event types.

In one database transaction I insert WebhookInbox under UNIQUE(provider, endpoint_id, provider_event_id) and insert an outbox row. I acknowledge only after commit. A concurrent duplicate conflicts on that key and also receives 2xx without new work. At 2,000 requests per second and 10 KiB average, raw ingress is about 19.5 MiB/s, so I scale ingress horizontally and size signature CPU, database commits, replication, and burst storage rather than counting requests alone.

An outbox relay publishes inbox_id; duplicate publication is safe. Workers lease the inbox record. The business mutation, processed-event marker, and inbox completion share one transaction when possible. Remote side effects use another outbox and inbox_id as an idempotency key. This avoids claiming exactly once across a network while ensuring retries do not repeat the logical effect.

I do not use arrival order. An authoritative object version gates updates; otherwise state-setting webhooks trigger a read of current provider state. Missing sequence gaps enter reconciliation. Before durable acceptance, an unverifiable request or storage outage is not acknowledged. After acceptance, a worker outage is absorbed by the inbox and still receives 2xx.

I rotate secrets with a bounded current/previous overlap and record the matched version. I monitor acknowledgment latency, rejection classes, duplicates, inbox age, outbox lag, failures, and secret-version usage. Finally, I test official signature vectors, byte mutations, stale timestamps, secret overlap, concurrent duplicates, out-of-order versions, and crashes before and after every commit. The pass condition is one durable row and one business mutation per logical event, no acknowledged loss, and eventual convergence after recovery.”

Common Mistakes

• Parse JSON before verification → reserialization changes the signed bytes → capture and verify the exact raw body first.
• Return 200 before persistence → a crash silently loses an acknowledged event → commit the inbox before responding.
• Save to a database and then publish once → a crash between writes strands work → commit an outbox with the inbox or lease inbox rows directly.
• Check for duplicates with a prior read → concurrent requests both pass → enforce a unique provider-event key atomically.
• Use only timestamp freshness → simultaneous duplicates still apply twice → combine freshness with stable-ID deduplication.
• Use only an event ID → a captured valid request can be replayed while its record is absent or expired → also verify signed freshness according to the provider contract.
• Assume arrival order is event order → late events roll state backward → use versions, monotonic transitions, or authoritative-state reconciliation.
• Perform remote side effects in the HTTP handler → latency causes retries and unknown outcomes → accept durably, then use asynchronous idempotent work.
• Log full payloads and signatures → observability becomes a data and secret leak → store encrypted evidence with restricted access and log redacted identifiers.

Follow-up Questions and Responses

Follow-up 1: Why return 2xx for a duplicate that has not finished processing?

The original event is already durably accepted, so another provider retry does not add recovery value. Returning a failure would create more duplicate traffic. The existing inbox row remains eligible for workers and reconciliation. This is safe only if the row is durable and not in a state that means acceptance was rolled back.

Follow-up 2: What if the process crashes after committing the inbox but before sending 2xx?

The provider retries. The unique key finds the committed row, no second job is created, and the receiver returns 2xx. This is the expected at-least-once path. If the client receives 2xx but the connection outcome is ambiguous to the provider, the same convergence applies.

Follow-up 3: Can Redis hold the idempotency keys?

It can be an accelerator, but a short-lived cache alone is weaker than the acceptance contract. Eviction, failover, or expiry could permit the same payment event to apply again. Keep durable processed identity for the required business and redelivery window; use Redis only when losing the key cannot violate that contract.

Follow-up 4: How do you handle secret-store downtime?

Use a bounded, encrypted in-memory cache of already trusted versions with explicit expiry and metrics. If no valid cached key exists, fail closed and return a retryable response so the provider can redeliver. Never accept unsigned work or fetch a key identifier from an untrusted request and trust it automatically.

Follow-up 5: How do you recover an out-of-order payment stream?

Prefer a provider object version or a monotonic domain transition and reject state regression. If the event only says that an object changed, retrieve its current authoritative state. For sequenced deltas, buffer a bounded gap, request missing versions, and alert when the gap exceeds the recovery window. Do not sort solely by local receive time.

Follow-up 6: Why is this still not exactly once?

The receiver can make its local mutation and processed marker atomic. It cannot atomically commit with an arbitrary remote email, bank, or provider service. A network failure can hide whether the remote side applied a request. A stable idempotency key, transactional outbox, retry, and reconciliation give an effectively-once business outcome where the remote API supports idempotency, while the transport contract remains at least once.