Prompt and setting
An event producer emits business events to external HTTP endpoints. The platform must preserve events durably, deliver at least once, make duplicates safe, and give each tenant clear retry and replay controls.
What the interviewer tests
- Choosing an explicit delivery guarantee instead of promising exactly once across HTTP.
- Separating ingestion, scheduling, delivery attempts, and subscriber-side effects.
- Combining event IDs, signature timestamps, retry policy, dead-letter handling, and fairness.
Clarifying questions before answering
- What is the event volume, payload size, endpoint count, and maximum delivery delay?
- Is ordering required per tenant, per endpoint, or not at all?
- Can consumers process an event idempotently, and how long must deduplication state live?
- What replay, deletion, privacy, and audit requirements apply to payloads?
30-second answer framework
I would persist an immutable event with a stable ID, enqueue delivery attempts, and return quickly from the receiver path. Workers sign the raw payload with a timestamp, apply exponential backoff with jitter, and classify responses into retryable and terminal failures. Consumers deduplicate by event ID before side effects. A per-tenant scheduler, circuit breaker, and dead-letter queue prevent one offline endpoint from starving others; replay creates a new attempt without changing the original event identity.
Step-by-step deep dive
1. Make the event durable first
Write the business event and its delivery record transactionally or through a reliable outbox. The delivery record stores tenant, endpoint, event ID, attempt count, next-attempt time, and status. A crash after sending but before recording success is expected; it produces another attempt, so consumers must tolerate duplicates.
2. Verify and sign safely
Sign the exact raw bytes plus a timestamp, and include the event ID in the signed message. Stripe documents timestamped signatures to limit replay attacks. Consumers verify the signature before parsing, reject timestamps outside a configured tolerance, and rotate secrets without invalidating an already queued event unexpectedly.
3. Classify responses and retry
Treat network timeouts, connection failures, and selected 5xx responses as retryable. Treat malformed authentication, unsupported event versions, and most 4xx responses as terminal or operator-reviewed. Use exponential backoff with jitter, a maximum attempt window, and a dead-letter state. Do not retry indefinitely against an endpoint that is failing permanently.
4. Make duplicates and replays explicit
The consumer stores processed event IDs with a durable uniqueness constraint and commits that record with the business side effect when possible. A manual replay reuses the immutable event payload and records a new delivery attempt, while dashboards distinguish original delivery, automatic retry, and operator replay. Exactly-once side effects require a consumer transaction; the network itself only gives at-least-once delivery.
5. Scale without cross-tenant starvation
Partition queues by tenant or endpoint, enforce per-tenant concurrency and rate limits, and reserve capacity for healthy tenants. A circuit breaker pauses an endpoint after repeated failures. Metrics should include age of oldest pending event, success latency, attempt distribution, duplicate rate, signature failures, and dead-letter volume.
High-quality sample answer
“I would persist each event with a stable ID before scheduling delivery and promise at-least-once semantics. Each attempt signs the raw payload with an event ID and timestamp. Workers classify timeouts and 5xx responses for jittered retry, while terminal 4xx responses move to dead letter. Consumers deduplicate event IDs inside their side-effect transaction. Per-tenant queues, circuit breakers, age-based alerts, and a replay workflow keep an offline subscriber from starving others and make recovery auditable.”
Common mistakes
- Promise exactly-once over HTTP → crashes create ambiguous outcomes → state at-least-once and require idempotent consumers.
- Sign parsed JSON instead of raw bytes → equivalent formatting can fail verification → sign and verify the exact payload bytes.
- Retry every 4xx forever → permanent failures consume capacity → classify errors and dead-letter terminal cases.
- Use one global queue → one tenant can starve everyone → partition, rate-limit, and reserve per-tenant capacity.
Follow-up questions and responses
How long should deduplication state live?
At least as long as automatic retries and the supported replay window, plus a safety margin. If replay can occur months later, keep a compact event ledger or require consumers to choose a replay identity and retention policy explicitly.
Should a replay get a new event ID?
Usually no: the business event identity stays stable, while the delivery attempt gets its own ID and audit record. This lets consumers recognize a replay as the same event and prevents duplicate business effects.
What if a consumer returns 200 before its side effect commits?
The consumer contract is broken; the platform cannot infer success from the response. Consumers should acknowledge after durable acceptance, use an inbox or transactional deduplication record, and expose reconciliation for ambiguous outcomes.