Question and when it applies
Design an asynchronous message flow in which messages enter a dead-letter queue (DLQ) after bounded retries and operators can replay selected messages after fixing the cause. Explain isolation, investigation, batch selection, duplicate side effects, and protection for normal traffic.
Amazon’s software-development interview topics emphasize applying knowledge to solve problems. AWS documents DLQs as isolation for unconsumed messages with retry limits and alarms; Google Pub/Sub documents dead-letter topics and replay or seek semantics. The key is an operational recovery loop, not a queue diagram alone.
What interviewers assess
- Classification of transient errors, poison messages, business rejection, and expiry.
- Metadata for version, tenant, partition, trace, attempts, and failure reason.
- Replay controls: idempotency, scope, rate, approval, and stop conditions.
- Ordering, retention, duplicate delivery, and at-least-once semantics.
- Metrics that prove recovery rather than simply moving messages back.
Questions to clarify before answering
- Is delivery at-least-once, at-most-once, or business-level exactly-once?
- Which failures are retryable?
- How long are messages retained and when do they lose value?
- Is order required for an aggregate key?
- Can consumers make idempotent side effects?
- Who may inspect, replay, or delete messages?
- What are normal SLOs, queue capacity, and replay capacity?
- Does a failed replay enter the same DLQ or a replay-DLQ?
30-second answer framework
“I define delivery semantics, retention, and ordering keys first. Retryable errors use bounded backoff; poison and business-rejection messages go to a DLQ with reason, attempts, version, and trace ID. After the fix, an operator creates a scoped replay batch, validates a sample in isolation, and replays at a limited rate. Consumers protect side effects with idempotency keys. I monitor DLQ age, replay failures, duplicates, and downstream latency, and pause when thresholds are crossed.”
Deep answer, step by step
Step 1: Define the failure state machine
Separate normal, retry, DLQ, manual-fix, and replay-DLQ states. Permanent business failures must not retry forever.
Step 2: Define message metadata
Keep event ID, business idempotency key, creation time, tenant or partition, schema version, attempts, original trace ID, and error class. Preserve the original payload immutably.
Step 3: Set retry and DLQ rules
Choose maximum receives, backoff, and retention. AWS SQS documents source-queue and Region constraints and recommends DLQ alarms. Expired or revoked messages need an audited disposition.
Step 4: Make replay safe
A replay request includes a filter, target consumer version, rate limit, batch size, approver, and expiry. Validate a sample first, then replay in batches. Isolate replay capacity from normal traffic.
| Control | Purpose | Failure action |
|---|---|---|
| Idempotency key | Prevent duplicate side effects | Reject or return prior result |
| Rate limit | Protect consumers and dependencies | Pause replay |
| Batch scope | Limit blast radius | Narrow filter |
| Approval and audit | Establish accountability | Block unauthorized action |
| Replay-DLQ | Isolate repeated failures | Create a new diagnosis batch |
Step 5: Handle ordering and concurrency
When order matters, partition by aggregate key and prevent normal and replay consumers from processing the same key concurrently. Do not claim end-to-end exactly-once delivery from the queue alone.
Step 6: Protect side effects
Use event IDs or business idempotency keys for conditional writes. Payments and email need idempotent request keys and result lookup; deleting a message is not a rollback.
Step 7: Operate with observability
Monitor DLQ depth, oldest age, error classes, replay throughput, replay failures, duplicate effects, and downstream latency. Record operator, reason, scope, timing, outcome, and stop events for every batch.
Step 8: Budget capacity and stop safely
Estimate replay capacity and keep new consumers compatible with old schemas. Pause when the cause is not fixed, dependencies overload, or duplicate rate rises; preserve evidence.
High-quality sample answer
“This is an at-least-once order-event flow. Network timeouts retry up to five times with backoff; schema errors, authorization failures, and expired events enter the DLQ. Each message keeps event ID, order ID, tenant, schema version, first-enqueue time, attempts, error class, and trace ID.
After the fix, an operator chooses a tenant and time window, target version, rate limit, approver, and expiry. Fifty messages are validated in an isolated consumer, then replay runs at ten percent of normal traffic. The order state uses a conditional write keyed by event ID, and payment requests reuse the business idempotency key. Normal and replay processing for one order cannot run concurrently.
Alerts cover oldest age, replay failures, duplicate writes, and dependency latency. Any threshold pauses the batch and routes repeated failures to a replay-DLQ. The batch record contains the filter, version, operator, result, and stop reason.”
Common mistakes
- Treating the DLQ as a trash can without evidence.
- Retrying poison messages forever.
- Requeueing without scope, approval, or rate control.
- Assuming the queue provides end-to-end exactly-once.
- Omitting idempotency and creating duplicate charges or emails.
- Ignoring ordering keys.
- Monitoring only queue length.
- Sharing capacity between replay and normal traffic without protection.
Follow-ups and how to answer
Follow-up 1: Why not increase retry attempts?
Retries fit transient faults; poison and permanent business failures consume capacity. Set limits from fault type, retention, and business waiting cost.
Follow-up 2: How do you preserve order for one order?
Partition by order key, block concurrent normal and replay processing, and explain how state transitions reject stale events.
Follow-up 3: What if the dependency is not idempotent?
Use local deduplication and result lookup; otherwise require manual reconciliation, compensation, or a provider idempotency mechanism.
Follow-up 4: What if replay fails again?
Send it to a separate replay-DLQ, retain the original batch and new error, pause the affected filter, and notify the owner.
Follow-up 5: How do you choose replay rate?
Use consumer capacity, dependency quotas, normal headroom, and recovery time. Load-test a small batch before increasing the rate.
Follow-up 6: When may you discard a message?
Only after an explicit business decision that it is expired, revoked, or valueless, with audit evidence and a retained reason.