Prompt and scope
Your B2B SaaS sends billing, access, or order events to customers through webhooks. Customers miss events when their endpoints time out, return 5xx responses, or deploy broken code, so support engineers manually inspect logs. Decide whether to offer a customer-facing replay console, and explain the MVP, metrics, and boundaries.
This is a product and integrations interview question. A public integrations interview guide treats replay handling, idempotency, backoff, and customer-visible health dashboards as useful answer elements. Stripe documents duplicate and out-of-order events, automatic retries, and separate manual retry windows in its Dashboard and CLI. GitHub documents redelivery of failed webhook deliveries as part of its webhook workflow.
What the interviewer evaluates
The interviewer wants you to turn “customers want a button” into evidence about scale, risk, value, and scope. A strong answer separates automatic retries from human-triggered replay, delivery from business success, and an immutable event from a new delivery attempt. It also states who may replay, how long data is retained, and how duplicate side effects are prevented.
A weak answer says “build a retry button.” A strong answer proposes retention, authorization, auditability, rate limits, idempotency guidance, failure reasons, and success criteria. It also explains when a query API, reconciliation flow, or support tool is safer than general-purpose replay.
Clarifying questions before answering
- Does the customer need to recover an undelivered event, or send a delivered event to a new endpoint? Those require different permissions and retention rules.
- Do payloads contain personal data, payment data, or tenant secrets? That determines masking, encryption, export, and audit requirements.
- How long do automatic retries run, what is the failure rate, and what share of support tickets is involved? Without a baseline, value is unproven.
- Does the client deduplicate by event ID? If not, the product must warn that replay can trigger a side effect again.
30-second answer framework
“I would measure failed-delivery volume, customer loss, and support cost before committing to the feature. If the value is real, I would start with failed events only: retain an immutable payload and delivery result, let an authorized tenant admin act within a retention window, enforce permissions and rate limits, and treat every replay as a new delivery attempt. I would measure recovery success, duplicate side effects, support tickets, and storage cost. If payloads are sensitive or clients lack idempotency, I would start with reconciliation and human approval instead of arbitrary replay.”
Step-by-step answer
First map the current path: event creation, signing, delivery, client response, automatic retries, and terminal failure. Stripe’s public behavior shows exponential-backoff retries in live mode, a Dashboard resend window of up to 15 days after event creation, and a CLI resend window of up to 30 days. Treat those numbers as competitor reference points, not as your own promise. Store the endpoint, status code, latency, latest attempt, and next action for each failure.
Then test the product value. If failures are rare, customers can compensate through a pull API, and replay could cause expensive duplicate charges, query and reconciliation may be safer. If failures cluster around customer deployments and support repeats the same recovery action, a console can reduce time to recovery and support cost. Success criteria should include recovery rate, duplicate business operations, replay volume per tenant, and retention cost.
The MVP should replay only terminally failed events that are still inside the retention window. Keep the event payload immutable. The action creates a new delivery attempt carrying the original event ID, an attempt ID, operator, reason, and timestamp. The exact signature, timestamp, and replay marker must fit the existing protocol; customers must not mistake it for a first delivery. For sensitive payloads, hide the body by default and expose status plus event ID, with step-up authorization when needed.
Make idempotency and ordering explicit. Stripe says event order is not guaranteed and an endpoint can receive an event more than once, so the UI and docs should require deduplication by event ID and warn that replay does not undo a side effect that already happened. If customers need a missing sequence, provide time filtering and per-event confirmation instead of replaying an entire history at once.
Security and cost are release gates: restrict actions to tenant administrators or a dedicated permission; set per-tenant and per-endpoint rate limits; make repeated clicks idempotent; audit the operator, event, target endpoint, and result; and align retention, encryption, and deletion with the privacy policy. Batch replays should be queued and cancellable so recovery does not become a traffic surge.
There are three alternatives. Keep automatic retries plus support-assisted recovery for the lowest build cost, accepting slow and non-scalable recovery. Offer event lookup and a pull API for customers with mature engineering teams. Or build a full event store with arbitrary time-range replay, which is most capable but carries the highest storage, compliance, and misuse risk. Start with failed-event replay and expand only when metrics show a need for historical replay.
High-quality sample answer
I would not frame this as “should we add a button?” I would start with the consequence of failed deliveries. Assume failures cluster around customer releases, support manually verifies and resends them every week, and tenant-scoped encrypted retention is acceptable. I would ship a constrained MVP. The first version lists terminally failed events, stores the immutable payload, status code, and latest attempt, and lets a tenant admin request replay inside a default 15-day retention window. Every action has authorization, a reason, rate limits, and an audit record. Replay creates a new delivery attempt, keeps the original event ID, and clearly requires client-side deduplication.
Automatic retries remain in place because manual replay is not a substitute for normal delivery. I would measure recovery rate, duplicate side effects after replay, time to resolve support tickets, replay volume per tenant, and storage cost. If duplicate charges or privacy risk rise, I would narrow the feature to human approval or a reconciliation API. If recovery improves while support cost falls, I would consider batch replay and longer retention.
Common mistakes
- Mistake → Treat replay as a new business command; failure → the customer may already have processed the event while the response was lost; fix → separate delivery attempts from business events and require event-ID idempotency.
- Mistake → Let users edit a historical payload and send it directly; failure → signatures, auditability, and event truth break; fix → keep the original read-only and isolate edited test variants in a sandbox.
- Mistake → Retain every webhook body forever; failure → privacy, compliance, and storage costs grow without bound; fix → define tenant-level retention, encryption, deletion, and masking.
- Mistake → Use replay count as the only success metric; failure → more replays may signal an unreliable delivery system; fix → combine recovery, duplicate side effects, support cost, and endpoint health.
Follow-up questions and responses
What changes if a customer asks to replay a payment event from 90 days ago?
First verify the lawful retention basis and whether the payload contains payment or personal data. If long retention is not justified, provide the event ID, current resource lookup, and reconciliation result rather than restoring the body. If the business truly needs 90-day recovery, use tiered storage, tenant authorization, step-up approval, stricter audit, and a plan or usage charge that reflects long-term storage.
Who is responsible if replay causes a duplicate charge?
Product controls cannot be replaced by a disclaimer. The UI should show that the event may already have succeeded, require client idempotency, and disable self-service replay or require confirmation for high-risk event types. The service records event ID and attempt ID, offers preview, rate limits, and cancellation for queued jobs, and keeps an auditable chain while the contract defines responsibility.
How do you recover from an outage that affected one million events without creating another incident?
Pause automatic replay and recover in batches by endpoint health and error class. Apply per-tenant quotas, exponential backoff, concurrency caps, and a circuit breaker. Send a small sample first, observe 2xx rate, latency, duplicate side effects, and downstream queue depth, then ramp gradually. Show queue estimates and a cancel action so support can stop the batch when signals deteriorate.
Why not offer arbitrary full-history replay immediately?
Arbitrary replay combines search, compliance, and business compensation into a dangerous action. Unless the product already has a durable event store, versioned payloads, permission controls, and client idempotency, close the terminal-failure recovery loop first. Roadmap expansion should follow failure distribution, customer value, and security evidence rather than feature completeness.