Prompt and context
The interviewer asks: “Our B2B SaaS replicates data into two regions. Should customers be able to trigger a switch when the primary region fails?” Assume customers cannot operate the underlying database directly, and the product must preserve tenant isolation, data residency, and auditability. This fits platform product manager, infrastructure product manager, and technical program manager interviews.
The test is product boundary-setting, not a reflexive “automation is faster.” AWS positions multi-Region designs for extreme resilience but warns that cross-Region dependencies weaken the posture; Google Cloud says recovery must be designed, built, and tested; Azure frames the choice around business requirements, RTO, RPO, cost, and complexity.
What the interviewer evaluates
- Can you turn “customers want control” into measurable RTO, RPO, residency, and compliance requirements?
- Can you distinguish platform-automatic, operator-approved, and customer-requested switching?
- Can you name failure modes involving replication lag, split brain, DNS caching, quota, and false signals?
- A strong answer scopes the operation to a tenant control plane with prechecks, approval, rehearsal, audit, and failback. A weak answer exposes a raw failover button.
Clarifying questions before answering
What failure are we solving?
A single-tenant application failure may be handled by isolation or restart; an entire regional outage is the case for cross-Region switching. If the need is only latency, multi-Region may not beat a simpler multi-AZ design.
What are the RTO and RPO?
Asynchronous replication can lose the newest writes. A zero-RPO requirement cannot be satisfied by an ordinary asynchronous replica button. RTO determines warm capacity; RPO determines whether replication lag is acceptable.
Which data may leave the original region?
Residency, encryption keys, backups, and logs change the eligible target regions. A “secondary region” is not automatically a compliant region.
A 30-second answer
“I would tier the experience by each tenant’s availability target, acceptable data loss, and residency constraints. The platform should normally switch automatically or with operator approval from strong health signals; only tenants that pass prechecks should submit an audited switch request, rather than promote a replica directly. The request checks replication lag, target capacity, keys, versions, and dependencies, then freezes or drains writes. After the switch, we watch errors, write state, and RPO, and restore normal operation only after failback criteria are met. I would rehearse with a small tenant set, compare recovery time, false switches, and customer impact, and then decide whether to expand access.”
Step-by-step solution
- Define the product contract. Treat regional failover as a tenant-scoped disaster-recovery operation. State target RTO, maximum RPO, unavailable features, price, and customer responsibilities. A replica that cannot meet the contract must not be shown as ready.
- Create three control tiers. Platform automation fits unambiguous regional health signals; operator approval fits shared dependencies with business impact; a customer request should create a constrained command evaluated by policy. Customers must not edit DNS, promote databases, or replay queues directly.
- Run prechecks. Verify replica catch-up time, target quota, application version, key availability, queue backlog, external dependencies, and residency. A failed check should return an actionable reason before accepting the operation.
- Protect write authority. Stop or drain writes in the primary, record the last confirmed position, and promote exactly one write authority. Expose the unknown window created by asynchronous replication; “replicated” does not mean zero loss.
- Verify and fail back. Use synthetic transactions for login, reads, writes, jobs, and audit. Watch error rate, replication position, queue age, and tenant success. Before failback, catch up in the reverse direction and rehearse conflicts so recovery does not create two writers.
- Choose a rollout. Start with read-only simulations for internal or willing tenants, then operator-approved switching, and only later consider automation. Pause on a defined stop condition: false switches, RPO breach, insufficient target capacity, or missing audit evidence.
Model answer
I would not hand customers a raw button. First I would confirm whether they need a cross-Region RTO, what RPO they can accept, and which residency constraints apply. The product can expose “request failover,” but the request must pass tenant policy, replication position, target capacity, version, key, and dependency checks.
During the switch, the platform freezes or drains writes in the primary, records the last confirmed position, promotes one write replica, and states the expected loss window and unavailable features. Afterward, synthetic transactions verify reads, writes, jobs, and audit. We measure errors, replication state, and recovery time per tenant. Failback starts only after reverse catch-up proves that there is no split brain.
I would rehearse with internal tenants, add operator approval, and consider automation only after the evidence is good. If false switches, RPO, or capacity prechecks cross a threshold, disable customer-triggered access and retain the audit trail. This gives customers visibility and bounded control without shifting disaster-recovery risk to them.
Common mistakes
- Treating multi-Region as automatic availability → ignores promotion, dependencies, and failback → document RTO/RPO, rehearsal, and failback per component.
- Letting customers promote the database directly → risks split brain or cross-tenant impact → expose an audited tenant command executed by policy.
- Checking only regional health → DNS health does not prove data, keys, or quota are ready → include replication position, capacity, version, keys, and dependencies.
- Promising zero data loss → asynchronous replication has an unconfirmed window → state RPO, record the last confirmed position, and show the possible loss range.
- Skipping failback → the recovered primary can create dual writes and drift → design reverse catch-up, conflict detection, and staged failback.
Follow-ups and responses
What if the customer requires zero RPO?
Explain that asynchronous replication cannot provide zero RPO. Evaluate synchronous replication or business-level dual writes, then recalculate latency, availability, cost, and consistency. If the target still cannot be met, replace it with a measurable maximum-loss window.
What if a false health signal triggers automation?
Require multiple signals, a minimum duration, and a human abort window. For high-value tenants, enter a frozen and prechecked state before promotion. Record every decision and tune thresholds using false-switch rate and recovery time.
What if the target region has no capacity after the request?
Precheck quota and capacity, and reserve a budget or autoscaling plan for critical tenants. On failure, return the reason and preserve the primary state; do not switch to an unprepared region to improve button success.
Can residency rules be waived during an outage?
Do not assume an exception. Put allowed regions, encryption keys, and log boundaries in the tenant contract and policy. If cross-border movement is prohibited, offer same-region multi-AZ resilience or an explicit degraded mode.