Prompt and context
An order, preference, or inventory service must offer local reads and writes in several Regions. Compare DynamoDB Global Tables MREC (multi-Region eventual consistency) with MRSC (multi-Region strong consistency), and explain routing, conflicts, failover, capacity, and recovery. The interview tests consistency and operational boundaries, not merely copying a database to another location.
Global Tables are managed multi-Region, multi-active replication: any replica can serve reads and writes. MREC is the default and replicates asynchronously; when the same item is modified nearly simultaneously in different Regions, DynamoDB resolves it at item granularity with an internal-timestamp last-writer-wins rule. MRSC synchronously replicates to at least one other Region before acknowledging a write, and strongly consistent reads on any replica return the latest value; it requires exactly three Regions (three replicas, or two replicas plus a witness).
What the interviewer is testing
A strong answer decomposes business invariants into items, partition keys, and cross-item transactions before choosing a mode from RPO, write latency, and read guarantees. Expect questions about silent loss under last-writer-wins, MRSC's three-Region constraint, visibility of MREC transactions across Regions, and ordering during failback.
Saying only “enable multi-Region replication” is insufficient. Explain local Region endpoints, how to avoid dual writes to one item, how to monitor ReplicationLatency, and how to handle deletes, retries, and duplicate events.
Questions to clarify first
Invariants and conflict shape
Ask whether order states may move backward, inventory must be linearizable, and one item can be edited concurrently in multiple Regions. Independently mergeable fields can be separate items or versioned subrecords. Cross-field atomicity requires confronting that MREC transactions are atomic only in the invoking Region and are not replicated as one transaction.
RPO, latency, and Region count
Clarify RPO, P99 write latency, failover time, and compliant Regions. MREC can use any number of available Regions and typically propagates within a second; MRSC trades some write latency for cross-Region strong reads and a zero-RPO target, while requiring three Regions.
Write ownership and recovery
Decide whether writes are truly multi-active or each tenant has a home Region with read replicas elsewhere. If the business cannot accept LWW, use IAM or routing to constrain writers and define the authority for conflict decisions during recovery instead of delegating semantics to replication.
30-second answer
“I choose the mode from business invariants: inventory decrements and cross-Region strong reads point to MRSC, while workloads that tolerate brief staleness and prioritize local write latency use default MREC. MREC is asynchronous and item-level LWW, so I home each order or tenant to one Region and add conditional writes, idempotency keys, and explicit versions; non-mergeable states do not rely on LWW. Every request uses its local endpoint, and traffic moves at the application edge during a regional failure. I monitor ReplicationLatency, replication errors, and conditional-write failures, then replay and audit recovery using versions, events, and business rules.”
Step-by-step solution
Step 1: Choose a consistency mode
MREC is the default, supports any number of Regions, and is eventually consistent, fitting preferences, catalogs, and stale-tolerant read models. MRSC synchronously replicates a write to at least one other Region before success; strong reads on any replica see the latest value. It requires exactly three Regions and does not support transaction operations. The mode is chosen at creation: replicas cannot mix modes and the table cannot be switched later.
Step 2: Define the write topology
Applications should use the DynamoDB endpoint in their local Region. Multi-active writes are appropriate only when the business can resolve concurrent conflicts. Strongly constrained objects such as orders and inventory can use home-Region writes with reads elsewhere, or single-writer partitions by tenant or order. Cross-Region calls add latency and failure surface; traffic switching belongs to the application entry point or routing layer.
Step 3: Constrain conflicts under MREC
MREC uses internal-timestamp LWW for near-concurrent updates to one item. Replicas converge, but discarded business changes do not become compensation work automatically. Use condition expressions for versions and include an idempotent request ID. Persist append-only events or conflict records for non-overwritable states. Represent deletes with an explicit tombstone or state so a stale update cannot resurrect a deleted object.
Step 4: Handle transactions and retries
MREC TransactWriteItems is atomic only in the invoking Region; other replicas can temporarily observe partial effects, so a cross-Region read is not transaction confirmation. Retries must distinguish conditional failures, throttling, and replication delay, with bounded exponential backoff and idempotency keys. MRSC does not support transaction operations, so cross-item invariants need a new model or service coordination.
Step 5: Plan failover and recovery
Monitor replication latency, replication errors, conditional-write failures, request Region, and business versions. During isolation, move entry traffic to a healthy Region. With MREC, first pause conflict-sensitive writes or re-home tenants, recording the cutover time and last visible version. After recovery, validate convergence with event logs, versions, and business rules; identical replicas alone do not prove a correct inventory or order.
Step 6: Capacity, security, and governance
Assess read/write capacity, auto scaling, and quotas for every replica. A new replica inherits source-Region capacity settings at creation and can be adjusted afterward. Enable deletion protection per replica, use IAM to restrict writer Regions and table operations, and remember that lost KMS permissions stop corresponding replication. Multi-account Global Tables support MREC, not MRSC, so account, Region, and audit ownership must be explicit.
Step 7: Verify and rehearse
Rehearse concurrent writes to one item in two Regions, conditional retries, partitions, deletes with late updates, replication spikes, traffic cutover, and failback. Verify eventual convergence, complete compensation events, and duplicate-safe replay, correlating ReplicationLatency and conflict counts with request IDs. Load tests must reflect real Region distances and capacity modes; single-machine latency is not a cross-Region estimate.
High-quality sample answer
I choose the mode from invariants. Critical data that requires cross-Region strong reads and a zero-RPO target can use MRSC, accepting exactly three Regions, higher write latency, and no transaction operations. Catalogs and preferences can use MREC. Because asynchronous item-level LWW cannot merge business semantics, orders and inventory use tenant or order home-Region single writes, condition expressions, versions, and idempotency keys. Mergeable fields are separate items; non-mergeable updates become conflict events.
The application uses local endpoints and the entry layer performs regional failover. I monitor ReplicationLatency, replication and conditional-write failures, and version differences, with bounded backoff for throttling and delay. During recovery I freeze affected writes, validate convergence from event logs and business rules, then reopen traffic gradually. I audit capacity, deletion protection, IAM, and KMS per replica and rehearse concurrent writes, partitions, late deletes, and duplicate replay.
Common mistakes
- Mistake: Assume Global Tables means unrestricted multi-active writes. → Why it fails: MREC LWW can discard non-mergeable business changes. → Fix: Assign write ownership or design explicit versions, events, and compensation.
- Mistake: Treat MREC transactions as globally atomic. → Why it fails: Atomicity is local to the invoking Region and other replicas may see partial effects. → Fix: Remodel cross-Region invariants and coordinate with events and idempotency.
- Mistake: Switch Regions blindly after a timeout. → Why it fails: Dual writes amplify conflicts and failback risk. → Fix: Pause or constrain writes, record versions, then switch along tenant or business boundaries.
- Mistake: Equate replica convergence with correct inventory. → Why it fails: LWW resolves replication, not business meaning. → Fix: Validate with conditional writes, event logs, compensation, and audit.
Follow-up questions and responses
Follow-up 1: When would you choose MRSC?
Choose MRSC when cross-Region strong reads and a zero-RPO target outweigh write latency and Region flexibility. It requires exactly three Regions, optionally one witness, and does not support transactions. If the workload needs arbitrary replica counts or multi-account replication, reassess MREC and service-level coordination.
Follow-up 2: What if LWW overwrites an inventory decrement?
Do not recover the quantity from LWW. Use a condition expression and version for the decrement, assign a stable write owner by product or inventory shard, and record every decrement as an idempotent event. A conflict detector compares event sequences and emits compensation or a human queue; reads check both version and visibility time.
Follow-up 3: What do you do when MREC replication lag rises?
Group alerts by source Region, destination Region, and business impact. Limit cross-Region writes or return tenants to their home Region to stop creating conflicts. Bound client retries, then reconcile versions, late events, and delete markers before restoring multi-active writes. ReplicationLatency measures propagation, not business correctness.
Follow-up 4: How do you test failback?
Exercise traffic cutover, old-Region recovery, requests arriving on both sides, and late deletes. Record Region, version, and idempotency ID for every request. Verify the old Region cannot overwrite newer state, conflict events remain traceable, and compensation is repeatable. Report RPO, RTO, conflict count, and human intervention.
Follow-up 5: Why cannot replicas mix MREC and MRSC?
Consistency mode is a table-level creation setting: replicas cannot use different modes and the table cannot be switched after creation. A changed requirement needs a new table, migration, and a controlled dual-write or replay window, with capacity, permissions, client compatibility, and rollback validated first.