Prompt and context
Several replicas share coordination storage and must ensure that only one instance runs a scheduled settlement job at a time. Design the lease record, campaign, renewal, handoff, and observability. Cover partitions, process pauses, clock drift, and storage failure. A lease controls who may start work; business writes still need idempotency and conditional checks.
What the interviewer is testing
- Whether you separate lease safety from business side effects.
- Whether you understand atomic compare-and-set, term fencing tokens, and quorums.
- Whether you can explain why pauses, partitions, and a revived old leader do not create valid dual writers.
- Whether you define measurable signals, fault injection, and recovery behavior.
Clarifying questions before answering
Confirm the allowed pause window, tolerated duplicate execution, coordination store consistency, cross-region latency, replica count, and clock-synchronization quality. If cross-region strong consistency is required, make the availability and election-latency trade-off explicit.
30-second answer framework
Use coordination storage that offers linearizable reads and conditional writes. Each candidate competes with a unique identity and monotonically increasing term; only the successful atomic create or update becomes leader. The leader renews before expiry. Every business write carries the term token, and downstream systems reject older tokens. A candidate takes over only after confirming the new state. During a partition or long pause, an instance stops side effects; a short outage is safer than dual writers.
Step-by-step deep dive
1. Data model and atomic operations
The lease record contains holder identity, expiry, term token, version, and last renewal time. Competition uses compare-and-set: create when absent, or update only when the version is unchanged and the lease is expired. Kubernetes represents leases as coordination objects containing holder and renewal information; implementation must verify the storage consistency model instead of trusting an eventually consistent cache.
2. Renewal and self-demotion
Renew well inside the lease duration, leaving room for network jitter and scheduler pauses. If renewal fails, confirmation cannot be read, or the process pauses past the safety window, stop side effects immediately and campaign again after recovery. Do not decide another holder's expiry from a local wall clock alone.
3. Fencing and business idempotency
Every successful campaign produces a monotonic token. Workers, conditional database updates, or downstream services reject requests carrying an older token. Thus an old leader that wakes after a pause cannot overwrite the new leader's writes. Settlement still needs idempotency keys, transaction boundaries, and retry handling.
High-quality sample answer
I define safety as preventing two valid leaders from writing the same resource at once; availability permits a short pause after lease expiry. The coordination store supplies linearizable CAS. A candidate records its identity and increasing term, and the leader renews with heartbeats. Every side effect carries the term token, which downstream conditional writes compare. If a partition, GC pause, or lost renewal confirmation occurs, the instance stops and campaigns only after reconfirming state; sleeping for a few seconds is not a safety proof. Raft uses terms and majority voting for log leadership, while a Kubernetes Lease is a lighter coordination record. Both require an explicit consistency model, timeout budget, and recovery plan. I would inject leader crashes, partitions, clock offsets, long pauses, and unavailable storage, then check for dual writes, stale-token writes, and recovery-time violations.
Common mistakes
- Keeping a lock only in process memory or relying on eventually consistent Redis reads while claiming no dual leader is possible.
- Comparing timestamps without monotonic terms and fencing tokens.
- Continuing the current batch after renewal failure, leaving an old leader a write window.
- Equating one leader at a time with no duplicate execution ever.
- Testing only normal elections and not pauses, partitions, storage latency, or storage failure.
Follow-up questions and responses
Does lease expiry guarantee that the old leader stopped?
No. A process can pause or remain isolated while still running. Validate the fencing token at the business boundary; expiry only means the coordination layer no longer recognizes that holder.
How do you choose lease and renewal intervals?
Derive them from the failure-detection target, cross-region p99 latency, scheduler-pause budget, and storage jitter, with several renewal cycles of margin. Monitor renewal failures and election churn instead of copying a fixed millisecond value.
What if coordination storage is unavailable?
Stop new side effects and keep only read-only or safely committed results. After recovery, reread the term and campaign. If continuous operation is mandatory, explicitly degrade to sharding, multi-active ownership, or human takeover and restate the safety proof.