Prompt and context
Several stateless workers must have exactly one instance run a scheduled settlement job at a time. Workers can crash, restart, pause, or become partitioned, and the system must not let two workers write for a prolonged period. Design leader election with a coordinator, lease, renewal, fencing, failover, and operational boundaries. The core is safe single-writer coordination, not merely a mutex.
What the interviewer is testing
Define safety and liveness
Safety means at most one leader for a term is accepted by the resource; liveness means a healthy candidate eventually takes over after the old lease is confirmed expired. A minority partition cannot announce a new leader just to preserve availability.
Choose a coordinator with consensus semantics
The leader record needs linearizable compare-and-set, leases, and watch semantics, such as etcd. A Redis TTL or local clock alone cannot prove that a stale leader is unable to write.
Prevent stale writes
After a lease expires, the old process may recover and continue. Every downstream write should carry a monotonically increasing fencing token, and the resource must reject older tokens to prevent split-brain damage.
Questions to clarify first
- Can the job run twice, or must the downstream accept strictly one serialized execution?
- Is the election global, per tenant, per shard, or per job?
- What failover time and pause window are acceptable?
- How many coordinator failure domains and what quorum/backups are available?
- Can downstream resources validate a fencing token and recover idempotently?
- Are watch events, audit history, alerts, and manual transfer required?
A 30-second answer
“I would define the election scope and failure budget first, then store a leased leader record in a consensus-backed, linearizable coordinator. Candidates compete with a transactional create-or-compare update; the winner gets a higher fencing term and renews within the TTL. Renewal failure stops new work and writes. Every downstream write validates the term, so a recovered stale leader cannot write. Watches only accelerate a new CAS attempt. I would monitor term, renewal latency, failover time, fencing rejects, duplicates, and quorum health.”
Step-by-step deep answer
Define the term record
Store election_name, leader_id, lease_id, term, candidate metadata, and timestamps. The term or fencing token must increase monotonically and be assigned atomically by the coordinator, not generated from a client clock.
Choose the coordinator and write condition
A candidate creates an ephemeral lease-backed record; a linearizable transaction may write only when the key is absent. If a leader exists, candidates watch the key and retry. The etcd election API lets participants compete on one election name with one successful leader at a time.
Renew and fail closed
The leader renews through an independent keepalive loop. Renewal timeout, connection loss, process pause, or a local-clock anomaly moves it to suspect, where it stops accepting new work and downstream writes. It must not keep working confidently while disconnected from the coordinator.
Handle failover
A candidate cannot decide that the old leader is dead from its local TTL. It must observe coordinator-confirmed lease expiry or deletion, then compete with a CAS. Failover time is the sum of TTL, detection, and scheduling delays, so set a bound with jitter margin.
Add a fencing token
The new leader obtains a higher term and attaches it to database updates, messages, or external API requests. A resource stores the highest accepted token and rejects a lower one. This blocks dangerous writes even when an old process has not stopped.
Handle partitions and split brain
A minority partition cannot issue new terms. A candidate unable to reach coordinator quorum must stop or remain read-only. After reconnecting, an old leader must reread the current term and compete again rather than trust cached state.
Pseudocode
~~~text campaign(): lease = coordinator.grant(ttl) result = coordinator.txn(key absent -> put(candidate, lease, next_term)) if result.succeeded: token = result.term keepalive(lease) runwithfencing(token) else: watch(key)
onkeepalivefailureorexpiry: stopnewwork() stopdownstreamwrites() ~~~
Complexity, recovery, and observability
Each election and renewal involves coordinator round trips; more candidates add watch and retry load, so use exponential backoff with jitter. Record current leader, term, renewal RTT, lease expiries, election duration, fencing rejects, duplicate jobs, and quorum health to reconstruct term transitions.
| Mechanism | Solves | Still required |
|---|---|---|
| Linearizable CAS | Prevents two winners | Coordinator quorum |
| Lease keepalive | Detects process failure | Pause and partition handling |
| Fencing token | Rejects stale writes | Durable downstream validation |
| Watch and backoff | Speeds failover and reduces load | Cannot replace safety proof |
Model answer
“I would store a leased leader record in a consensus-backed coordinator with linearizable transactions. Candidates compete by creating the record only if absent; the winner receives a monotonically increasing term and renews it within a TTL. If renewal fails, it immediately stops new jobs and downstream writes. Every database write, message, or external call carries the fencing token, and the resource rejects a token below its highest accepted value, so a paused or partitioned old leader cannot continue writing. A minority cannot issue a new term, while watch only reduces election delay. I would monitor renewal RTT, terms, failover duration, fencing rejects, duplicate jobs, and quorum, with a manual kill switch.”
Common mistakes
Using only a Redis TTL
Lease expiry and a client's observation of expiry are not the same. Network delay and pauses can make two clients believe they may work. Use linearizable coordination plus downstream fencing.
Treating a lease as write protection
A lease helps detect failure but cannot stop an old process instantly. Without token validation, the stale leader can overwrite the new leader's result.
Generating terms from local time
Clocks can drift, jump, or pause. The coordinator must allocate and persist terms atomically.
Forcing takeover during a partition
A minority cannot confirm the old leader's state. Forced takeover creates split brain; a safe design accepts a short availability loss.
Letting watch events decide safety
Watches can be delayed, lost, or reconnected. They should trigger a reread and CAS, not replace linearizable reads and writes.
Omitting job idempotency
Correct election does not prevent duplicates after crashes or message redelivery. Jobs need idempotency keys, progress records, or replayable transactions.
Follow-up questions and responses
How is leader election different from a distributed lock?
A lock protects a critical section; election maintains a long-lived coordinator role with terms, renewal, watches, fencing, and failover. They can share a coordinator, but their safety surface is broader.
How should you choose the TTL?
Cover normal renewal RTT, GC or scheduler pauses, network jitter, and the failover SLO. A short TTL causes churn; a long TTL delays takeover. Calibrate with fault injection.
Why must the resource check the fencing token?
The coordinator cannot instantly stop every stale process. Resource-side rejection blocks dangerous writes while that process is still alive.
What happens when an etcd cluster loses quorum?
It cannot commit a new term or lease state; the current leader must stop writes after it cannot renew. Candidates compete again after quorum returns.
How do you handle a long leader pause?
Renewal fails during the pause and a new candidate can take over. When the old process resumes, its old token is rejected and it must rejoin the election.
How do you test for split brain?
Inject process pauses, partitions, clock jumps, and coordinator failures. Verify that only one token per term is accepted for writes, and inspect failover duration and rejection records.