Prompt and context
Design an event log for a multi-tenant account service. Each balance, permission, or configuration change emits an event, and consumers can replay history to rebuild state. An instance failure cannot scan an unbounded history from the beginning. The system needs high-throughput appends, per-tenant ordering, independent consumer progress, snapshot recovery, and bounded storage cost. Explain deletes, out-of-order events, and schema evolution.
This fits senior system-design, platform, and data-infrastructure interviews. Martin Fowler’s Event Sourcing article defines the core idea of keeping state changes as an event sequence. Apache Kafka’s design documentation explains partitioned logs, consumer positions, and log compaction that retains the latest value for each key. Public system-design prompts also list partitioned append-only logs, replication, retention, compaction, and recovery as evaluation areas. These sources support the topic’s representativeness, but do not establish a fixed company prompt or interview frequency. The category is system-design because the core skills are end-to-end consistency, recovery boundaries, and capacity trade-offs.
What interviewers evaluate
First, can the candidate separate event history from materialized current state? A snapshot is a derived acceleration layer; it cannot replace immutable events or cross an inconsistent event boundary.
Second, does the partition key satisfy both ordering and scale? Partitioning by tenant or aggregate ID preserves one aggregate’s order, but hot tenants, cross-aggregate transactions, and global order require explicit limits.
Third, do they understand compaction semantics? Key compaction retains the latest record per key and fits a state-change stream; it is not an event history. Deletes need tombstones and a retention window, and consumers cannot assume arbitrary offsets mean the same thing before and after compaction.
Finally, do they cover operations: snapshot validation, event versions, atomic checkpoints, replication acknowledgements, retention cost, consumer lag, and schema compatibility must be observable.
Clarifying questions to ask first
- Are events immutable audit facts or rebuildable current-state changes? Auditing needs full history; state streams can use key compaction.
- What is the ordering scope? Same aggregate only, per tenant, or global? Each guarantee changes partitioning and throughput.
- Must consumers replay from any time? If so, latest-key compaction alone is insufficient; keep an archive or separate audit log.
- Who creates and validates snapshots? Producers, consumers, and snapshot workers have different responsibilities; bind snapshots to a log position and schema version.
- How are deletes represented? A versioned tombstone or a business deletion event has different compaction and compliance meaning.
30-second answer framework
“I partition each aggregate’s events by aggregate ID, append only, and acknowledge producers at a committed replicated offset. Events include an event ID, aggregate version, schema version, and timestamp. Consumers checkpoint offsets after updating their materialized view, use event IDs for at-least-once deduplication, and rebuild from a snapshot plus subsequent events. A snapshot stores state, its last event offset, and schema version; recovery validates it before replaying the next offset. Key compaction applies only to rebuildable current-state streams and retains tombstones for a defined window; audit events go to an uncompacted archive. I monitor replication lag, consumer lag, snapshot age, compaction backlog, and replay verification differences.”
Deep-dive answer
1. Define event fields and ordering boundaries
An event includes eventId, aggregateId, aggregateVersion, schemaVersion, payload, creation time, and source. aggregateVersion increases monotonically for one aggregate. Conditional append rejects an old version, preventing two concurrent writers from silently overwriting one another.
Use aggregate ID as the partition key so one aggregate enters one ordered log. Do not promise a global cross-partition order. If the product needs a cross-aggregate atomic fact, encode a transaction result as one aggregate event or use a transactional outbox rather than sorting by timestamps.
2. Append, replicate, and acknowledge
A leader appends locally and replicates to enough followers; only after the configured commit condition does it acknowledge acceptance. Event IDs and producer sequence numbers support retry deduplication. Disk segments roll by size or time, and indexes help consumers locate offsets.
Make the acknowledgement explicit: it means the event is in a recoverable committed log, not that every consumer processed it or that a materialized read model is current. A consumer failure does not roll back an appended event.
3. Consumer checkpoints and idempotency
Each consumer group stores its own partition offsets. Process the event and update the read model before committing the checkpoint; a crash may repeat processing, so handlers must be idempotent by event ID or aggregate version. If the model and checkpoint need atomic coupling, write both in one transactional store or record the result in an outbox.
Keep the log when a consumer falls behind; do not skip unprocessed events just to reduce lag. A rebuildable view can start from a snapshot offset. An external side effect that cannot be rebuilt needs compensation or review rather than blind replay.
4. Snapshot protocol
A snapshot stores aggregate ID, serialized state, the last applied offset, aggregate and schema versions, checksum, and creation time. Generate it at a stable boundary: record the target offset, apply events through that offset, and write the snapshot with the same boundary. Recovery accepts only a validated snapshot whose offset belongs to the aggregate’s partition.
Recovery loads the snapshot and replays from snapshotOffset + 1. If the schema is old, run a versioned migration before publishing state; migration failure must block an invalid state. A snapshot is a cache: deleting it damages no log and only increases recovery time.
5. Separate retention, compaction, and archive
Time retention removes old log data and fits events with a defined replay window. Key compaction keeps the latest value for each key in a partition and lets a state consumer rebuild current state from a shorter log. It cannot support audit or arbitrary point-in-time replay because intermediate events may be gone.
A tombstone represents a deleted key. Keep it until every consumer covered by the contract can observe it, then allow compaction to remove it. Store audit events in an immutable archive with access control, encryption, and retention rules; a compacted state log is not complete audit evidence.
6. Schema, disorder, and poison events
Use backward-compatible schema rules: add optional fields, preserve old meanings, and let consumers ignore unknown fields. Breaking changes need a new schema version, dual-read period, or migration window. Record schema versions and parse failures; never silently commit the offset of an unparseable event.
Out-of-order events for one aggregate usually indicate a producer or replication violation. Reject an old aggregate version and hold a gap for repair. Cross-aggregate events need a business-time or causal-ID compensation rule; server arrival time is not a substitute for causality.
7. Capacity, recovery, and observability
Model events per second, average and tail payload size, replication factor, retention window, snapshot size, compaction savings, and replay speed. Shorter snapshot intervals improve recovery but add write amplification and storage. More aggressive compaction lowers state-rebuild cost but weakens audit and historical queries.
Monitor producer acknowledgement latency, replication lag, disk segments, compaction backlog, consumer lag, snapshot age, replay rate, schema errors, duplicate rate, and snapshot checksum differences. Periodically rebuild sampled state from snapshots and events and compare it with the materialized view; retain offsets and event samples when a mismatch appears.
High-quality sample answer
“I use aggregate ID as the partition key, append immutable events, and reject concurrent stale writes with an aggregate version. A producer acknowledgement means the event is in a replicated committed log; each consumer commits its own partition offset after applying the event, so a crash causes only deduplicable repeats.
Snapshots contain aggregate state, the last event offset, aggregate and schema versions, and a checksum. Recovery validates the snapshot and replays from the next offset; deleting a snapshot only slows recovery. Key compaction is limited to rebuildable current-state streams, with tombstones retained for the consumer contract. The audit stream remains an immutable archive.
I promise order only within one aggregate, not across partitions. The capacity model includes replication, retention, compaction, and snapshot write amplification. Metrics cover lag, snapshot age, compaction backlog, schema errors, and replay verification differences, with regular snapshot-plus-event rebuild checks.”
Common mistakes
- Treat snapshots as the event source → deleting one removes recovery and audit evidence → snapshots are offset-bound derived accelerators.
- Treat a compacted log as full history → intermediate events and point-in-time state are gone → archive audit events and compact only state streams.
- Sort all events by timestamp → clock skew creates false order → define order by aggregate version and partition contract.
- Commit offsets after arbitrary side effects → duplicates are inevitable but unspecified → make handlers idempotent and define model/checkpoint coupling.
- Delete tombstones immediately → a lagging consumer can resurrect a key → retain them through the consumer contract window.
- Skip an unknown schema and commit → the log and view silently diverge → pause, isolate the poison event, and retain evidence.
- Skip capacity modeling → snapshots, compaction, or replay become the outage bottleneck → calculate lag, recovery time, and write amplification.
- Claim exactly-once solves every duplicate → external side effects can still repeat → use idempotency keys, transactions, or compensation.
Follow-up questions and answers
Why not store only current state in a database?
If only current reads matter, a database is simpler. An event log provides replay, audit, multiple consumers, and reconstruction of different views, at the cost of schema, replay, capacity, and side-effect idempotency. Choose it for those requirements rather than assuming event sourcing is universally superior.
How can a new consumer rebuild from the beginning after compaction?
It can rebuild only the current state still represented after compaction; deleted intermediate history cannot be reconstructed. If history matters, keep an uncompacted archive or separate change log and document each topic’s replay capability.
What if new events arrive while a snapshot is being built?
Choose a stable target offset. Events after that offset continue appending but are not part of the snapshot; recovery loads the snapshot and replays from the next offset. If the snapshot write fails, keep the old snapshot and log rather than publishing a partial file.
How do you migrate event schemas?
Define compatibility and version fields first, then let consumers read old and new versions during a migration window. Writers add new fields and stop old fields only after all consumers upgrade. Breaking changes use a new event type or offline migration, validated by replay tests over historical events.