System Design Interview: How Would You Design a Real-Time Chat System?

Prompt and Applicable Context

Design a text chat service with 50 million daily active users, 5 million peak concurrent connections, and 2 billion messages per day. It supports direct conversations, groups of at most 200 members, multiple devices per user, offline catch-up, delivery and read receipts, presence, and typing indicators. Attachments, public channels, cross-region active-active writes, full-text search, and end-to-end encryption are follow-up scope.

The service acknowledges a send only after the message is durably stored. It must not silently lose an acknowledged message. For recipients who are already online under normal load, the p99 from durable acceptance to arrival at a connected device should be below one second. The transport may redeliver, so clients must see one logical message after deduplication. Ordering is required inside one conversation, not across unrelated conversations.

Three public system-design guides published or updated in 2026 present chat as a direct interview prompt and repeatedly examine persistent connections, connection routing, per-conversation ordering, offline synchronization, receipts, and presence. The WebSocket standard defines the bidirectional transport and control frames, while the Matrix client-server specification provides production-grade examples of client transaction IDs, incremental sync tokens, read receipts, and ephemeral typing events. Those sources establish that the topic and its failure boundaries are current and technically grounded; the scale and SLO in this prompt are fictional interview inputs.

What the Interviewer Evaluates

The first signal is contract precision. “Sent,” “delivered,” and “read” are different facts. A server acknowledgment means durable acceptance. A delivery receipt means a selected recipient device received the message. A read receipt requires an explicit product rule that the user actually displayed the conversation. Treating all three as one state creates false reliability claims and incorrect unread counts.

The second signal is whether the candidate avoids an end-to-end exactly-once promise. A client may time out after the server commits but before the acknowledgment arrives. A gateway may repeat a delivery after reconnect. The practical contract is at-least-once transport combined with a stable client message ID, a server idempotency record, and client deduplication by server message ID.

The third signal is the ordering boundary. A global order would serialize unrelated conversations. A useful product contract gives every accepted message a monotonic sequence inside its conversation. All writes for one conversation therefore reach one current owner or sequencer. This preserves local order while allowing different conversations to scale independently, and it exposes a real hot-conversation limit.

The fourth signal is separation of durable and ephemeral state. Messages, membership intervals, and read cursors survive failures. Presence and typing can expire and be dropped. Letting typing traffic share the durable message log wastes storage and lets a cosmetic burst delay real messages.

The final signal is recovery thinking. The answer should cover a lost send acknowledgment, gateway failure, reconnect storms, stale conversation owners, slow devices, membership changes, duplicate fan-out, and a hot group. Capacity numbers should be derived from the prompt and then calibrated by load tests rather than presented as universal server limits.

Questions to Clarify Before Answering

• What content and conversation types are in scope? This answer handles text, direct conversations, and groups up to 200. Attachments use object storage and metadata messages; public channels need a different fan-out and history strategy.
• What does each receipt mean? Server accepted, device received, and user read are separate monotonic facts. “Read” occurs only when the client displays the relevant conversation, not when it merely receives a push.
• What ordering is required? The prompt requires a stable order per conversation. It does not promise that a user's wall clock determines order or that two different conversations share one sequence.
• Can a user send from multiple devices? Yes. Each device has its own authenticated connection and sync cursor; sender retries reuse the same client message ID.
• What happens when membership changes? Authorization is checked at acceptance. The product must define whether a new member can read older history and what a removed member retains locally. This design stores membership intervals by conversation sequence.
• How long are messages and idempotency records retained? Message retention is a product and compliance decision. The idempotency mapping must cover the maximum client retry window; deleting it earlier can recreate duplicates.
• Which regions are used? The base answer assigns each conversation a home region and fails it over deliberately. Simultaneous writable replicas in multiple regions require a more complex conflict and ordering model.
• What is outside the one-second target? Offline push display, reconnect catch-up, user read time, and cross-region disaster recovery are measured separately from live delivery to an already connected device.

30-Second Answer Framework

“I would separate connection management from the durable message path. The client sends over an authenticated WebSocket with a stable client message ID. A conversation-routed service rechecks membership, assigns the next per-conversation sequence, atomically stores the message and idempotency result, and only then acknowledges it. An asynchronous fan-out worker looks up active recipient devices and sends the message to their gateways; offline or failed devices recover from a cursor-based sync API. Delivery and read cursors advance monotonically, while presence and typing use a separate expiring path. The transport is at least once, so both server and client deduplicate. At the stated scale I would plan for roughly 23,000 average and 230,000 peak sends per second, about 2 TB of logical message data per day at 1 KB each, and load-test persistent connections, hot groups, reconnect storms, retries, membership changes, and owner failover against the SLO.”

Step-by-Step Deep Dive

Step 1: Freeze the product contracts and calculate the first capacity envelope.

Two billion messages per day average about 23,148 sends per second. If the peak is ten times the average, ingress planning starts around 230,000 sends per second. At an assumed 1 KB per stored message including ordinary metadata and indexes, the logical write volume is about 2 TB per day; three replicas make about 6 TB per day before compaction, filesystem reserve, backups, and receipt writes. These are planning assumptions, not measured hardware limits.

Five million concurrent connections dominate gateway planning. Suppose a load test on the chosen instance, TLS setup, heartbeat interval, and target p99 proves 50,000 healthy connections per gateway. The bare minimum is 100 gateways. Operating at 70% of that measured limit requires about 143, rounded to 150, plus zone-failure reserve. The test must include messages, reconnects, and slow clients; an idle-socket count alone is misleading.

Step 2: Define the client protocol and its identities.

Every device opens an authenticated WebSocket to a regional gateway. The gateway validates origin where applicable, limits frame size and send rate, refreshes authorization, and uses ping/pong plus a lease to detect dead connections. RFC 6455 supplies connection mechanics; the application protocol still owns authentication, acknowledgments, sequencing, retries, and backpressure.

The central frames can be expressed as:

SEND {
  conversation_id, client_message_id, body, client_sent_at
}

ACK {
  client_message_id, message_id, conversation_seq, accepted_at
}

MESSAGE {
  conversation_id, message_id, conversation_seq, sender_id, body, accepted_at
}

SYNC {
  device_sync_cursor, limit
}

client_message_id is generated once and reused for every retry from that sending device. message_id is the server identity used for recipient deduplication. conversation_seq is the display and catch-up order inside one conversation. accepted_at is server time for diagnostics, not the ordering authority.

Step 3: Commit once per logical send before acknowledging.

The gateway forwards SEND to the chat service. The service obtains the current conversation shard, authenticates the sender, verifies membership at the current membership version, validates size, and applies per-user and per-conversation limits. The shard owner serializes accepted writes for that conversation.

In one durable transaction, it allocates the next sequence, writes the message, and writes a unique mapping such as (sender_device_id, client_message_id) → message_id. A duplicate request returns the prior result instead of appending another message. Only a quorum-committed record receives ACK. If the acknowledgment is lost, retry is safe. If storage fails before commit, no acknowledgment is sent.

Messages are partitioned by conversation_id and sorted by conversation_seq. Membership changes are also ordered, or reference effective sequence boundaries, so authorization does not depend on an eventually consistent current-member cache alone. A fenced owner epoch prevents an old owner from accepting writes after failover.

Step 4: Fan out after the durable commit.

The committed log emits a delivery task. A connection directory maps (user_id, device_id) to a gateway and connection lease. For a direct conversation or a group of at most 200, the fan-out worker loads the membership snapshot effective at the message sequence, groups online devices by gateway, and sends batched gateway commands. The body is stored once; fan-out carries a reference or compact event instead of copying a durable body per recipient.

The gateway places the event in each connection's bounded outbound queue. A device returns a monotonic delivery cursor after it has accepted the event locally. If a connection is slow, the gateway stops buffering without limit, marks it for resync, and closes it with an application reason. The durable log remains the truth, so dropping an in-memory live delivery does not lose the message.

For an offline user, the system may send a privacy-safe mobile push hint. Push is a wake-up mechanism, not message storage or proof of delivery. On open, the client authenticates and syncs from the durable service.

Step 5: Make reconnect and multi-device sync explicit.

Each device persists an opaque device_sync_cursor. GET /sync?after=cursor or the equivalent frame returns ordered conversation deltas, membership changes, receipt deltas, and a new cursor. The server must not advance the cursor beyond events omitted from the response. If a cursor is expired or the gap is too large, the server returns a bounded snapshot plus continuation tokens instead of attempting an unbounded replay on one connection.

Live delivery and sync can overlap, so the client merges by message_id and orders by (conversation_id, conversation_seq). A new device receives policy-allowed history and establishes its own cursor. Read state is user-level and monotonic per conversation; delivery state may remain per device. Aggregation rules should say whether “delivered” means any device or every active device.

Step 6: Keep receipts, presence, and typing honest.

A read update carries the highest displayed conversation_seq; the server uses a conditional maximum so late requests cannot move the cursor backward. The Matrix specification's receipt model illustrates why read is a delta that replaces an older position and why mere receipt is insufficient evidence that a user saw content.

Presence and typing follow a different path. Gateways refresh a short presence lease. Typing events are authorized, rate-limited, scoped to one conversation, coalesced, and expired after a few seconds. They may be dropped during overload and never enter the durable message log. “Last seen” needs an explicit privacy policy and coarse updates to avoid turning heartbeat traffic into a write storm.

Step 7: Design the failure paths before claiming reliability.

• Lost ACK: the client retries the same client_message_id; the server returns the stored result.
• Gateway crash: devices reconnect with jitter and resume from cursors; the connection lease expires.
• Fan-out worker crash: the durable delivery task is retried; gateways and clients deduplicate.
• Conversation owner crash: a new epoch and quorum state elect a replacement; stale owners are fenced.
• Slow device: bounded queues trigger resync rather than consuming unbounded memory.
• Hot conversation: one sequencer limits write rate. Batch commits, isolate the hot shard, and apply a product rate limit; adding ordinary hash partitions cannot parallelize one strict sequence.
• Membership race: sequence membership changes with messages and authorize against the effective interval.
• Regional outage: route the conversation to a promoted replica only after fencing the old home region; recovery time and possible unacknowledged loss are separate from the no-loss guarantee for acknowledged messages.

Reconnect attempts use jittered exponential backoff and admission control. Otherwise a regional gateway restart can turn five million healthy clients into a handshake storm that prevents recovery.

Step 8: Prove the invariants with layered tests and observability.

Property tests generate retries, reordered deliveries, membership changes, and owner epochs, then assert one logical message per client ID, unique conversation sequences, monotonic cursors, and no unauthorized message after a removal boundary. Integration tests crash the process before commit, after commit but before acknowledgment, and during fan-out. Chaos tests remove a gateway, worker, shard owner, zone, and connection-directory partition.

Load tests model five million long-lived connections, 230,000 peak sends per second, group fan-out, slow recipients, and mass reconnects. Observe acceptance latency, live-delivery latency, sync lag, duplicate rate before and after client deduplication, sequence gaps, hot-shard saturation, outbound queue bytes, reconnect admission, and unauthorized-read denials. A canary rollout expands only while the durability and authorization invariants remain clean.

High-Quality Sample Answer

“I would first define three different outcomes: durable server acceptance, device delivery, and user read. The service acknowledges only after a quorum has stored the message. Network delivery remains at least once, so a sender-generated client message ID makes retries idempotent and recipients deduplicate by server message ID.

Clients connect to regional WebSocket gateways. A connection directory tells fan-out workers which gateway holds each user device, but gateways do not own history. A chat service routes every write by conversation ID to the current fenced owner. That owner rechecks membership, allocates a per-conversation sequence, and atomically stores the message plus the idempotency result. Unrelated conversations execute in parallel; one very hot conversation remains a deliberate serial bottleneck.

After commit, workers fan out to online devices. Failed or offline delivery is repaired through a cursor-based sync API. Live and replay paths may overlap, so message IDs absorb duplicates and conversation sequences restore local order. Read cursors advance by maximum sequence. Presence and typing use expiring, best-effort state and cannot block durable messages.

At 2 billion messages per day, average ingress is about 23,000 per second; I would plan around 230,000 at a ten-times peak. At the prompt's 1 KB planning assumption, logical message storage is roughly 2 TB per day and 6 TB with three replicas. For five million connections, a measured 50,000-connection gateway limit implies 100 bare nodes or about 150 at 70% utilization before zone reserve.

I would validate lost acknowledgments, gateway and owner failures, stale epochs, duplicate fan-out, slow devices, membership races, a hot group, and a reconnect storm. The release gate is no loss of acknowledged messages, no duplicate logical message after deduplication, no sequence regression, no access outside membership intervals, and the stated live-delivery p99 under representative load.”

Common Mistakes

• Mistake: Call server acceptance, delivery, and read one status → Consequence: Reliability and unread metrics become unverifiable → Fix: Define separate monotonic states and owners.
• Mistake: Promise exactly-once delivery over the network → Consequence: A lost acknowledgment or reconnect produces an unexplained duplicate → Fix: Use at-least-once transport with stable send IDs and deduplication.
• Mistake: Order all messages globally → Consequence: Unrelated conversations share one bottleneck → Fix: Assign a sequence only within each conversation.
• Mistake: Acknowledge before durable commit → Consequence: A process or zone failure can silently erase an accepted message → Fix: Ack only a quorum-committed record.
• Mistake: Store history on WebSocket gateways → Consequence: Connection movement becomes data movement and gateway loss threatens history → Fix: Keep gateways stateless beyond bounded connection state.
• Mistake: Buffer endlessly for a slow device → Consequence: One client can exhaust gateway memory → Fix: Bound the queue, disconnect, and resync from durable storage.
• Mistake: Put presence and typing in the durable log → Consequence: Expiring cosmetic traffic increases cost and can delay messages → Fix: Use an authorized, rate-limited, best-effort TTL path.
• Mistake: Authorize only from a stale current-member cache → Consequence: Removed users may send or receive across a race → Fix: Order membership boundaries and fence acceptance against them.
• Mistake: Size gateways by idle socket count → Consequence: TLS, heartbeats, fan-out, and reconnects break the plan → Fix: Benchmark the full workload at the target p99 and reserve failure headroom.

Follow-Up Questions and Responses

Follow-up 1: How would you add end-to-end encryption?

Encrypt message bodies on sender devices and store only ciphertext plus required routing metadata. Every device needs an identity key, signed device list, and conversation key distribution; adding or removing a device or member rotates or redistributes keys according to policy. The server can still sequence and route ciphertext, but server-side search, content moderation, recovery, previews, and abuse handling become constrained. Delivery metadata, participant sets, timing, and sizes may remain observable, so encryption does not eliminate metadata privacy work.

Follow-up 2: Can one hot group be split across many sequencers?

Not while preserving one strict contiguous sequence without another ordering authority. Hashing the conversation to more ordinary shards still leaves a merge or consensus point. First batch commits and isolate the shard, then enforce per-conversation limits. If the product can accept partial order, partition by thread or sender and expose causal relationships, but that changes the contract and client complexity.

Follow-up 3: The sender sees an ACK, but a recipient stays offline for a week. Was the message delivered?

It was accepted, not delivered to that device. The durable message remains available under retention policy, and a push hint may wake the device. On reconnect the device syncs from its cursor and then reports delivery. Product UI and metrics should keep server-accepted, any-device-delivered, all-active-devices-delivered, and read separate.

Follow-up 4: How do edits and deletes interact with ordering?

Represent them as new ordered events referencing the original message rather than mutating history invisibly. Clients fold the event stream into a current view. Authorization is checked when the edit or delete is accepted, and policy defines time limits and whether deletion removes content, marks a tombstone, or triggers asynchronous erasure from secondary stores.

Follow-up 5: What if a removed member reconnects with an old cursor?

The sync service evaluates the user's membership intervals, not merely possession of a cursor. It returns only events authorized for that user and includes the membership removal boundary. A cursor is an opaque position, not a capability. Cached attachments and local copies require separate revocation and retention expectations because the server cannot erase data already saved on a device.

Follow-up 6: How would you support very large public channels?

The group-of-200 fan-out-on-write design no longer fits. Store the channel log once, let followers pull by cursor, fan out only compact unread or push hints, cache popular segments, and relax individualized delivery receipts. Partitioning, moderation, discovery, and celebrity-channel hotspots become first-class problems, so this should be treated as a different workload rather than a larger numeric value in the same design.