WhatsApp

What is WhatsApp?

WhatsApp is an end-to-end encrypted messaging app where users exchange text messages in real time, in 1-to-1 conversations and groups of up to 1,024 people. At 2 billion registered users sending 100 billion messages per day, the interesting engineering challenge is not storing the messages. It is routing each one across a fleet of thousands of stateful servers to find the right WebSocket connection, guaranteeing delivery to devices that may be offline for days, and fanning out a single group message to 1,024 members without the sender waiting for all of them.

This makes it a rich interview question because it tests real-time connection management, distributed routing without a lookup table, delivery guarantees under failure, and large-scale fan-out. A strong interview opening frames it as a routing problem, not a storage problem, because that is where the interesting constraints live.

TL;DR

Persist each text message before attempting delivery. Keep WebSocket connections on stateful Chat Servers, route online recipients through Redis Pub/Sub channels keyed by user, use Cassandra as durable message history, and recover offline recipients from a bounded per-user queue plus reconnect history.

Handle group chats with a hybrid fan-out path: inline publishes for small groups, Kafka and parallel Fan-out Workers for large groups. Client-generated client_msg_id makes retries idempotent; delivery and read receipts are monotonic status updates rather than a second message path.

Scope and assumptions

The following are illustrative interview planning assumptions, not a specification of the named product or a guarantee from any vendor:

• About 2 billion registered users, 500 million daily active users, and 100 billion text messages per day: roughly 1.16 million messages per second on average and about 3.5 million per second at peak.
• The core path models 1-to-1 chats and groups of up to 1,024 members. Multi-device synchronization, media, calls, and encryption-key management are separate extensions.
• Online delivery targets sub-200ms p99 when both users are connected; offline messages should become available within about 5 seconds of reconnecting.
• Delivery is at least once. Clients deduplicate with client_msg_id or message_id, and Cassandra remains the durable source of truth when the fast routing path is unavailable.
• Redis Pub/Sub, Redis offline lists, Kafka, Cassandra, and push providers are illustrative choices whose limits depend on topology, payload size, and capacity testing.

Functional Requirements

Core Requirements

1. Users can send and receive 1-to-1 text messages in real time.
2. Users can see delivery receipts: sent (single tick), delivered to device (double tick), and read (blue ticks).
3. Users who are offline receive messages when they reconnect.
4. Users can participate in group chats with up to 1,024 members.

Below the Line (out of scope)

• Media file sharing (images, video, voice notes, documents)
• End-to-end encryption key management (Signal Protocol)
• Voice and video calls
• Status and Stories (ephemeral broadcast)

The hardest part in scope: Routing messages across a fleet of thousands of stateful Chat Servers. Each server holds WebSocket connections for a fraction of active users. When a sender is on Server A and their recipient is on Server B, the message must cross servers in under 200ms without any server scanning the full fleet or maintaining a global connection table.

Media file sharing is below the line because it uses the same routing path but introduces a separate binary storage and CDN delivery tier. An extension can use a pre-signed S3 URL pattern: the client uploads the raw file directly to S3, then sends a message containing the S3 object key. The receiving client fetches the media from a CDN edge node, not from the Chat Server.

End-to-end encryption is below the line because it is a property of the message payload, not the routing layer. WhatsApp uses the Signal Protocol: keys live entirely on the devices and the server stores and forwards ciphertext it cannot read. Adding E2EE does not change the routing, storage, or delivery architecture described in this article.

Voice and video calls are below the line because they require a separate WebRTC signaling layer and media relay infrastructure. The signaling messages flow through the existing WebSocket path, but media streams bypass routing entirely via TURN/STUN servers.

Status and Stories are below the line because they introduce a separate ephemeral broadcast model with a 24-hour TTL. An extension can store story metadata in Redis with a 24-hour TTL and fan out to followers using the same Kafka-based fan-out infrastructure described in the group messaging deep dive.

Non-Functional Requirements

Core Requirements

• Availability: 99.99% uptime. Availability over consistency: a message delivered slightly out of order is acceptable; a dropped connection or lost message is not.
• Latency: End-to-end message delivery under 200ms p99 when both users are online. Offline queue delivery within 5 seconds of reconnection.
• Delivery guarantee: At-least-once delivery to device. The client deduplicates using a client-assigned client_msg_id to prevent displaying the same message twice.
• Scale: 2B registered users, 500M DAU. 100B messages per day at baseline (~1.16M messages per second, peaking at ~3.5M per second).
• Group fan-out: Groups up to 1,024 members. At average group participation of 30 members per message, fan-out generates roughly 35M per-member delivery events per second across the fleet.

Below the Line

• Sub-50ms delivery for same-region users via edge routing
• Multi-device message sync and session ordering

Read/write ratio: Roughly 1:1 at the message level (every send creates one receive). Group fan-out is the amplifier: a single message to a 1,024-member group triggers 1,024 delivery writes. That fan-out multiplier, not raw message throughput, determines the size of the async pipeline.

A 200ms p99 target for online delivery leaves roughly 150ms for storage writes, routing, and delivery if a nearby WebSocket round-trip takes 20-50ms.

Synchronous fan-out for large groups does not fit this budget. Any delivery path that fans out beyond a single routing hop must go through an async message queue.

30-second answer / outline

1. Terminate authenticated WebSocket connections on a fleet of stateful Chat Servers and use Redis Pub/Sub user channels for cross-server routing.
2. Generate a server message_id, deduplicate client_msg_id, and write the message to Cassandra before publishing the delivery event.
3. Treat a zero-subscriber publish as an offline signal, enqueue the message ID, send a push notification, and replay from Cassandra on reconnect.
4. Fan out small groups inline, but publish large-group events to Kafka and let parallel workers deliver to online channels or offline queues.
5. Make receipts, retries, reconnects, and fan-out idempotent; monitor delivery latency, queue age, connection health, and replay gaps.

5-minute explanation

Start with the routing split. A Chat Server owns only the WebSocket connections currently attached to it, so a sender cannot assume that the recipient is local. Publishing to user:{recipient_id} lets the server holding that subscription deliver the message without a global connection lookup table.

The message is durable before the fast path runs. Cassandra stores the content and the client deduplication key, Redis Pub/Sub handles online delivery, and a zero-subscriber result triggers the offline queue and push notification path. Reconnects fetch queued IDs and can fall back to a history cursor, so a Redis delivery failure does not erase the message.

Group messages change the fan-out shape. Small groups can use bounded inline publishes, while large groups write one durable Kafka event and let Fan-out Workers process members in parallel batches. Delivery receipts travel back through the same user channel and advance status monotonically from sent to delivered to read.

The remaining design work is ordering, duplicate suppression, group membership consistency, connection recovery, Cassandra partitioning, abuse controls, and behavior when Redis, Kafka, push providers, or a Chat Server fails.

45-minute interview approach

This is a time-boxed plan for answering the design question in an interview, not a claim that the article should be read in 45 minutes.

• 0–5 minutes — Clarify the contract: Confirm 1-to-1 versus group scope, online/offline behavior, receipts, ordering, maximum group size, multi-device sync, media, encryption, and regional requirements.
• 5–10 minutes — Establish scale: Use the illustrative user, message, fan-out, connection, latency, and retention assumptions. Separate message writes from per-member delivery events and persistent connections.
• 10–15 minutes — Define APIs and invariants: Walk through WebSocket frames, reconnect history, group membership, client_msg_id, and the rule that a retry must not create a second message.
• 15–22 minutes — Draw the online and offline paths: Show the gateway, Chat Servers, Cassandra, Redis Pub/Sub, offline queue, push provider, and the reconnect cursor. State which components are durable.
• 22–30 minutes — Deep dive on routing and ordering: Compare broadcast, a connection table, and per-user Pub/Sub channels; then explain Snowflake IDs, client deduplication, receipts, and reconnect races.
• 30–35 minutes — Deep dive on group fan-out: Compare synchronous and asynchronous fan-out, choose a measured threshold, partition Kafka by group, and bound worker batches and retries.
• 35–41 minutes — Reliability, security, and operations: Cover Pub/Sub loss, Cassandra replay, Redis queue limits, WebSocket reconnects, authentication, privacy, abuse limits, and metrics.
• 41–45 minutes — Trade-offs and close: Compare WebSocket with polling, Cassandra with relational storage, and push with pull recovery. Recap the routing, durability, and fan-out planes and invite follow-ups.

Core Entities

• Message: The core unit. Carries a server-assigned message_id, client-assigned client_msg_id (for deduplication), chat_id, sender_id, content, created_at, and a mutable status field (sent, delivered, read).
• User: Account with a user_id, a device push token for offline notification delivery, and a presence timestamp for last-seen.
• Chat: A 1-to-1 or group conversation identified by a chat_id. For 1-to-1 chats, chat_id is derived deterministically from the two participant IDs. For groups, chat_id maps to Group metadata.
• Group: Group metadata including group_id, name, and a member list. The member list is the input to every fan-out operation in the system.

Schema details, indexes, and Cassandra partition key design are deferred to deep dive 4. The four entities above are sufficient to drive the API design and High-Level Design.

API Design

WhatsApp uses a persistent WebSocket connection for real-time delivery. REST endpoints handle connection setup, offline message fetching, and group management.

Establish a WebSocket connection:

GET /ws
Upgrade: websocket
Headers: Authorization: Bearer {token}

Send a message (WebSocket frame):

// Client → Server
{ type: "send_message", chat_id, content, client_msg_id }

// Server → Sender (acknowledgment)
{ type: "ack_sent", message_id, client_msg_id }

// Server → Recipient Client
{ type: "new_message", message_id, chat_id, sender_id, content, created_at }

Acknowledge delivery and read (WebSocket frames):

// Recipient Client → Server (auto-sent on message receipt)
{ type: "ack_delivered", message_id }

// Client → Server (sent when user views the conversation)
{ type: "ack_read", chat_id, last_read_message_id }

// Server → Original Sender (status propagation)
{ type: "status_update", message_id, status: "delivered" | "read" }

Fetch offline messages (REST, on reconnect):

GET /chats/{chat_id}/messages?after={cursor}&limit=50
Response: { messages: [...], next_cursor }

Create a group:

POST /groups
Body: { name, member_ids }
Response: { group_id, chat_id }

WebSocket over HTTP polling: Long polling holds one request thread per client while it waits for a message. WebSocket maintains a persistent TCP connection at a fraction of the resource cost. At 500M DAU with many users actively chatting, HTTP polling is not viable. WebSocket is the only practical choice for bidirectional real-time delivery at this scale.

client_msg_id: Each client generates a unique client_msg_id before sending. The server stores it alongside the server-assigned message_id. If the client sends a message, loses the ack_sent, and retries the send, the server recognises the duplicate client_msg_id and returns the existing message_id rather than inserting a second row.

High-Level Design

1. Users can send and receive 1-to-1 messages in real time

The write and deliver path: the client sends a message, the server stores it in Cassandra, and routes it via Redis Pub/Sub to whichever Chat Server holds the recipient's WebSocket.

The core architecture is a fleet of stateful Chat Servers. Each server maintains WebSocket connections for a slice of online users. Routing between servers happens through Redis Pub/Sub: each Chat Server subscribes to channels keyed by the user_ids of its connected clients.

Components:

• Client: Mobile or web app holding a persistent WebSocket connection to a Chat Server.
• Chat Server: Stateful service managing WebSocket connections. Writes messages to the Message DB, then routes delivery via Redis Pub/Sub.
• Redis Pub/Sub: Each Chat Server subscribes to user:{user_id} for every connected client. Publishing to user:{recipient_id} delivers to whichever server holds that connection, without any lookup table.
• Message DB (Cassandra): Durable message store partitioned by chat_id. Source of truth for all message content and status.

Request walkthrough:

1. Sender sends { type: "send_message", chat_id, content, client_msg_id } over WebSocket.
2. Chat Server A generates a Snowflake message_id and inserts { message_id, chat_id, sender_id, content, status: "sent" } into Cassandra.
3. Chat Server A publishes { message_id, chat_id, content } to Redis channel user:{recipient_id}.
4. Chat Server B (subscribed to user:{recipient_id}) receives the event and pushes new_message to the recipient via WebSocket.
5. Chat Server A returns ack_sent to the sender.

Chat Server A returns ack_sent to the sender once Cassandra commits and Redis acknowledges the publish. The Redis Pub/Sub publish returns the subscriber count; zero means the recipient is offline. The offline delivery path fires on that signal, covered in the next requirement.

Write to Cassandra before publishing to Redis Pub/Sub. Reversing that order could deliver a message that is never persisted, leaving nothing to replay on reconnect.

2. Users receive messages when offline

The offline path: when Redis Pub/Sub returns zero subscribers, the message_id is queued in Redis, a push notification wakes the device, and the full message is delivered on reconnect.

When the recipient is not connected to any Chat Server, the Redis channel has no subscriber. The message must persist durably until the recipient reconnects, then be delivered reliably without polling.

Components: