Kafka Clone

TL;DR

• Model the system as an append-only log split into ordered partitions. Producers append records; consumers fetch by offset and commit their own progress.
• Use producer batching, sequential segment writes, the OS page cache, and a low-copy fetch path to make the common case efficient. Measure the actual hardware rather than relying on fixed throughput claims.
• Replicate each partition with a leader and followers. acks=all plus a minimum in-sync replica policy makes acknowledged writes survive the configured broker failure model.
• Hash a record key to preserve per-key ordering, and use partitions as the unit of broker distribution and consumer-group parallelism.
• Store consumer offsets in a replicated, compacted internal topic. Keep delivery at least once and make consumers idempotent; exactly-once application effects require a separate protocol.

Scope and assumptions

This article designs a Kafka-like distributed message streaming platform: producers publish records to named topics, consumers read them at their own pace, records persist in partition logs, and consumer groups commit offsets. It covers append-only storage, batching, partitions, replication, leader election, offset state, and the choice between a log and a traditional task queue.

The interview scenario uses these illustrative assumptions:

• A high-volume target of 10 million messages per second across the cluster and about 1 trillion messages per day as a planning scenario. Record size, retention, partition count, and broker capacity must be derived and load-tested.
• Producer-to-consumer latency target below 10ms p99 under normal load, with durability defined against a single broker failure and the configured replication policy.
• Per-partition ordering and per-key ordering are required; a global order across partitions is not promised.
• At-least-once delivery is in scope. Exactly-once producer and transactional-consumer semantics, cross-datacenter replication, schema registry, stream processing, and tiered storage are extensions.
• The rates, latencies, broker counts, segment sizes, and configuration values below are illustrative design inputs, not facts about any named company or product guarantee.

Functional Requirements

Core Requirements

1. Producers publish messages to named topics.
2. Consumers subscribe and read messages from a topic with persistent offsets.
3. Messages are durable and survive broker restarts.
4. Support at-least-once delivery.

Scope Exclusions

• Stream processing (Flink, Spark Streaming) beyond basic consumer logic.
• Schema registry or message transformation.

Non-Functional Requirements

Core Requirements

1. Write throughput: Illustrative target of at least 10 million messages per second across all topics.
2. Latency: Illustrative target of producer-to-consumer latency under 10ms p99 at normal load.
3. Durability: No acknowledged data loss for the stated single-broker-restart failure model, with replication factor 2 or more and an explicit acknowledgement policy.
4. Availability: Illustrative target of 99.99% uptime. The cluster should continue serving reads and writes during a single broker failure when the remaining ISR and capacity permit it.
5. Scale: Illustrative high-volume scenario of roughly 1 trillion messages per day distributed across many topics; record size and retention determine the actual storage and broker count.

Below the Line

• Exactly-once delivery semantics (builds on the at-least-once foundation we will design)
• Cross-datacenter replication (MirrorMaker / active-active topology)
• Built-in schema evolution (Confluent Schema Registry)
• Tiered storage to object stores (Kafka 3.x feature)

The hardest engineering problem in scope: Balancing the illustrative 10-million-message-per-second target, sub-10ms normal-load latency, and the configured durability policy. Sequential appends are generally more efficient than random indexed writes, so the write path is shaped around batching and append-only segments. Exact throughput depends on record size, hardware, replication, flush settings, and network capacity.

Exactly-once semantics is below the line because it requires the idempotent producer protocol and transactional APIs, which form a separate abstraction layer on top of at-least-once. A later extension could enable enable.idempotence=true on the producer and wrap produce plus offset commit in a transaction, while still defining the boundary of the transaction and the downstream side effects.

Cross-datacenter replication is below the line because it requires active-passive or active-active topology decisions, lag monitoring, and a conflict policy. A later extension could add a replication pipeline between clusters and route producers to a chosen home or nearest region.

30-second answer

Use topics split into ordered partitions. Producers batch records and append them to a partition leader; followers replicate the log, and the leader acknowledges according to the configured ISR policy. Consumers fetch by offset, commit progress per consumer group, and can replay records independently. Key-based partitioning preserves per-key order, while partition count controls parallelism. Keep delivery at least once and make consumers idempotent; add transactions only when the application needs a defined exactly-once boundary.

5-minute explanation

The write path is an append to the active segment of a partition. Batching reduces syscall and network overhead; the OS page cache makes sequential writes and reads efficient; segment indexes map offsets to file positions. Consumers fetch ranges by offset, so one record can be read by many independent groups without deleting it for others.

Durability comes from replicated partition logs. A leader accepts writes, followers copy them, and the in-sync replica set records which copies are caught up. With acks=all and a minimum ISR policy, the producer acknowledgment reflects the configured failure model. A controller quorum changes partition leadership when a broker fails; unclean election is a deliberate data-loss trade-off and should not be used where acknowledged data must be retained.

Partitions distribute leaders, disk, network, and consumer work. Hashing a key keeps related records together and preserves order for that key, but no global ordering exists across partitions. Consumer groups own one partition at a time, and their committed offsets live in a replicated compacted internal topic so each group can resume or replay independently.

The core entities, client operations, and concrete flows below show those responsibilities before the deep dives compare storage, partitioning, replication, offset, and queue choices.

Core entities

• Topic: A named, logical stream of records. Topics are append-only and hold records indefinitely subject to configurable retention.
• Partition: An ordered, immutable append-only log. The unit of parallelism and distribution across brokers.
• Segment: A physical file on disk within a partition. Kafka writes to the active segment and rolls to a new one at a configurable size or time boundary.
• Record (Message): The fundamental unit: a key, a value, a timestamp, and an offset. The key drives partition assignment; the offset is the record's position within its partition.
• ConsumerGroup: A named set of consumers sharing a single logical read position per topic. Each partition is assigned to exactly one consumer within the group at any time.
• Offset: The monotonically increasing position of a record within a partition. Consumers commit their current offset to checkpoint read progress and resume after restarts.

Schema and serialization format are deferred to the deep dives. These six entities are sufficient to drive the API design and high-level architecture.

API design

Kafka exposes an SDK-style client API rather than REST. The naive single-producer, single-consumer shape is useful as a baseline; the partitioned and keyed evolution below adds the durability and throughput controls needed for the illustrative workload.

Produce messages (naive shape, no partitioning):

// Simple produce: one broker, one partition, fire and forget
producer.produce(topic="events", value="payload")

This is the starting point. There is no partition key, acknowledgement mode, or batching. It is useful as a development baseline, but it does not establish the durability or throughput properties required by the illustrative workload.

Produce messages (evolved shape, keyed with acks):

// Keyed produce: consistent partition assignment by key hash
producer.produce(
  topic="events",
  key="user_id:12345",   // murmurhash(key) % num_partitions = target partition
  value="payload",
  acks="all"             // wait for all ISR replicas to acknowledge before returning
)

The key determines which partition receives the message. All messages with the same key land on the same partition, preserving ordering per key. acks=all waits for the leader and all in-sync replicas to acknowledge before returning to the caller.

Subscribe and poll (consumer side):

// Register consumer in a named group; broker assigns partitions
consumer.subscribe(topics=["events"], group_id="analytics-workers")

// Poll loop: fetch records from last committed offset
while running:
  records = consumer.poll(timeout_ms=500)
  for record in records:
    process(record)
  consumer.commit(offsets=current_offsets)  // checkpoint after successful processing

subscribe() registers the consumer in a consumer group. The broker group coordinator assigns partitions to each consumer in the group. commit() advances the checkpoint so a restart resumes from the correct position rather than replaying from the beginning.

Admin API (topic management):

// Partition count controls max consumer parallelism; set it right at creation
admin.create_topic(
  name="events",
  num_partitions=64,
  replication_factor=3
)

Partition count is chosen at topic creation and bounds consumer-group parallelism. A replication factor of 3 is a common illustrative choice for a three-copy failure model, but the right value depends on broker count, storage, network, and the tolerated failure domain.

45-minute interview approach

Use this section only as the pacing plan for a Kafka-like streaming-system design prompt; keep the storage and protocol mechanics in the architecture and deep dives.

• 0-5 minutes — clarify the contract: Confirm record size, retention, ordering scope, consumer replay needs, delivery semantics, and the broker failure model. Separate a log from a consume-once task queue.
• 5-10 minutes — requirements and estimates: State the illustrative message rate, daily volume, latency target, replication factor, retention, and number of topics or consumer groups. Recalculate storage and network from record size.
• 10-15 minutes — entities and APIs: Identify Topic, Partition, Segment, Record, ConsumerGroup, and Offset. Sketch produce, subscribe/poll, commit, and topic-management operations.
• 15-25 minutes — baseline architecture and flows: Draw a single partition append, then evolve to replicated brokers, keyed partitioning, consumer groups, and offset commits. Walk through both produce and fetch paths.
• 25-35 minutes — choose deep dives: Let the interviewer select sequential I/O, partition scaling, fault tolerance, offsets, or Kafka-versus-queue trade-offs. Compare the naive option with the selected design.
• 35-41 minutes — reliability, security, and operations: Cover ISR lag, controller failure, consumer lag, producer retries, idempotent consumers, ACLs, encryption, retention, disk pressure, and recovery testing.
• 41-45 minutes — trade-offs and close: Explain per-key ordering versus parallelism, acknowledgement versus availability, replay versus deletion, and what changes for transactions or cross-datacenter replication.

High-level architecture and critical flows

The system has four critical flows: produce appends records to a partition leader, replicate copies the log to followers, consume fetches ranges by offset, and commit stores each group's progress. Controller metadata and group membership coordinate ownership but are not substitutes for the partition log.

1. Producers publish messages to named topics

The write path at its simplest: a producer connects to a broker, the broker appends the message to a partition file on disk, and returns an acknowledgment.

Components:

• Producer Client: The application sending records. Handles serialization, batching, and retry.
• Broker: The single server that owns the partition. Appends each incoming record to the active segment file on disk.
• Partition Log: An append-only file. Records are never modified; they are only appended and eventually deleted by the retention policy.

Request walkthrough:

1. Producer calls produce(topic="events", value="payload").
2. The broker receives the produce request and deserializes the record.
3. The broker appends the serialized record to the active segment file for partition 0.
4. The broker assigns the next integer offset and writes it to the segment index.
5. The broker acknowledges success back to the producer.

This is the write path only: one producer, one broker, one partition file. There is no fault tolerance and no parallelism here; both are added in later requirements and deep dives. Starting with this single-partition sketch makes the later replication and partitioning decisions easier to reason about.

2. Consumers subscribe and read with persistent offsets

Consumers do not move messages off the broker. They track their own position (offset) in each partition and fetch records at their own pace, which means the same data can be read by multiple independent consumer groups without any coordination between them.

Components:

• Consumer Client: Calls poll() in a loop. Fetches records from a specific partition starting at the last committed offset.
• Offset Store: The broker records the last committed offset per consumer group and partition. On restart, the consumer resumes from this stored checkpoint.
• Group Coordinator: A designated broker partition that tracks which consumers are alive and which partitions each consumer owns.

Request walkthrough:

1. Consumer calls subscribe(topics=["events"], group_id="my-group").
2. The group coordinator assigns partition 0 to this consumer.
3. Consumer calls poll(). The broker returns records starting at the consumer's last committed offset.
4. Consumer processes records, then calls commit(offsets) to checkpoint progress.
5. On restart, the consumer fetches its last committed offset from the broker and resumes from that position.

Adding offset tracking separates the log from a simple file tail. Multiple consumer groups can read the same topic independently, consumers can replay from an offset, and restarts resume from a durable checkpoint. The offset model is also the key distinction from a consume-once queue.

3. Messages survive broker restarts (durability)

A file on a single broker is not durable. To survive restarts, records must be replicated: the leader broker writes each record, and follower brokers replicate it before the producer receives an acknowledgment.

Components: