How Kafka delivers exactly-once

The Problem Statement

Interviewer: "Your payment processing pipeline reads order events from Kafka, charges the customer, and writes a confirmation back to Kafka. How do you guarantee each order is processed exactly once, even if the consumer crashes mid-processing or the producer retries a failed send?"

This question tests three things: whether you understand the delivery semantics spectrum (at-most-once, at-least-once, exactly-once), whether you know the specific Kafka mechanisms that enable exactly-once (idempotent producers, transactions, epoch fencing), and whether you understand the boundary of what exactly-once covers and what it does not.

Most candidates either say "Kafka supports exactly-once" without explaining how, or they confuse idempotent consumers (an application-level pattern) with Kafka's built-in exactly-once semantics (a broker-level mechanism). The interviewer wants to see the internal mechanics: producer IDs, sequence numbers, the Transaction Coordinator, epoch fencing, and the __transaction_state topic.

The reason this matters: in payment processing, inventory management, and financial ledgers, a duplicate message means a double charge, a phantom stock reduction, or a corrupted account balance. At-least-once with idempotent consumers works for most use cases, but understanding the Kafka-native exactly-once path shows deep distributed systems knowledge.

Clarifying the Scenario

You: "Let me make sure I scope this correctly."

You: "When you say 'exactly once,' are we talking about the Kafka-internal guarantee (a message is written to a topic partition exactly once), or end-to-end exactly-once that includes external side effects like charging a credit card?"

Interviewer: "Both. Start with the Kafka-internal mechanism, then tell me where it stops and what you need for end-to-end."

You: "Got it. And should I cover both idempotent producers (single-partition dedup) and Kafka transactions (multi-partition atomic writes), or just transactions?"

Interviewer: "Walk me through both. I want to see how they build on each other."

You: "One more thing: should I cover Kafka Streams' exactly-once processing, or focus on the producer/consumer APIs?"

Interviewer: "Cover the consume-transform-produce pattern. That is where the real complexity lives."

You: "OK. I will structure my answer in four parts: the delivery semantics spectrum, idempotent producers as the foundation, Kafka transactions for multi-partition atomicity, and the consume-transform-produce pattern that ties them together for stream processing."

My Approach

I break this into five parts:

1. The delivery semantics spectrum: Why at-most-once and at-least-once are simpler but insufficient for certain domains, and what exactly-once actually means.
2. Idempotent producers: How the broker deduplicates retried writes using a Producer ID and sequence numbers. This is the foundation layer.
3. Kafka transactions: How transactions extend exactly-once across multiple partitions and tie consumer offset commits to produced messages atomically.
4. Epoch fencing: How Kafka prevents zombie producers from corrupting transaction state after a crash and restart.
5. The boundary: What exactly-once does not cover, and what you need for end-to-end guarantees with external systems.

The core insight: Kafka's exactly-once is not magic. It is a specific implementation of two-phase commit, scoped to Kafka-internal writes. The broker maintains enough state (producer IDs, sequence numbers, transaction epochs) to detect and reject duplicates without requiring the application to implement its own deduplication. But the moment you need atomicity with an external system (a database, a payment API), you are back to application-level patterns like the outbox or idempotent consumers.

The Architecture

Here is how the three mechanisms layer on top of each other. Idempotent producers handle single-partition dedup. Transactions handle multi-partition atomicity. The consume-transform-produce pattern uses both to create end-to-end exactly-once processing within Kafka.

Let me walk through the flow.

The producer application starts by calling initTransactions(). This contacts the Transaction Coordinator (a specific broker elected to manage transactions for this transactional.id). The TC assigns a Producer ID (PID) and increments the epoch. If an older instance of this producer exists with the same transactional.id, it is now fenced.

The producer calls beginTransaction(), then sends messages to multiple partitions. Each message batch carries the PID and a per-partition sequence number. If the producer retries (because an ack was lost), the broker checks the sequence number and silently drops the duplicate.

When the producer is ready to commit, it also sends the consumer offsets it wants to atomically commit via sendOffsetsToTransaction(). This ties the "I consumed up to offset X" with "I produced these output messages" into a single atomic unit.

Finally, commitTransaction() triggers a two-phase commit through the Transaction Coordinator. Phase 1 writes a PREPARE_COMMIT marker to the __transaction_state topic. Phase 2 writes COMMIT markers to every partition involved in the transaction. Only after all markers are written does the data become visible to consumers reading with isolation.level=read_committed.

If the producer crashes between Phase 1 and Phase 2, the next Transaction Coordinator handles recovery by replaying the __transaction_state log.

Idempotent Producers and Sequence Numbers

The foundation of exactly-once is deduplication at the broker. Without it, a network timeout between the producer and broker creates an ambiguous state: did the broker receive the message or not? The producer retries, and if the broker did receive it, you get a duplicate.

Here is how Kafka solves this at the protocol level.

The broker maintains a small in-memory map per partition: {PID → last_sequence_number}. Each incoming batch is checked against this map:

• Sequence matches expected (last + 1): Accept and write to the log
• Sequence equals last seen: Duplicate retry, silently ack without writing
• Sequence is greater than expected + 1: Gap detected, indicating data loss between producer and broker. The broker rejects with OutOfOrderSequenceException
• Sequence is less than last seen: Stale retry, reject

This dedup state is also stored in the partition's snapshot file, so it survives broker restarts. Kafka keeps the dedup window for each PID for a configurable duration (default: 7 days via transactional.id.expiration.ms).