Two-phase Commit Coordinator Crash

TL;DR

• Two-phase commit (2PC) is a distributed transaction protocol where a coordinator asks all participants to "prepare to commit," then issues a final "commit" or "abort" instruction.
• If the coordinator crashes after participants have responded "ready" but before it sends the final decision, participants are stuck: they've locked resources but can't commit or abort without hearing from the coordinator.
• This is the blocking problem: participants hold locks until the coordinator resolves the transaction. During a prolonged outage, those locks can make writes that touch the same data unavailable and can exhaust connection pools.
• Cross-service business workflows often use sagas (accept eventual consistency) or outbox-driven messaging; storage engines that need replicated agreement may use Raft.

30-second explanation

Two-phase commit is safe only while the coordinator can finish the decision. After participants vote “prepared,” they must keep their locks until they learn whether to commit or abort. If the coordinator disappears in that window, every participant waits, so a small coordinator outage can turn into blocked transactions and exhausted connection pools.

5-minute explanation

Walk through the two phases: participants first make their local work durable and vote; the coordinator then records the decision and broadcasts commit or abort. The difficult case is a coordinator crash after the votes but before all decisions are delivered. A participant cannot safely choose on its own because another participant may already have received a different decision. Recovery needs durable coordinator state and bounded operational handling, but a timeout alone cannot preserve atomicity. For cross-service workflows, compare that blocking trade-off with a saga or an outbox-driven workflow that accepts compensation or eventual consistency.

---

The Problem

Consider this illustrative scenario: it's 3 a.m. and the payment service's coordinator has crashed. Three microservices (inventory, payments, orders) have all voted "yes, I can commit" and are holding database locks. They're waiting for the coordinator to tell them whether to commit or abort. The coordinator may be unavailable for 47 minutes.

For example, if the inventory service holds row-level locks on 200 SKUs while the coordinator is unavailable, checkout requests that touch those rows queue behind the locks. A long enough outage can exhaust the connection pool and take down the wider checkout flow, not just the transactions that were mid-2PC.

2PC has exactly two phases:

Phase 1 (Prepare): Coordinator sends "prepare" to all participants. Each participant writes to its local WAL, acquires locks, and responds "yes, I can commit" or "no, I cannot."

Phase 2 (Commit/Abort): If all participants said yes, coordinator sends "commit" to all. If any said no, coordinator sends "abort" to all.

The failure window is the gap between Phase 1 completing and Phase 2 beginning:

Participants in the "prepared" state cannot unilaterally commit (they might be the only ones committing, violating atomicity) and cannot unilaterally abort (the coordinator might have already sent "commit" to others before crashing). They must wait.

The cascade effect

The blocking doesn't just affect the 2PC transaction itself. Those held locks block every other transaction that touches the same rows. Here's what the cascade looks like in practice:

Within minutes, the connection pool on the inventory service fills up with blocked queries. New requests can't get a connection. The entire service becomes unresponsive, even for requests that have nothing to do with the 2PC transaction.

This can be difficult to diagnose: monitoring may mark the inventory service as "unhealthy" without showing that a coordinator in another service caused the blocked transactions. Restarting the inventory service can temporarily clear connections, but the symptom returns if the coordinator remains unavailable. Trace transaction state and coordinator health together.

---

Why It Happens

Teams reach for 2PC because the requirement sounds simple: "I need this operation to be atomic across two services." That's a legitimate requirement. The problem is that 2PC solves it with a single-coordinator design that creates a blocking failure mode.

Here's how teams typically arrive at 2PC:

1. The database grew into multiple services. What was once a single-database transaction spanning two tables became a cross-service call when those tables were split into separate services. The team reaches for 2PC to preserve the atomicity they had before.
2. The ORM or framework makes it easy. Spring Boot with JTA, for example, makes distributed transactions look like local transactions with @Transactional. Teams don't realize they've introduced a coordinator until it fails.
3. Saga complexity feels disproportionate. For a simple two-step workflow, writing compensating transactions for both steps plus error handling feels like over-engineering. 2PC looks simpler (until the coordinator crashes).
4. The failure mode wasn't tested. 2PC implementations may work normally for long periods. The blocking problem appears when the coordinator fails, which teams often don't simulate in testing.

You might think: "Just have participants time out and abort." Unfortunately, this is unsafe. Consider:

• Coordinator sent "commit" to A before crashing.
• B and C waited, timed out, and aborted.
• A committed. B and C aborted.
• You now have a partial commit: money transferred on one side, not the other.

The normal safe recovery path is for the coordinator to restart, read its WAL for the transaction's state, and re-send the final decision. Participants remain blocked while they wait for that restart. If the coordinator's disk is lost, manual intervention may be needed to determine which participants to commit and which to abort.

The partial-commit scenario is the key data-integrity risk:

This is why "just add a timeout" is unsafe. The timeout-and-abort approach can add a data-integrity problem to the availability problem it was intended to solve.

Why this is worse in practice than theory

2PC assumes the coordinator will eventually recover. In practice:

• The coordinator's database might be on a failed disk with no recent backup.
• Kubernetes might restart the coordinator on a new node without local WAL access.
• Multi-datacenter deployments can hold locks across DCs during a DC-level outage.
• The "just restart it" recovery takes minutes, during which all participating services are blocked.

A coordinator failure can freeze writes that depend on the affected participants. If many workflows share those participants, the impact can be broad, and the blast radius grows with the number of services in the 2PC transactions.

An illustrative cascade calculation

Consider these illustrative assumptions:

• 3 services participate in 2PC transactions
• Each service has a connection pool of 50 connections
• 2PC transactions take 200ms on average
• The coordinator crashes with 10 prepared transactions in flight