Split-Brain

Introduction

Split-brain is a coordination failure in which two nodes or partitions believe they are the active writer and accept conflicting updates. It is especially dangerous for mutable business data because both sides can return successful responses while the system's state diverges.

Mental model: electing a new leader answers "who should lead?" Fencing answers "can the old leader still write?" A safe single-primary design needs a way to make stale writers rejectable, unreachable, or otherwise harmless before concurrent writes can become accepted state.

TL;DR

• Split-brain occurs when a network partition causes two nodes to each believe they are the only active primary, and both accept writes, creating divergent state that cannot be automatically reconciled.
• Even with a quorum-based election, a primary/replica setup can leave the old primary accepting writes during the interval before demotion or fencing takes effect.
• The consequences (duplicate orders, double payments, lost messages) can be silent and may only be discovered during reconciliation or auditing, sometimes days later.
• Prevention uses fencing tokens (monotonic IDs that reject stale writes), STONITH (forcibly killing the old primary), or a properly implemented consensus protocol such as Raft that prevents two leaders in the same term.

30-Second Explanation

Split-brain is a failure mode in which two nodes both believe they are the active primary and accept writes during a partition. The safe design is to make stale writers unable to commit: use majority-based leader election together with fencing tokens or an equivalent mechanism that stops the old primary. Quorum can choose a new leader, but it does not by itself stop the old leader from serving clients that can still reach it.

5-Minute Explanation

Start with the failure scenario: a primary becomes unreachable, a replica is promoted, and the old primary is still alive on the other side of a network partition. If both sides accept writes, the system creates divergent records and downstream systems may observe different truths. Explain that recovery is domain-specific: payments, inventory, sessions, and analytics may each need a different reconciliation rule.

Then separate the protections. Consensus or a witness prevents two nodes from winning the same election, while fencing tokens or STONITH prevent a demoted node from continuing to write. Close with the trade-off: a short period of unavailability is usually safer than silent data divergence, unless the data model explicitly supports active-active conflict resolution.

What It Is

Your primary database fails a health check at 2:47 a.m. Your high-availability setup promotes the replica to primary. Normal so far.

But the original primary didn't actually fail. It experienced a 30-second network partition. Its health check packets were lost. It's still running, still accepting writes from one subset of application servers that can reach it. The replica is now also accepting writes from another subset. Both believe they are the authoritative primary.

As an illustrative scenario, for 90 seconds the Orders table in "Primary A" receives 400 order writes while "Primary B" receives 600. When the network heals and both nodes reconnect, there can be 1,000 divergent order records with overlapping auto-increment IDs.

You now have a data reconciliation problem with no automatic solution. Some orders exist in only one database. Some IDs were created twice for different orders. Your payment processor ran against one view; your fulfillment system ran against the other.

The bottom line: silent data divergence can be harder to contain than a visible outage. An outage triggers alerts; split-brain can return successful responses while corrupting or forking data.

What reconciliation actually looks like

When the partition heals, an HA system may detect the dual-primary situation and demote one node, but accepted writes may already have diverged. You're left with two datasets that need comparison and reconciliation.

Reconciliation requires answering questions like:

• Order #1001 exists on both primaries with different customer IDs. Which is real?
• Payment for order #1001 was charged on Primary A's version. The customer on Primary B's version never paid. Do we charge them retroactively or cancel?
• Inventory was decremented on both primaries. The actual stock count is neither value.
• Analytics pipelines consumed data from both primaries. Reports generated during the window are wrong.

There is no single merge policy that resolves every business conflict. Each table needs domain-specific merge logic, and one-off scripts can introduce bugs if their conflict rules have edge cases.

For this illustrative scenario, the reconciliation strategy is:

• Orders table: Use the payment processor's transaction records as the source of truth. If a payment went through, the order is real regardless of which primary created it.
• Inventory table: Recount physical stock and adjust. Both primaries' counts were wrong.
• User sessions: Discard both and force re-login. Acceptable data loss for sessions.
• Analytics events: Deduplicate by event ID, keep earliest timestamp. Accept that some metrics for the split-brain window are approximate.

The reconciliation process itself should be idempotent and auditable. Document merge decisions so you can explain to the business team (or a regulator) why certain records were prioritized over others. In regulated industries, this documentation may be required by the applicable controls or rules.

Concrete before/after: order writes during failover

Before: Primary A and the promoted Primary B both accept an order for the same logical request during a partition. Rejoining the nodes leaves duplicate or conflicting records and requires domain-specific reconciliation.

After: a fencing token, STONITH action, or quorum-enforced consensus rule causes writes from the stale side to be rejected or makes the stale node unreachable before the new primary accepts writes. The cluster may be briefly unavailable, but it returns to one authoritative write path.

How It Develops

A common misconception is: "We use majority quorum, so we can't have split-brain." A quorum can constrain elections and acknowledged writes when the system implements those rules, but it does not by itself fence an old primary that still accepts client writes during a failover window.

The old leader doesn't know it has been demoted. It feels fully healthy. Without a mechanism to explicitly tell it "stop accepting writes, you are no longer the leader" (or kill it outright), it may continue as a rogue primary.

The critical distinction is: the problem isn't only the election; it's the demotion. A system may promote a new leader correctly while the old leader remains reachable by some clients. The gap between promotion and effective fencing is where split-brain can occur.

Split-brain emerges from individually reasonable decisions:

1. You enable automatic failover because manual promotion takes too long during outages.
2. You set aggressive health check timeouts (5-10 seconds) because slow failover means downtime.
3. You don't implement fencing because it adds complexity and most failovers work fine without it.
4. You don't test network partitions because chaos engineering feels risky in production.

Each decision is defensible. The combination creates a system where a transient network blip triggers promotion while the old primary is still alive and writing.

Common trigger scenarios

Split-brain doesn't require a dramatic infrastructure failure. These mundane events can trigger it:

• Switch firmware upgrade: A top-of-rack switch reboots during a firmware update. The primary and HA controller are on different sides of the switch. The HA controller can't reach the primary for 15-30 seconds and promotes the replica.
• GC pause on the primary: A long garbage collection pause (Java, .NET) causes the primary to miss health check deadlines. The HA controller thinks it's dead. When the GC finishes, the primary resumes writing.
• DNS resolution failure: The HA controller resolves the primary's hostname to a stale IP after a cloud networking event. It can't reach the primary and promotes the replica, even though the primary is running fine.
• Asymmetric partition: The primary can reach the database clients but not the HA controller. From the clients' perspective, everything is fine. From the HA controller's perspective, the primary is dead.

The asymmetric partition is particularly dangerous because the primary appears healthy to its clients. Write traffic continues normally. The only sign of trouble is the HA controller promoting a replica. If your clients don't re-resolve the primary address after failover (common with connection pooling), they can keep writing to the old primary until routing or fencing changes take effect.

Here's the timeline that makes this so dangerous:

Be careful not to confuse the problem with the solution. Adding a tie-breaker node (arbiter) or a witness can prevent election ambiguity, but does not necessarily prevent the old primary from accepting writes during the partition. Single-primary systems need fencing or another mechanism that makes concurrent stale writes unsafe.

Symptoms and Diagnosis

Split-brain is insidious because both primaries can look healthy when examined individually. You often see the divergence only when you compare them, so apparently random data inconsistencies should be correlated with failover and partition windows.

A useful proactive detection method is a heartbeat: each primary writes a unique token to a shared external store (etcd, ZooKeeper) every second. If two different active-primary tokens appear for the same cluster, investigate for split-brain. Without this kind of check, detection may be retroactive during reconciliation.

Proactive monitoring setup

The key insight: don't wait for data divergence to detect split-brain. Monitor for the preconditions.

// Pseudo-code: split-brain detector running on each node
async function checkForSplitBrain(): Promise<void> {
  const myRole = await db.query("SELECT pg_is_in_recovery()");
  const clusterPrimaries = await etcd.get("/cluster/primaries");

  if (!myRole.isRecovery && clusterPrimaries.length > 1) {
    await alerting.fire("SPLIT_BRAIN_DETECTED", {
      severity: "critical",
      primaries: clusterPrimaries,
      message: "Multiple nodes reporting as primary",
    });
    // Optionally: self-fence if our token is lower
    if (myFencingToken < Math.max(...clusterPrimaries.map(p => p.token))) {
      await db.setReadOnly(true);
    }
  }
}

Set up alerts on:

• Two nodes with primary role in the same cluster (immediate page)
• Replication lag dropping to zero on a known replica (it may have been promoted)
• Fencing token conflicts at the storage layer (stale writes being rejected)
• Network partition duration exceeding failover timeout (the risk of an unsafe promotion is rising)

Proactive alerts shorten the time between an unsafe promotion and containment. Size the monitoring and escalation work to the value of the data being protected.

What to do if split-brain is detected in progress

If your monitoring detects an active split-brain (two nodes reporting as primary), respond immediately:

1. Identify the legitimate primary. Check which node has the higher fencing token or was most recently elected by the HA controller.
2. Force the rogue primary to read-only. On PostgreSQL: ALTER SYSTEM SET default_transaction_read_only = on; SELECT pg_reload_conf();
3. Redirect all clients to the legitimate primary. Update DNS, service discovery, or load balancer configuration.
4. Assess divergence. Compare row counts and recent write timestamps to quantify how many records diverged.
5. Begin reconciliation. Use the strategies from "What It Is" section to merge divergent data.

Treat the first three steps as immediate priorities. Steps 4 and 5 are the long tail and can take days.

Remediation and Prevention

There are three main approaches to preventing split-brain, each suited to different system architectures. The right choice depends on your infrastructure, your availability requirements, and how much complexity you're willing to add.

Fix 1: Fencing tokens

Every lease or lock grant includes a monotonic token. Storage nodes reject any write carrying a token older than the most recently seen token. When a new primary is elected with token 42, the old primary's writes with token 41 are rejected at storage.

// Storage node validates fencing token on every write
async function handleWrite(request: WriteRequest): Promise<WriteResponse> {
  if (request.fencingToken < this.highestSeenToken) {
    return { status: "REJECTED", reason: "stale fencing token" };
  }
  this.highestSeenToken = request.fencingToken;
  await this.storage.write(request.key, request.value);
  return { status: "OK" };
}

Fencing tokens are a standard way to make stale leader writes rejectable during leader transitions.

How it works with leases: The primary holds a lease (a time-limited lock) from a coordination service like ZooKeeper or etcd. The lease includes a monotonically increasing token. When the lease expires or the primary is fenced, the new primary gets a higher token. Even if the old primary doesn't know its lease expired, its writes are rejected at the storage layer because its token is stale.

The lease TTL is a critical tuning parameter. Too short (1-2 seconds) and normal GC pauses or network jitter can cause spurious failovers. Too long (30+ seconds) and the system may have a wider split-brain window before the lease expires. A 5-10 second lease is an example balance, not a universal setting; tune it against observed pauses, network behavior, and recovery objectives.

// Lease renewal with fencing token
async function renewLease(): Promise<LeaseGrant> {
  const lease = await etcd.grant({ ttl: 10 }); // 10-second lease
  const token = lease.id; // Monotonically increasing
  await etcd.put("/cluster/primary", nodeId, { lease: lease.id });
  return { leaseId: lease.id, fencingToken: token };
}

// Primary must renew before TTL expires
setInterval(async () => {
  try {
    await etcd.leaseKeepAlive(currentLease.leaseId);
  } catch (error) {
    // Lease renewal failed: self-demote to read-only
    await db.setReadOnly(true);
    logger.error("Lease renewal failed, demoting to read-only");
  }
}, 3000); // Renew every 3s for a 10s lease

Trade-off: Every storage node must track and validate tokens. Adds a small amount of write-path latency (microseconds) and requires all clients to include the token. If even one write path skips token validation, the fencing is bypassed.

Fix 2: STONITH (Shoot The Other Node In The Head)

When a new primary is elected, send a hard shutdown command to the old primary via an out-of-band channel (IPMI, iLO, PDU power cut). A conservative controller can block the election if the old primary cannot be confirmed dead. A momentary outage may be preferable to concurrent writes for data that requires single-primary semantics.

AWS RDS Multi-AZ uses a variant of this: the old primary is fenced at the network level (security group update) before the replica promotion completes.

In on-premises environments, STONITH is typically implemented via:

• IPMI/iLO/DRAC: Send a hardware-level power-off command to the old primary's BMC (baseboard management controller). Works even if the OS is hung.
• PDU power cut: Cut power to the old primary's rack position via a managed power distribution unit. The most reliable method but requires physical infrastructure access.
• SBD (STONITH Block Device): A shared disk that acts as a "poison pill." The old primary periodically reads the SBD; if its "slot" is marked as fenced, it self-terminates. Used in Pacemaker/Corosync clusters.

Trade-off: Requires out-of-band management access. If the management network is also partitioned, you're stuck. Some teams use a "poison pill" approach where the old primary self-terminates when its lease expires.

Fix 3: Raft-based consensus

Properly implemented Raft structurally prevents two leaders in the same term. A leader can only commit log entries after a majority acknowledges them. If it loses majority connectivity, it cannot commit new entries, and a new leader can only be elected by a majority.

With those quorum rules, a committed split-brain would require both a new leader to be elected and the old leader to continue making quorum-backed progress, which the protocol is designed to prevent. Systems such as etcd, CockroachDB, and TiKV use Raft-based designs for this class of coordination.

Trade-off: Leader elections can cause brief unavailability; the duration depends on timeouts, load, and implementation. An odd number of voting nodes (3, 5, 7) is common so a partition is less likely to produce a tied election.

Comparison of prevention mechanisms

Real-world implementations

PostgreSQL + Patroni: Uses etcd for leader election and implements fencing via leader key TTL. The old primary checks its leader key before accepting writes; if the key doesn't belong to it, it demotes itself to read-only. This is a lease-based fencing approach.

MySQL Group Replication: Uses a Paxos-based group communication layer. A node that can't reach a majority of the group automatically switches to read-only mode. This is structurally similar to Raft.

Redis Sentinel: An imperfect solution. Sentinel elects a new primary, but clients that cached the old primary's address can still write to it until they refresh. Redis doesn't implement fencing tokens natively. This is why Redis Cluster (which uses a gossip protocol with majority agreement) is preferred for use cases where split-brain matters.

Amazon Aurora: Uses a shared storage layer with a single writer. The storage layer itself enforces that only one node can write at a time, which is a form of storage-level fencing. Failover promotes a replica to writer by updating the storage layer's writer ID.

MongoDB Replica Sets: Uses Raft-like consensus for elections. A node must receive votes from a majority to become primary. If the old primary is partitioned, it steps down when it can't reach a majority. However, clients that cached the old primary's connection may still send writes to it; the MongoDB driver's retryWrites feature handles this by retrying failed writes on the new primary.

For MongoDB specifically, writes to a partitioned old primary fail with a NotWritablePrimary error once the node steps down. But there's a brief window between the partition starting and the node detecting the loss of majority where writes succeed on the old primary. This is the same fundamental issue as other systems: the demotion is not instantaneous.

TrueTime-style systems: Use bounded clock uncertainty together with consensus to order writes across leader transitions. This requires specialized clock-synchronization infrastructure and is not a drop-in substitute for fencing in most systems.

The common thread

All of these implementations solve the same problem: ensuring that the old leader stops writing before (or simultaneously with) the new leader starting. The mechanism differs (lease expiry, majority loss, storage-layer enforcement, clock bounds), but the principle is the same. Pick the mechanism that fits your infrastructure. Managed shared-storage databases can provide storage-level fencing; a PostgreSQL deployment can use a lease-based controller; a new coordination system can use a well-tested consensus library rather than inventing its own election protocol.

Which fix to use?

Why It Matters: Severity and Blast Radius

Split-brain can be among the more severe distributed-systems failures because the damage may be invisible at the time it happens. Both primaries can accept writes successfully and clients can receive 200 OK responses. The divergence may surface later during reconciliation, auditing, or a customer report. This can be harder to contain than a clean outage because split-brain is silent while it is creating divergent state.

The blast radius fans out to every system that reads from the divergent primaries:

• Payments: One primary charged the customer; the other didn't. Or both did, resulting in a double-charge.
• Fulfillment: One primary created the shipping order; the other didn't. Customer gets charged but nothing ships.
• Analytics: Dashboards show different numbers depending on which replica they query. Business decisions made on corrupted data.
• Audit logs: Compliance reports become unreliable because the event timeline is forked.

Recovery difficulty: Hard. You can't just "pick one primary and discard the other" because both contain valid writes that the other is missing. Merging requires domain-specific conflict resolution logic that may not exist when you need it. For a 90-second split-brain window with 1,000 divergent writes, reconciliation could require several days of manual work; longer windows or higher write throughput can increase that effort.

A 4-minute split-brain incident can require weeks of reconciliation when downstream systems have consumed both versions. The most time-consuming part may be identifying which systems need correction, not just merging the source rows.

Trade-offs and Legitimate Exceptions

Not every system needs the same split-brain controls. The severity depends largely on what happens when two nodes diverge. Before adding fencing complexity, ask: "If both nodes accepted writes for 60 seconds, what's the worst business outcome?"

• Read-only replicas with no promotion: If replicas never get promoted to primary, this two-writer failure mode is not present. Stale reads are a different problem.
• CRDT-based systems: Conflict-free replicated data types are designed for concurrent writes. Counters, sets, and LWW-registers merge automatically. You accept the convergence semantics by design. DynamoDB streams with CRDTs is a common implementation.
• Active-active with last-writer-wins: If your data model tolerates LWW semantics (session caches, view counters, non-critical metadata), two writers are acceptable. DynamoDB global tables use this approach.
• Dev/staging environments: Controlled split-brain simulation is valuable for validating your fencing implementation. Run it in an isolated non-production environment to verify detection and prevention.
• Immutable append-only logs: If both primaries are only appending (never updating existing rows), reconciliation is much simpler: merge both logs and deduplicate. This is why event sourcing systems are more resilient to split-brain than mutable-state systems.

Testing for split-brain resilience

You cannot gain much confidence in fencing without testing it. Here's a practical approach:

1. Simulate network partition: Use iptables rules or tc (traffic control) to block traffic between the primary and the HA controller while keeping the primary reachable from some app servers.
2. Verify promotion: Confirm the replica gets promoted to primary.
3. Attempt writes to old primary: Send writes to the old primary from a client that can still reach it.
4. Verify fencing: Check that writes to the old primary are rejected (fencing tokens) or that the old primary is unreachable (STONITH).
5. Heal the partition: Remove the iptables rules and verify the cluster recovers to a single-primary state.

If step 4 fails (the old primary accepts writes), your fencing implementation has a gap that should be fixed before production reliance.

Chaos engineering tools can automate this type of testing. For database-specific split-brain testing, tools like toxiproxy can simulate network partitions between specific hosts without affecting the rest of your infrastructure.

The key rule: test your fencing in staging before relying on it in production. A fencing implementation that hasn't been tested under partition conditions remains an untested assumption.

// Chaos test: verify fencing token enforcement
async function testFencingTokens(): Promise<void> {
  // Simulate stale primary with old token
  const staleToken = 41;
  const currentToken = 42;
  
  const result = await storage.write({
    key: "test-key",
    value: "stale-write",
    fencingToken: staleToken,
  });
  
  assert(result.status === "REJECTED", 
    `Fencing broken: stale write accepted with token ${staleToken}`);
  
  const validResult = await storage.write({
    key: "test-key",
    value: "valid-write",
    fencingToken: currentToken,
  });
  
  assert(validResult.status === "OK",
    "Valid write should be accepted");
}

Common Mistakes and Misconceptions

• "Majority quorum alone prevents split-brain." It can constrain an election, but it does not necessarily stop an old primary from accepting writes that do not pass through the quorum or fencing path.
• "A witness or arbiter is fencing." A witness can resolve election ambiguity; it does not automatically make a partitioned old primary unreachable or read-only.
• "A successful demotion call proves the old node stopped writing." Demotion is only useful when every write path and storage layer honors it; stale clients and alternate paths need protection too.
• "Raft means the cluster never pauses." A properly implemented Raft cluster avoids two quorum-backed leaders in one term, but elections can make writes briefly unavailable.
• "Active-active is always unsafe." CRDTs, last-writer-wins, and append-only models can make concurrent writes acceptable for selected data; payment state and inventory usually need stricter semantics.

A robust design review should include:

• Naming the split-brain risk explicitly when discussing failover
• Explaining why quorum alone doesn't prevent old-primary writes
• Describing at least one fencing mechanism (tokens, STONITH, or Raft)
• Mentioning that silent divergence is worse than downtime

Test Your Understanding

Recap

1. Split-brain happens when two nodes both believe they are the primary and accept concurrent divergent writes.
2. Network partitions trigger it; quorum can prevent re-election ambiguity but doesn't necessarily stop the old primary from writing during the partition window.
3. Adding an arbiter or witness can prevent election ties, but single-primary data still needs fencing or another mechanism that blocks stale writes.
4. Fencing tokens ensure stale writes are rejected at storage regardless of which node sent them. Every write path must validate the token.
5. STONITH forcibly kills the old primary. If you can't confirm it's dead, don't promote the replica.
6. Properly implemented Raft-based consensus prevents two quorum-backed leaders in one term, at the cost of possible unavailability during elections.
7. Silent data divergence can be harder to contain than visible downtime because downstream systems may consume both versions before detection.
8. For active-active designs, use CRDTs or LWW semantics for data that tolerates it, and single-primary for data that doesn't.
9. Test your fencing implementation under real partition conditions in staging. An untested fencing mechanism remains an assumption.

Related Concepts

• Replication — Primary/replica roles, failover, and replication lag.
• Consistency Models — The consistency choices behind single-primary and active-active designs.
• CAP Theorem — Why partitions force trade-offs between availability and consistency.
• Consensus Algorithms — Raft, Paxos, quorum, and leader election.
• Databases — Storage constraints, transactions, and recovery implications.