Post-mortem: GitHub MySQL replication lag

Takeaway

Large-table schema changes can turn ordinary asynchronous replication lag into a platform incident. A blocking DDL event queues writes on replicas, stale reads trigger fallback traffic, and that fallback can overload the primary. The durable response is to canary migrations, monitor lag and recovery rate, keep a kill switch, and size the system for bounded staleness and fallback load.

Scope and Evidence

5-Minute Incident Walkthrough

1. Trigger: A large schema migration emits work that replicas cannot apply at the primary's write rate.
2. Queue: The replica SQL applier blocks on DDL while subsequent binlog events accumulate.
3. User impact: Lagged replicas return stale or missing data, which looks like an application inconsistency.
4. Amplification: Lag-aware routing sends more reads to the primary, increasing CPU, connections, and write latency.
5. Recovery: Pause or throttle the migration, preserve a healthy read path, and let replicas drain faster than new events arrive.

Causal chain: large-table DDL → serial replica apply → binlog backlog → stale reads → primary fallback load → slower writes and binlog production → longer recovery.

Incident Summary

Date: Composite incident (GitHub has documented multiple replication lag events; this is a representative analysis based on public disclosures) Duration: Representative scenario: 4-8 hours of elevated replica lag, with partial degradation continuing for 12+ hours Systems affected: In the modeled topology, MySQL read replicas serving GitHub.com features such as issue trackers, pull request views, repository listings, and user profile data Impact: In the representative scenario, read queries return stale data. Users see missing issues, outdated pull request statuses, and phantom "file not found" errors on recently created repos. Some reads fall back to primary, increasing primary load to dangerous levels. Root cause: In the modeled scenario, a schema migration (ALTER TABLE or gh-ost operation) on a multi-billion-row table causes replica lag to grow from milliseconds to hours. Single-threaded replication on replicas means large DDL operations block subsequent replication events, creating a cascading stale-data problem across the platform.

GitHub runs one of the world's largest MySQL deployments. At this scale, operations that are routine for smaller databases become risky production events. Schema migrations sit at the intersection of every database risk factor: table locking, replication lag, disk I/O, and query plan invalidation. This composite scenario illustrates what happens when those factors converge on a table with billions of rows and thousands of writes per second.

The scenario is a useful counterexample to the idea that "it's just an ALTER TABLE, we'll run it during low traffic." Schema migrations on large tables at GitHub's scale are production deployments that deserve the same caution as a code release.

---

What Happened: The Timeline

The timeline below is a representative reconstruction for the composite analysis. Its timestamps, row counts, lag values, and user symptoms are illustrative unless tied to a cited public disclosure.

The timeline reveals the fundamental problem with large DDL operations on replicated databases: the impact is not immediate, it is cumulative. Lag starts small and grows linearly. By the time it is noticeable to users, the backlog is already large enough that recovery takes hours even after the migration is paused.

The user-visible symptoms were varied and confusing. A developer opens a pull request at 11:00 AM. They see it in their dashboard (served from the primary via write-read-your-own-writes routing). Their teammate refreshes the PR list at 11:05 AM but the PR does not appear (served from a replica that is 40 minutes behind). The teammate thinks the developer has not opened the PR yet. Meanwhile, CI status checks that ran against a lagged replica returned outdated file contents for the diff. The symptoms looked like "GitHub is randomly broken" rather than "GitHub has replication lag," which made it harder for users and support to pinpoint the issue.

Another symptom: merge conflicts that should not exist. If a developer pushes a commit and immediately tries to merge a PR, the merge operation might read the base branch from a lagged replica that does not have the latest commit. The merge sees a conflict that would not exist if it read from the primary. This is one of those edge cases that only surfaces at scale with minutes of replication lag.

---

Representative GitHub-Scale MySQL Architecture

Before GitHub's migration to Vitess, public materials described a large MySQL deployment. The diagram below is a representative model; it explains why replication lag can have broad impact without claiming to reproduce an unpublished production topology.

A few things stand out about this architecture:

Read-heavy workload. GitHub's traffic is overwhelmingly reads: viewing issues, browsing code, loading pull requests. The read-to-write ratio is roughly 10:1 or higher. This is why they use multiple read replicas. The primary cannot handle both writes and the full read load alone.

Asynchronous replication. MySQL replication is asynchronous by default. The primary writes to its binary log and returns success to the client immediately. Replicas pull from the binary log and apply changes at their own pace. This means replicas are always slightly behind the primary. Under normal conditions, the lag is sub-second and invisible to users. Under stress, it can grow to minutes or hours.

ProxySQL as the routing layer. ProxySQL sits between the application and MySQL, routing writes to the primary and reads to replicas. It monitors replica lag and can reroute reads to the primary when lag exceeds a configurable threshold. This is a safety mechanism, but it has a dangerous side effect: when replicas are lagged, the primary absorbs the full read load on top of its write load.

For interviews, this is a textbook read-replica architecture. If you are asked about database scaling, you should be able to draw this diagram from memory and explain the tradeoff: replicas give you read throughput, but replication lag means reads can be stale. The question is always "how stale is acceptable?"

---

Root Cause: Schema Migration on a Hot Table

The specific trigger was a schema migration on one of GitHub's largest and most actively written tables. Think of tables like issues, pull_requests, or repository_events, tables with billions of rows that receive thousands of writes per second.

What the migration looked like:

GitHub uses gh-ost (GitHub Online Schema Trickle) for online schema changes. gh-ost works by:

1. Creating a shadow copy of the table with the new schema
2. Copying existing rows from the original table to the shadow table in small batches
3. Tailing the binary log to capture any writes that happen during the copy
4. Applying those captured writes to the shadow table to keep it in sync
5. Performing an atomic table swap once the shadow table is fully caught up

On the primary, this is relatively graceful. gh-ost copies rows in small chunks, throttles based on server load, and does not lock the original table for the duration. Writes continue unimpeded. The primary barely notices.