GitHub Went Down for 24 Hours Because of CAP Theorem That You Learned in Just 5 Minutes
On October 21, 2018, GitHub had a network problem that lasted 43 seconds. That small problem caused GitHub to be broken for 24 hours and 11 minutes.
This article explains why that happened, using GitHub’s own official incident report. At the end, there’s a small code example you can run yourself to see the actual problem in action.
What is CAP theorem, in plain words
When you run a system on more than one server, three things matter:
Consistency — every user gets the same, latest data.
Availability — every request gets a response (success or failure, but a response).
Partition tolerance — the system keeps working even when servers can’t talk to each other.
The rule is simple: you can’t have all three at the same time. And since network problems (partitions) will happen no matter what, the real choice you’re making is between Consistency and Availability when that network problem hits.
So the real question is:
When two servers can’t talk to each other, do you show old data, or do you show an error?
That’s the whole theorem. Now let’s look at what GitHub picked, and why.
How GitHub stores data
GitHub stores things like issues, pull requests, and comments in MySQL databases. Each database cluster has:
One primary server, which handles all writes.
Several replica servers, which handle reads, and copy data from the primary.
GitHub uses a tool called Orchestrator to manage this. If the primary server becomes unreachable, Orchestrator picks a new primary automatically, using a voting system called Raft consensus.
Here is roughly what that logic does:
def handle_primary_down(nodes):
reachable_nodes = [n for n in nodes if n.is_reachable()]
if len(reachable_nodes) > len(nodes) / 2:
new_primary = elect_leader(reachable_nodes)
redirect_all_writes_to(new_primary)This sounds reasonable. If the primary disappears, promote someone else so writes can keep happening.
What went wrong
At 22:52 UTC, GitHub’s East Coast data center lost network connection to the rest of the world for 43 seconds.
During those 43 seconds:
Orchestrator, running on the West Coast, could not reach the East Coast primary.
It still had enough servers to form a majority (a quorum).
So it did exactly what it was built to do — it promoted a new primary on the West Coast.
Then, 43 seconds later, the East Coast came back online.
Now there were two primaries that had both accepted writes:
The East Coast primary had a few seconds of writes that never reached the West Coast.
The West Coast primary had several minutes of new writes that the East Coast didn’t have.
Two databases, both partially correct, both missing data the other one had. This is called a split-brain.
The decision GitHub made
GitHub had two options:
Pick one database, discard the other one’s data, and go back online fast.
Stop the bleeding, keep the site degraded, and carefully fix both databases without losing anything.
GitHub picked option 2. In their own report, they said the extended downtime was worth it to keep user data accurate and complete.
What that decision cost, in real numbers from their report:
24 hours 11 minutes of degraded service.
954 writes on just one busy cluster that had to be manually checked and fixed.
Over 5 million webhook events and 80,000 Pages builds stuck in a queue.
About 200,000 webhook payloads were dropped because they sat in the queue too long.
This is a textbook example of choosing Consistency over Availability during a network partition.
Why “just show old data” wasn’t an option here
For a lot of systems, showing slightly old data is fine. If you see an old profile picture for a minute, nobody cares.
GitHub’s issue here was different. It wasn’t stale data — it was two different sets of real writes that conflicted with each other. There was no single “latest” version to just serve. Merging it wrong could mean deleting or losing something a user had already done.
Here’s how that compares to other common systems:
Availability VS Consistency Cases
How it was fixed
GitHub restored full backups, resynced both sides, slowly moved writes back to the East Coast, and processed the entire backlog before marking the incident as resolved.
Afterward, they changed Orchestrator’s configuration so it can no longer promote a primary across regions during a partition. The tool wasn’t broken — it did exactly what it was told. The problem was nobody had told it how expensive a cross-country promotion would be until this incident happened.
Proof of concept: see the tradeoff yourself
Reading about CAP theorem is one thing. Seeing the actual tradeoff happen in code makes it click. Below is a small, self-contained Python simulation of two database nodes, “East” and “West,” with a network partition between them — just like GitHub’s incident.
Run it as-is, and you’ll see both strategies play out on the same failure.
import time
class Node:
def __init__(self, name):
self.name = name
self.data = {"issue_1_status": "open"}
class System:
def __init__(self):
self.east = Node("East")
self.west = Node("West")
self.partitioned = False
def write(self, node, key, value):
node.data[key] = value
if not self.partitioned:
# replicate to the other node
other = self.west if node is self.east else self.east
other.data[key] = value
def read(self, node, key, strategy):
if strategy == "availability":
# always answer, even if this node might be stale
return node.data.get(key)
if strategy == "consistency":
if self.partitioned:
# refuse to answer if we can't confirm we're in sync
return "ERROR: cannot guarantee latest data during partition"
return node.data.get(key)
# --- Simulation ---
system = System()
print("Before partition:")
print("East:", system.read(system.east, "issue_1_status", "consistency"))
print("West:", system.read(system.west, "issue_1_status", "consistency"))
print("\nNetwork partition happens...")
system.partitioned = True
# A write happens on East that West never receives
system.write(system.east, "issue_1_status", "closed")
print("\nDuring partition, AVAILABILITY strategy:")
print("East:", system.read(system.east, "issue_1_status", "availability"))
print("West:", system.read(system.west, "issue_1_status", "availability"))
# West will incorrectly say "open" - it's serving stale data, but it answered.
print("\nDuring partition, CONSISTENCY strategy:")
print("East:", system.read(system.east, "issue_1_status", "consistency"))
print("West:", system.read(system.west, "issue_1_status", "consistency"))
# Both nodes refuse to answer rather than risk giving wrong data.If you run this, here’s what you’ll see:
With the availability strategy, West happily tells you the issue is still “open” — even though it’s actually “closed” on East. It answered fast, but it lied.
With the consistency strategy, both nodes refuse to answer once the partition starts, rather than risk giving you wrong information.
That’s the entire tradeoff, in 40 lines of code. GitHub’s real system is obviously far more complex, but the core decision is exactly this one, just at a much bigger scale, with real user data on the line.
Summary
Network partitions will happen. That part isn’t a choice.
When they do, you either serve possibly-wrong data (availability) or you refuse to serve data until you’re sure it’s right (consistency).
GitHub chose consistency during their October 2018 incident, and it cost them over 24 hours of downtime.
The same system can make different choices for different types of data — GitHub normally favors availability for reads, but switched to consistency the moment real conflicting writes were on the line.
If you want to go deeper, GitHub’s full incident report is public and worth reading directly — it’s one of the most detailed postmortems a major company has ever published.