GitHub Upgraded 1,200 MySQL Hosts Without Dropping a Single Query

How GitHub’s database team spent a year migrating 300 terabytes of data and 5.5 million queries per second from an end-of-life database — and why the rollback path was more important than the upgrade itself.

MySQL 5.7 was dying.

Oracle had announced end-of-life for October 2023. No more security patches. No more bug fixes. For most companies, this would mean scheduling a maintenance window, upgrading the database, and moving on.

GitHub is not most companies.

GitHub’s production database fleet spanned 1,200 hosts, 300 terabytes of data, and 5.5 million queries every second. It was the foundation storing every repository, every pull request, every code review comment, every user account for 100 million developers.

Getting from MySQL 5.7 to MySQL 8.0 without disrupting any of them was going to take more than a weekend.

It took over a year.

Why This Was Different

Every software engineer eventually has to do a database upgrade. Install the new version, test the migrations, deploy in a maintenance window. Done.

GitHub’s situation made every one of those steps an order of magnitude harder.

The fleet was not one database. It was 50+ independent clusters, each serving a specific product domain — repositories, issues, pull requests, billing, the social graph. The largest of these were horizontally sharded via Vitess, adding another layer of complexity. Each cluster had its own primary-replica topology, its own traffic pattern, its own quirks.

Upgrading meant doing it 50+ times, independently, on a live system that couldn’t be taken offline.

But the real challenge wasn’t the scale. It was a single technical discovery that threatened to make the entire migration impossible.

The Hidden Breaking Change

When GitHub’s team tested MySQL 8.0 in staging, they found something that changed everything.

MySQL 8.0 changes the default character set to utf8mb4 with a new collation — utf8mb4_0900_ai_ci — that MySQL 5.7 simply doesn't support.

This sounds like a small compatibility detail. It wasn’t.

In a database cluster, the primary writes data and replicas receive a stream of those changes to stay synchronized. This stream is called the binary log. When an 8.0 primary wrote to the binary log, the collation metadata it included was unreadable by any 5.7 replica downstream.

Replication broke. Immediately. Completely.

GitHub’s entire rollback strategy depended on maintaining backward replication from 8.0 to 5.7 — the ability to promote a new 8.0 primary while keeping 5.7 standbys ready to take over if something went wrong. If 8.0 primaries couldn’t replicate to 5.7 replicas, there was no rollback path.

There was a second problem too. MySQL 8.0 introduced a new ROLE management system for database permissions. The scripts that managed those permissions now generated 8.0-syntax statements in the binary log that 5.7 replicas couldn’t parse.

Both issues had to be fixed before a single production primary could be promoted.

The team solved the collation problem by explicitly setting databases and tables to a 5.7-compatible collation before any promotion — overriding 8.0’s new default. The roles problem was solved by temporarily stripping role-expansion from permission grants during the upgrade window. Surgical, careful, and validated extensively in staging before any production host saw it.

The Playbook: Five Steps, Always Rollback Available

The actual upgrade playbook was built around one non-negotiable constraint: the rollback path must exist at every single step until a 24-hour validation window has passed.

This meant the upgrade was never a point of no return. It was a series of reversible decisions, each one adding confidence before the next step began.

Step 1: Upgrade replicas one data center at a time. Route read traffic to 8.0 replicas while the primary stays on 5.7. Rollback at this step: disable 8.0 replicas, re-enable 5.7 ones.

Step 2: Reconfigure the replication topology to create two parallel chains — one of 8.0 replicas, one of 5.7 standbys. Both receiving the same writes from the 5.7 primary.

Step 3: Promote an 8.0 replica to primary via Orchestrator’s graceful failover. This is the moment the primary version flips. Rollback still available: the 5.7 chain is still warm and synchronized.

Step 4: Monitor for a full 24 hours of production traffic. Not 2 hours. Not until it looks stable. A complete diurnal cycle — morning rush, peak business hours, overnight lows — because GitHub’s traffic has strong daily patterns that reveal different failure modes at different times of day.

Step 5: Only after 24 hours of clean traffic: decommission the 5.7 standbys. This is the moment the rollback window closes. No going back.

This five-step sequence took the team across 50+ clusters over the course of a year. Not because each step was slow — but because the 24-hour validation window was honored every single time.

The Preparation That Made It Possible

The upgrade work didn’t actually start in 2022 or 2023. It started in July 2022 — a full year before any production host was promoted.

The team added MySQL 8.0 to CI alongside 5.7 for every application using MySQL. This meant every code change, every pull request, every merge was continuously tested against both versions simultaneously. Application teams discovered query incompatibilities, deprecated feature usage, and reserved keyword conflicts in automated tests — not during live promotions.

By the time a cluster was ready to upgrade, the application code was already known-compatible. The upgrade was validating infrastructure, not discovering application bugs at the worst possible moment.

They also built MySQL 8.0 Codespaces debug containers, created a GitHub Project board to track every cluster’s upgrade status, and wrote a reusable automation framework — not a one-time script, but a library that would survive to make the next major version upgrade faster.

What MySQL 8.0 Actually Unlocked

The upgrade wasn’t just about staying on supported software.

MySQL 8.0 brought features GitHub’s database team had genuinely wanted for years.

Instant DDLs allow many schema changes to be applied without rebuilding the entire table. On a fleet where some tables receive millions of queries per second, the old approach of running ALTER TABLE and locking the table for hours was genuinely dangerous. Instant DDLs made schema management measurably safer.

Invisible indexes let engineers create an index, test it under production traffic without it being used by the query planner, and only then make it active. The ability to validate an index on real production data before committing to it is a significant operational safety improvement.

Compressed binary logs reduce the replication bandwidth between primary and replicas — a meaningful saving when you’re running 5.5 million queries per second across 1,200 hosts.

The Tool That Kept It Safe

Sitting underneath the entire upgrade was Orchestrator — GitHub’s open-source MySQL topology manager that handles automated failover and candidate promotion.

During the upgrade, Orchestrator required one specific configuration: all 5.7 hosts had to be blacklisted as failover candidates.

Without that blacklist, an unplanned primary failure during the upgrade window could cause Orchestrator to automatically promote a 5.7 host as the new primary — an automated rollback that would undo hours of upgrade work and leave the system in a confused mixed-version state. The blacklist was the safety guard that prevented automation from working against the migration.

One additional discovery during testing: a replication bug in MySQL 8.0 pre-28 that only manifested under intensive long-running load. A host could eventually exhaust commit-order sequence numbers and stall. The fix was in MySQL 8.0.28. This meant every host needed to be on 8.0.28 or later — a version-pinning requirement added to an already complex upgrade matrix.

They upgraded 1,200 database hosts without a single user noticing — which means they either did extraordinary engineering or extraordinary documentation, and based on the blog post, it was both.

The Real Lesson

The GitHub MySQL upgrade is not primarily a story about databases. It’s a story about sequencing.

Every decision in the upgrade was ordered to preserve the ability to undo the previous one. Replicas before primaries. One data center before all. Validation window before decommission. Mixed-version CI before any production change. Automation library before the first cluster.

The team that wrote “upgrading the fleet with no impact to our SLOs was no small feat — planning, testing and the upgrade itself took over a year and collaboration across multiple teams within GitHub” wasn’t bragging about the engineering. They were explaining why it worked.

When you have 100 million developers depending on your infrastructure, speed is not the goal. Reversibility is.

The upgrade took a year. Zero users noticed.

That’s the whole story.

This breakdown was adapted from the full engineering case study at TechLogStack — where we retell real production incidents from Netflix, Stripe, Cloudflare, GitHub, and more in plain English. No jargon walls. Just the disaster, the diagnosis, and the fix.

Read the full technical deep-dive, including GitHub’s dual replication topology diagrams and the complete five-step playbook: **techlogstack.com/explore/github-mysql-8-upgrade-2023**

Follow TechLogStack for one real engineering disaster per week — retold in plain English, the way your smartest friend would explain it.