The Database That Broke the Rules: Inside Google Cloud Spanner

The Database That Wasn’t Supposed to Exist

How Google Cloud Spanner pulls off something distributed systems said was impossible, and what our case study revealed about the real cost of getting it right

For years, building large-scale databases came with an uncomfortable rule: you had to choose.

You could have a proper relational database. SQL, ACID transactions, strong consistency. But it would eventually hit a ceiling. One server can only grow so big, and when your application outgrew it, you started manually splitting your data across multiple databases. Engineers call this sharding. It worked, but managing it was a nightmare, and somewhere in the process you quietly gave up global consistency.

Or you could go the NoSQL route. Cassandra, DynamoDB, Bigtable. Massive horizontal scale, but no relational structure, no joins, and you often settled for eventual consistency. Your users might briefly read stale data after a write. You accepted that, because the alternative was worse.

The academic term for this dilemma is the CAP theorem. It says a distributed system can only fully guarantee two out of three things: Consistency, Availability, and Partition tolerance. For a long time, people treated this as a hard ceiling. You picked your two properties and engineered around the one you gave up.

Google Cloud Spanner largely ignored that ceiling.

That is not a marketing claim. It is the reason the 2012 OSDI research paper introducing Spanner became one of the most referenced papers in distributed systems. For our SE3030 Software Architecture case study at SLIIT, our group spent several weeks working through that paper alongside Google’s documentation, real-world benchmarks, and competitor analyses. We wanted to understand not just how Spanner achieves what it does, but what it gives up to get there.

This article is that study, written for anyone who wants to understand it without a research background.

Why Google Built It

The story starts in 2007, and it starts with a problem.

Google was running on Bigtable, a proprietary key-value store that was excellent at scale but could not do multi-row transactions. If you needed to update two rows atomically, where either both succeed or neither does, Bigtable could not guarantee that. Megastore was built to fix this, layering transactions on top of Bigtable. But Megastore was slow, hard to use, and struggling under Google’s internal workloads.

So Google built Spanner.

The first real stress test came through a system called F1, Google’s SQL engine for AdWords, which generates most of Google’s revenue. F1 needed to handle billions of rows, thousands of transactions per second, and global distribution across datacenters on multiple continents, all while giving engineers a familiar SQL interface. Spanner powered it. The fact that AdWords did not collapse is your proof of concept.

In 2012, Google published the OSDI paper. That paper is the architectural blueprint for everything in this article. In 2017, Spanner became publicly available on Google Cloud. In 2020, it gained a PostgreSQL-compatible interface so companies could migrate without rewriting application code. In 2024, a Cassandra-compatible interface followed.

Each compatibility layer is Google saying: whatever you are running right now, we have built you a migration path. That is not purely a technical decision. It is a market strategy aimed directly at Spanner’s two biggest competitors.

Three Ideas That Make It Work

Spanner’s architecture sits on three core concepts. If you understand these three things, you understand the system.

Splits

Every table in Spanner is divided into splits, which are just contiguous ranges of rows. If your table has user records keyed by ID, user IDs 1 to 1,000 might form one split, 1,001 to 2,000 another, and so on.

Think of it like cutting a long book into chapters so multiple people can read different sections at the same time.

Spanner manages splits automatically. If one grows too large, Spanner breaks it. If one becomes a hotspot because too many reads or writes are hitting the same range, Spanner redistributes it across nodes. None of this requires any changes to your application. You write SQL exactly as you always have.

There is one important thing to get right though. If your primary key is a timestamp or an auto-incrementing integer, all your newest writes land on the same split. That split gets overwhelmed while the others sit idle. The fix is to use UUIDs or hash-based keys so writes are spread evenly. This is the difference between a system that scales the way Spanner promises and one that bottlenecks under real load.

For related data, Spanner also supports interleaved tables, a performance tactic where child rows are stored physically close to their parent rows. This reduces cross-split joins for data that is frequently accessed together, and it makes a meaningful difference at scale.

Paxos Replication

Each split is copied across multiple zones. Typically three in a regional deployment, five in a multi-region one. Within each split’s group of copies, one is elected the leader using an algorithm called Paxos. The leader handles all writes for that split. The others serve reads and participate in the voting that confirms each write.

Spanner uses three types of replicas. Read-write replicas hold a full copy of data, vote in Paxos, and can become leaders. Read-only replicas serve reads but do not vote, so adding them scales read capacity without affecting how many replicas you need for a write quorum. Witness replicas vote in Paxos but do not store data, providing quorum membership at a lower storage cost.

The clever part of all this is that different splits can have their leader in different zones at the same time. Split 0’s leader might be in Zone A, Split 1’s in Zone B, Split 2’s in Zone C. No single zone carries all the write traffic. Load is spread across the infrastructure by design.

When a zone fails, the surviving replicas elect a new leader. According to the original OSDI paper, this typically completes within 10 seconds for a zonal failure and under 60 seconds for a regional one. No human intervention needed.

TrueTime

This is the most unusual part of Spanner’s architecture, and the one that required Google to build custom hardware.

The problem is a fundamental one. If you have databases in Singapore, London, and Sao Paulo, and a user writes a record in Singapore, how do you guarantee that a user reading that record from London a millisecond later gets the updated version? Without a global lock, which would destroy performance, or eventual consistency, which means you might read stale data, there is no obvious answer.

TrueTime solves this by giving every Spanner node a globally synchronised clock with a known, bounded uncertainty. Google built this using GPS receivers and atomic clocks together, each correcting for the failure modes of the other.

Before committing a write, Spanner assigns it a timestamp and waits out the uncertainty window, typically around five milliseconds, before confirming the commit. This is called commit wait. By the time the acknowledgement reaches your application, the write’s timestamp is definitively in the past everywhere in the world. Any read from anywhere in the system will see this write in the correct order.

The result is what Google calls external consistency. If transaction B starts after transaction A commits, B is guaranteed to see A’s writes. Always. Globally. Without locks.

The cost is that five-millisecond commit wait on every write. For most applications it is completely invisible. For systems needing sub-millisecond write confirmation, it is a real constraint worth knowing about before you commit to the platform.

Scalability: The Numbers Behind the Claims

When we looked at Spanner’s scalability in our case study, the numbers that stood out were not from synthetic benchmarks. They were from Google’s own internal usage.

Spanner handles over 6 billion queries per second across Google’s global infrastructure. More than 2,000 internal Google applications rely on it, including Gmail, the Play Store, and Google Search. The data it manages sits at exabyte scale, far beyond what any single database cluster could handle.

These figures matter not just as statistics but as evidence that the architecture actually holds at the scale it claims to target.

The scalability comes from three compounding design decisions.

Colossus decoupling. Spanner’s storage runs on Colossus, Google’s distributed file system, completely separate from the compute nodes. When Spanner needs to rebalance splits across nodes, it does not move any data. Only the serving assignment changes. This is why adding capacity is fast. You add nodes and Spanner reassigns splits to them almost immediately.

Linear node-to-capacity scaling. Spanner’s throughput scales approximately linearly with node count. Double your nodes and you roughly double your capacity. That near-linear relationship is rare among distributed databases. Most systems hit coordination overhead that causes throughput to plateau as you scale.

Write batching. Multiple writes are grouped into single Paxos rounds where possible, reducing round-trip overhead for high-throughput workloads. You can tune this depending on whether you prioritise raw throughput or low latency per individual write.

In real-world terms: Mahindra used Spanner to handle over 100,000 bookings in 30 minutes during a peak product launch, with zero downtime and zero manual intervention. Google Ads migrated from sharded MySQL to Spanner and cut database management time by 80 percent. Both cases demonstrate the core promise: scaling is not something you engineer around. It just happens.

Availability: What the SLA Actually Means in Practice

Availability is not abstract when you look at what downtime costs across industries.

Banking loses around $1.25 million per hour when systems go down. E-commerce loses $690,000 per hour on average, and significantly more during peak events. Healthcare systems lose around $636,000 per hour. Telecom sits at $560,000 per hour.

Against that backdrop, Spanner’s availability guarantees become concrete:

Deployment SLA Downtime per year Downtime per month Single-region 99.99% ~52 minutes ~4.4 minutes Multi-region 99.999% ~5.2 minutes ~26 seconds

The mechanism behind the multi-region guarantee is a five-step fault tolerance flow. A failure is detected by heartbeat monitoring. The Paxos quorum recognises the leader is unreachable. Surviving replicas vote for a new leader. Traffic is redirected. Service is restored, typically within 10 seconds for a zonal failure and under 60 seconds for a regional one.

Even during that failover window, TrueTime commit waits ensure that transaction ordering is never violated. Consistency survives leadership transitions intact.

For read-heavy workloads, Spanner also offers bounded staleness reads. You opt in to accepting data that may be a few seconds old, and in return you get significantly lower latency because any replica can serve the read without querying the leader. For applications where perfect freshness is not always required, this is a useful tool for managing cost and performance together.

The Trade-offs: What Spanner Gives Up

Every architectural decision has a cost. Spanner is not an exception.

Write latency is inherently higher. Every write requires quorum acknowledgement across replicas plus commit wait. For single-region deployments this adds roughly 5 to 10 milliseconds. For cross-region writes, the speed of light adds another 40 to 100 milliseconds between distant regions. For most applications this is completely acceptable. For high-frequency trading or real-time gaming scenarios, it is a dealbreaker.

Cost scales steeply. Multi-region Spanner is three to five times more expensive than comparable single-region alternatives, and significantly more than self-managed PostgreSQL or MySQL. The managed infrastructure, global replication, and custom hardware are all priced into the service. This is the most common reason organisations look at alternatives even when they technically need what Spanner offers.

Schema design still matters. Automatic sharding only distributes load evenly if your keys are distributed. Sequential primary keys funnel all new writes to the last split and create a hotspot. The system does not protect you from poor key design choices.

Vendor lock-in is real. The PostgreSQL and Cassandra interfaces reduce migration friction, but TrueTime and Spanner’s multi-region configurations have no direct equivalent anywhere else. Moving off Spanner is possible, but it is not a weekend project.

How It Compares to the Alternatives

Our case study compared Spanner against four alternatives across scalability and availability dimensions.

Sharded MySQL was the industry standard before systems like Spanner existed. You partition data manually across multiple MySQL instances and write application-level logic to route queries to the right shard. It works, but every schema change becomes a multi-shard operation, cross-shard joins are painful or impossible, and global consistency requires careful application design. Google Ads ran on sharded MySQL before Spanner. The migration cut database management overhead by 80 percent, which tells you most of what you need to know.

Cassandra is built for write throughput and availability at scale. It uses hash-based partitioning and tunable consistency, letting you choose how many replicas must acknowledge each write. The trade-off is eventual consistency by default, no SQL joins, and no ACID transactions across multiple rows. For write-heavy workloads where eventual consistency is acceptable, Cassandra is excellent. For anything requiring relational structure or strong consistency, it is the wrong tool.

CockroachDB is the closest architectural rival. It is an open-source distributed SQL database directly inspired by the 2012 OSDI paper. It uses range-based partitioning and the Raft consensus algorithm for strong consistency. The key difference is that CockroachDB runs anywhere, on any cloud or on-premise, while Spanner requires Google Cloud. For teams that need distributed SQL without vendor lock-in, CockroachDB is the most architecturally similar alternative available.

Aurora Global Database from AWS is the strongest cloud-platform alternative. It offers strong global reads and solid disaster recovery, but its architecture uses a single primary region for writes with asynchronous replication to secondary regions. Reads can be served globally with low latency, but writes stay in one region. That is a meaningful difference from Spanner, where write leaders are distributed across zones globally. Aurora is the right choice when read scalability matters more than global write consistency.

System Scalability Availability Best when Spanner Auto splits, read scaling Sync replication, 99.999% multi-region Global correctness and uptime both matter CockroachDB Range splits, rebalancing Raft consensus, locality controls Distributed SQL without GCP lock-in Aurora Global DB Strong global reads, single-primary writes Regional HA, async cross-region Read scale matters more than write consistency Cassandra Hash-based, high write throughput Tunable, eventual by default Write-heavy, consistency can be relaxed Sharded MySQL Manual sharding Single-master per shard Simple, low-cost, single-region

Who Should Actually Use It

Spanner makes sense when your application serves users across multiple regions and consistency across those regions is non-negotiable. Financial systems, healthcare records, global identity platforms. When your SLA requirements are at or above 99.999% and you cannot afford to engineer that reliability yourself. When you need online schema changes on tables with billions of rows without any downtime. When you are already on Google Cloud and the managed operations value justifies the cost premium.

Spanner does not make sense when your application is single-region and a well-tuned PostgreSQL handles your load comfortably. When budget is a primary constraint because the cost differential is substantial. When your write workload requires sub-millisecond latency. When your organisation needs multi-cloud or on-premise deployment flexibility.

The honest summary is this: Spanner wins when correctness and uptime matter more than lowest cost. The main barriers are cost, GCP lock-in, and the need for deliberate schema design upfront. If you can accept those three constraints given your requirements, there is no distributed database that currently matches what Spanner offers.

What This Study Taught Us

We came into this case study knowing the surface description of Spanner. Globally distributed, strongly consistent, fully managed. What we did not expect was how tightly all the design decisions connect to each other.

TrueTime is not just a clock mechanism. It is the reason external consistency is possible without global locking, and it required Google to build and operate custom GPS and atomic clock hardware in their datacenters. The Colossus decoupling is not just a storage choice. It is what makes horizontal scaling fast enough to be practically useful. The Paxos leader distribution across splits is not just a load balancing trick. It is what allows different parts of your data to tolerate failures in different zones simultaneously.

Every piece of the architecture exists to solve a specific problem that the other pieces create.

The trade-off analysis was equally useful. Spanner does not give you something for nothing. Write latency, financial cost, vendor dependency, schema design discipline. These are all real prices. The question an architect must answer is not which architecture has no downsides, but which trade-offs are acceptable given what this system actually needs to do.

That framing turned out to be the most transferable thing we took from this study. It applies well beyond database design.

References

• Corbett et al. (2012). Spanner: Google’s Globally-Distributed Database. OSDI 2012.
• Shute et al. (2013). F1: A Distributed SQL Database That Scales. VLDB 2013.
• Bacon et al. (2017). Spanner: Becoming a SQL System. SIGMOD 2017.
• Google Cloud Spanner Documentation: cloud.google.com/spanner/docs
• Spanner SLA: cloud.google.com/spanner/sla
• TrueTime and External Consistency: cloud.google.com/spanner/docs/true-time-external-consistency
• CockroachDB Architecture: cockroachlabs.com/docs/stable/architecture/distribution-layer
• Amazon Aurora Global Database: docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/

This article is based on a case study completed for SE3030 Software Architecture at SLIIT as part of a BSc (Hons) in Software Engineering. Group SE_53(2026 JAN-JUNE)