The Point Where One Machine Is Never Enough

Continuing the series where I share what I’m learning about cloud computing each week.

Photo by Homa Appliances on Unsplash

Imagine a startup that begins with three people in a single room. Communication is effortless. Everyone sees the same whiteboard, hears every conversation, knows exactly what everyone else is working on. There’s no coordination overhead — you just turn your chair and ask.

Now imagine that startup grows to three hundred people. Suddenly the single room doesn’t work. People are stepping on each other, waiting to access shared resources, shouting over each other to be heard. So you move to an office building — separate rooms, separate teams, structured communication. People send messages instead of tapping a colleague on the shoulder. Coordination gets harder, but it scales.

Cloud infrastructure follows exactly this arc. And understanding where “one room” stops working — and why — turns out to explain a lot about how the cloud is actually built.

Two Ways to Parallelise Work

Before getting into architectures, it helps to know that “doing things in parallel” isn’t one thing — it’s at least two fundamentally different things.

Data parallelism is when you take a massive dataset, split it into chunks, and apply the same operation to every chunk simultaneously. Processing a million photos: give each processor ten thousand, have them all apply the same filter at the same time, collect the results. The work is identical across all processors — only the data differs. MapReduce, the technique behind early Google search indexing, is the canonical example of this in cloud computing.

Task parallelism is when different processors are doing genuinely different things simultaneously. One thread is handling your web request. Another is writing to a database. A third is running an authentication check. The data may overlap, but the tasks themselves are distinct.

Both happen constantly in cloud systems. Understanding which one you’re dealing with shapes every architectural decision that follows.

SIMD and MIMD: The Hardware Behind Each Model

At the hardware level, these two types of parallelism map to two distinct processor architectures.

SIMD — single instruction, multiple data — is built for data parallelism. One instruction is broadcast to many processing units simultaneously, and each applies it to its own slice of data. Think of a military drill where a hundred soldiers perform the same movement at exactly the same moment. Every person executes the same command; only their position differs. GPUs are fundamentally SIMD machines, which is why they’re so effective at graphics rendering (applying the same calculation to millions of pixels at once) and, more recently, at machine learning (applying the same matrix operations to enormous datasets).

MIMD — multiple instructions, multiple data — is what your CPU is. Different cores can execute completely different instructions on completely different data at the same time. The kitchen analogy fits well here: one chef is making the sauce, another is grilling the protein, a third is plating desserts. Different tasks, happening in parallel, coordinating around shared goals. Most general-purpose computing — including everything running your cloud applications — is MIMD.

Getting More From Each Core: Hyperthreading

One of the more elegant tricks in modern processor design is hyperthreading (Intel’s name for what’s more formally called chip-level multithreading).

Here’s the problem it solves. When a CPU core needs to read data from main memory, it has to wait. Memory is much slower than the processor — the core can execute hundreds of instructions in the time it takes for a single memory read to complete. During that wait, the core is essentially idle, burning time doing nothing.

Hyperthreading gives each physical core two hardware threads — two complete sets of registers and execution state. When thread A stalls waiting on a memory read, the core immediately switches to thread B and does useful work while the memory system catches up. When the data arrives, it can switch back to A.

From the operating system’s perspective, a quad-core processor with hyperthreading looks like eight logical processors. You’re not getting double the raw compute power — the physical execution units are still shared — but you’re getting much better utilisation of the compute capacity that’s already there. Idle time becomes productive time.

This matters in the cloud because cloud providers want to squeeze every cycle of performance out of their hardware. The virtual machines you rent on AWS are carved out of physical servers running hyperthreaded processors. When AWS advertises a certain number of vCPUs, they’re often referring to hardware threads, not physical cores.

Load Balancing: Keeping Everyone Useful

Parallel hardware only helps if all the processors are actually doing work. The problem is that workloads are rarely perfectly even. Some tasks finish quickly; others run long. If you’re not careful, you end up with half your processors sitting idle while the other half are overwhelmed.

Load balancing is the mechanism that prevents this.

There are two basic strategies, and they mirror how work gets redistributed in human teams.

Push migration is top-down: a scheduler periodically surveys all processors, identifies which ones are overloaded and which are idle, and actively moves tasks from the busy ones to the free ones. It’s the manager who notices someone is drowning and reassigns their workload.

Pull migration is bottom-up: an idle processor goes looking for work. It scans the queues of busier processors and pulls a waiting task to handle itself. No central authority needed — the idle workers self-select into useful work.

Real systems often use both, applying push migration for gross imbalances and pull migration for fine-grained ongoing balancing. The goal in both cases is the same: maximise the percentage of time that every processor is doing something useful.

Processor Affinity: Why Moving Isn’t Always Faster

Here’s a counterintuitive wrinkle: just because a processor is available doesn’t mean moving a thread to it is the right call.

When a thread runs on a processor, that processor builds up a warm cache — a small, fast memory store holding the data the thread has been accessing recently. The next time the thread needs that data, the processor can serve it from cache in nanoseconds rather than fetching it from main memory, which takes orders of magnitude longer.

Move that thread to a different processor, and the cache is cold. The new processor has no idea what data the thread is likely to need. For the first period of execution on the new processor, every memory access is slow while the cache warms up again.

This is processor affinity — the preference for keeping a thread on the same processor it’s been running on, specifically to preserve cache warmth.

Operating systems offer two modes. Soft affinity means the OS will try to keep a thread on its current processor but makes no guarantees — it may migrate the thread if load balancing demands it. Hard affinity lets an application explicitly specify which processors a thread is allowed to run on, overriding the scheduler’s discretion.

In cloud environments, this matters for latency-sensitive workloads. A database process that keeps getting migrated around will have persistently cold caches and slower-than-expected memory access times. For applications where microseconds count, affinity settings can make a meaningful difference.

The Ceiling: When Shared Memory Stops Scaling

Shared memory multiprocessors — all the cores we’ve been discussing so far — have a hard scalability limit. At some point, adding more processors to a shared memory system stops helping and starts hurting.

The problem is the memory bus. Every processor reads from and writes to the same physical memory. As you add more processors, more of them compete for access to that shared memory system. The bus becomes a bottleneck. Processors spend more time waiting for memory access than doing actual computation. Beyond a certain number of cores, the coordination overhead outweighs the parallelism gains.

This is where the startup-in-a-room analogy hits its wall. The shared whiteboard works for three people. It doesn’t work for three hundred.

For large-scale parallel computing, you need a different model entirely.

Clusters: Many Machines, One System

A cluster is a collection of independent computers, each with its own private memory and its own operating system, connected by a network and coordinated to work together as a unified system.

The critical difference from shared memory is that processors in a cluster don’t share memory at all. Instead of reaching into a common pool of RAM, they communicate by sending messages to each other — explicitly passing data over the network. One machine says “here are my partial results” and another receives them, processes them, and perhaps sends something back.

This model scales in a way shared memory never can. You can have hundreds of machines in a cluster, thousands, or hundreds of thousands — each with its own independent memory, communicating via high-speed network. The coordination is harder to program, but the ceiling is essentially removed.

The trade-off shows up in two places.

Programming complexity. Shared memory code can be written almost like single-threaded code with some locks added. Message-passing code requires explicitly managing what data is where, who is sending to whom, and how to synchronise across machines that can’t directly see each other’s state. It’s more like writing a protocol between separate programs than writing a single program.

Efficiency. Shared memory is fast — a cache hit takes nanoseconds, and even a main memory access takes microseconds. Sending a message across a network, even a fast one, takes milliseconds. For tightly coupled tasks that need to exchange data constantly, message passing introduces latency that shared memory avoids entirely.

So why does the cloud use clusters? Because scalability wins. A shared memory system might top out at 64 or 128 cores before becoming unwieldy. A cluster can span thousands of machines. And when you need to handle millions of requests per second — or process datasets measured in petabytes — there’s no alternative.

The Cloud Is a Message-Passing System

This is the insight that reframed everything for me.

Every time a Lambda function returns data to API Gateway, that’s message passing. Every time an ECS container writes to DynamoDB and another service reads from it, that’s message passing. Every time a service calls another service’s API, that’s message passing. The cloud is, at its architectural core, a loosely coupled cluster of independent machines communicating by exchanging messages.

The same trade-off that makes clusters harder to program than shared memory systems is what makes distributed cloud applications harder to reason about than monolithic ones. Services can be unreachable. Messages can be delayed or lost. State is scattered across many independent stores. These aren’t bugs in the cloud’s design — they’re the unavoidable costs of the model that allows the cloud to scale at all.

Understanding this explains a lot of cloud architecture decisions that otherwise seem arbitrary. Why does AWS encourage you to make Lambda functions stateless? Because stateless functions are easier to scale as independent message-passing nodes. Why does DynamoDB use eventual consistency by default rather than strong consistency? Because strong consistency in a distributed message-passing system is expensive. The architecture shapes the constraints, and the constraints shape the design choices.

What Shifted

Week 8 introduced the problem of concurrency — what goes wrong when threads share data carelessly. This week pulled back to the larger picture: the different hardware architectures built to handle parallelism, the fundamental split between shared memory and message passing, and how that split maps directly onto the cloud infrastructure we’ve been building things on top of for the past several weeks.

The cloud isn’t one big computer with a lot of cores. It’s thousands of independent computers passing messages to each other — scaled-out clusters all the way down. Once you see that, the design decisions start making a lot more sense.

What’s Coming Next Week

We’re moving from how processors and clusters work to where they actually live: data centre infrastructure.

Next week is about the physical layer of the cloud — the buildings, the power, the cooling, the networking, the hardware that turns “AWS” from an abstraction into something that actually exists in the world. It’s the part of the stack that’s easy to forget about when you’re writing Lambda functions, but that shapes everything above it.

Back next week.