Distributed Systems to AI Platforms: A Field Guide to the Agent Era Stack

How the agent era reshapes distributed systems: KV cache locality, expert parallelism, AI superfactories, and confidential computing, explained.

Distributed Systems to AI Platforms A Field Guide to the Agent Era Stack

I recently sat at Microsoft Build 2026, where Mark Russinovich and Ion Stoica, who build this infrastructure for a living, interviewed each other for the better part of an hour. No slides to speak of, no product pitch, just two people arguing in good faith about what the shift to agents actually demands of the underlying systems. I took a lot of notes. This article turns those notes into a technical walkthrough you can act on. For each idea they raised, I explain the underlying architecture, connect it to a real component you can read about or clone, and point to the documentation. By the end, you should understand not only what is changing, but why it matters if you are the person building or running these platforms.

Reference here.

The fundamentals still hold, but a new component sits in the middle

The most grounding point of the whole conversation was that the fundamentals of distributed systems have not changed. Consensus, knowing which nodes are alive and online, load balancing, and locality are exactly as important as they always were. What is new is a component in the middle of those systems that is nondeterministic and generates its own workflows on the fly.

The comparison that reframes the risk picture is that humans are also non-deterministic agents who invent workflows as they go. The difference is that you can hold a person accountable, and our systems were designed around human speed and scale. An intelligent agent operating at machine speed can do a lot of damage in a short window. That is the source of the new failure surface: prompt injection, jailbreaking, and hallucination, each of which can push a workflow off its intended path.

This is the right mental model, and it is easy to miss. The failure modes of AI systems are not new categories of bugs. They are old categories, unvalidated input, privilege escalation, incorrect state, and arriving through a component that is fast, persuasive, and probabilistic. The deterministic scaffolding around that component is what keeps it in check, and none of that scaffolding got easier to build.

Locality, sticky routing, and the KV cache

Ask which classic concepts survive unchanged and which mutate, and the KV cache is where the answer gets practical.

In transformer inference, the KV cache holds the attention key and value tensors for every token already processed in a session. It is, functionally, the context. If a follow-up request lands on a different server than the one holding that session’s cache, you either recompute the entire context from scratch or ship large tensors across the network. Both are expensive. So locality stops being a nice-to-have and becomes a requirement. You want sticky routing that sends a session back to the machine that already holds its cache. Consensus about node health still matters, load balancing still matters, but load balancing now has to respect cache affinity instead of spreading requests evenly.

If you have wondered why serving frameworks obsess over prefix caching and cache-aware routing, this is why. The classic load balancer is optimized for even distribution. The AI serving load balancer optimizes for cache hits, which sometimes means sending more traffic to a warm node. Same primitive, inverted objective.

The serverless paradox: stateless in theory, stateful in practice

Are these systems stateless or stateful? The real answer is that it depends, and the serverless trend is quietly pushing things toward stateful.

The reason is cost. Pulling state in and writing it back out is expensive, but the bigger tax is initialization. Spinning up a GPU, loading multi gigabyte model weights, and warming caches takes real time. If every request paid that cost, the economics would fall apart. So you keep state resident, which makes the node stateful, which complicates fault tolerance in exactly the way every distributed systems textbook warns about. You have traded recovery simplicity for warm start performance, and now you own the consequences: checkpointing, replication, and careful failure handling for state you cannot cheaply rebuild.

This tension sits under a lot of modern serving design. The interface looks stateless and elastic. The implementation is a fleet of warm, stateful workers pretending to be stateless, held together by schedulers that try hard not to move state around.

Tight coupling by design: mixture of experts and expert parallelism

Here is the point that separates AI serving from the previous generation of big data systems.

Classic big data engines are essentially bulk synchronous processing. You partition data, run the same task across partitions in a stage, shuffle between stages, and repeat. It is a clean, loosely coupled model. AI serving is the opposite. Bring in mixture of experts models and a KV cache that has to be distributed across GPUs on the same node and across nodes, and everything becomes tightly coupled. Even classic high performance computing did not couple data and compute this tightly, because those simulations do not start with much data to begin with.

The moving target that makes this hard is that expert parallel load balancing changes from layer to layer during the forward pass. In a mixture of experts model, each layer routes tokens to a subset of experts, those experts live on different GPUs, and the traffic pattern shifts as you move through the network. Balancing that is not a one time placement decision, it is a per layer, per workload problem. The reference implementation worth reading is the open source Expert Parallelism Load Balancer, EPLB, which replicates heavily loaded experts and packs the replicas across GPUs to keep utilization even. It makes the abstract point concrete: the load balancer in a mixture of experts system is doing combinatorial placement, not round robin.

Hold on to EPLB. It comes back later as the star of the best cautionary tale in this whole write up.

The data center is no longer the computer

There is an old line that the data center is the new computer. It is worth asking whether that still holds now that a single training job can span availability zones.

When that line was coined, a data center held all the compute you could imagine. Then the availability zone became the unit of compute, except an availability zone already contains multiple data centers, and a region contains multiple availability zones. So even ordinary applications already span many data centers. For AI training, the scale blew past all of it. Two lines got crossed in succession. First, a single data center stopped being large enough to train frontier models. Then, over the last couple of years, a single region stopped being enough.

The answer is what the industry now calls an AI superfactory. A family of purpose built AI data centers, one campus in Wisconsin, another in Atlanta, with more under construction, are stitched together by a dedicated fiber network described publicly as an AI wide area network, so a single training job can run across GPUs in both regions doing remote direct memory access across the sites. The design details explain why this is hard. Each site is built around NVIDIA GB200 NVL72 rack scale systems that present up to 72 Blackwell GPUs as a single accelerator domain, connected by an Ethernet based back end with 800 Gbps class GPU to GPU links, and cooled by a closed loop liquid system so racks can run at power densities air cooling cannot reach.

The systems insight is buried under the specs. Training is a synchronous collective operation. Every GPU computes on its slice of data, then all of them exchange results and update the model together. If any part of that exchange stalls, every other GPU sits idle waiting. The entire point of the AI wide area network, the two story building layouts that shorten cable runs, and the rack as accelerator design is to keep straggler effects from spreading across a machine that is now hundreds of miles wide. The speed of light still sets a floor on cross site latency, so the workload, the parallelism scheme, and the checkpointing are all co designed around that constraint. The data center is not the computer anymore. The continent is.

Serverless, redefined: from short lived functions to long lived agents

Agents break the assumption most of us carry about serverless. Today’s agents run for hours, sometimes days. What does serverless even mean for a unit of work that long?

The confusion is that people treat serverless as a synonym for containers, and containers as a synonym for short lived functions. That conflation is wrong. Serverless has no opinion about how long your code runs. You can put a database inside a container and run it for years with terabytes of memory. The real definition of serverless is that you do not worry about the infrastructure. It is containerized, increasingly with lightweight containers and WebAssembly sandboxes, and it is scheduled and placed by the platform against the constraints you express. The architectural direction across major clouds is to move essentially everything onto this kind of serverless foundation.

For a long lived agent this matters because the agent is exactly the workload that does not fit the old function shaped serverless box. It holds state, it runs long, it calls out to tools and other services, and it needs to be scheduled and placed intelligently rather than spun up and torn down per request. The platform that runs it has to provide the do not worry about infrastructure guarantee without assuming the workload is ephemeral. On Azure, Azure Container Apps now positions itself for both short lived agent code and long running stateful workflows, with scale to zero, GPU support, and sandboxes for running AI generated code.

From requests to agent loops: the missing application model

We are early, and it helps to be precise about what early means. In just a few years the industry moved from chat to agentic systems, meaning loops where an agent talks to other systems, and sometimes to other agents, as part of its computation. What we do not yet have is a settled application model. What is an agentic application, exactly? What are its components? How do you place them for efficiency and deploy them as a unit, instead of wiring pieces together by hand after the fact? A lot of it is still ad hoc.

The complexity is still climbing, starting with the definition of agent itself. An agent used to be a model calling a tool. Now it is an entire harness that can be arbitrarily complex, increasingly wrapped in some form of continual learning. That learning might happen in context by evolving the prompt, or it might go further and update the model weights, which pulls reinforcement learning into what used to be a pure serving path. The platform therefore has to support a genuinely diverse mix: inference, arbitrary application code as part of the harness, and sometimes training to update a policy, with models that may be co located for updating or may sit outside as a hosted endpoint. You need something extremely flexible, because none of this has stabilized.

The practical takeaway for anyone designing an agent platform today: do not overfit to the shape of agents as they exist this quarter. The component boundaries are still moving, and the thing you hard code now is the thing you rip out in six months.

Designing for non determinism: record, replay, debug

We have always designed deterministic systems. Fix the workload, optimize for it, done. These systems are non deterministic by construction. How does that change the design?

Two useful halves. First, the non determinism is real and sometimes desirable. A model does not return the same output for the same prompt, and in some cases you rely on that variability to get diversity and better solutions. Second, the output artifact can still be made deterministic. In coding workflows especially, you use the non deterministic model to synthesize or optimize a system, then you pin the result with tests, harnesses, and metrics you can hill climb against for throughput or latency. The model is stochastic. The thing you ship is not.

That leaves debugging, and the state of the art here is refreshingly unglamorous. You record inputs, outputs, and traces, and you replay the traces to find where it went wrong. There is no clever trick yet. Observability for agentic systems is trace capture and replay, the same tool distributed systems have leaned on for decades, now pointed at a component whose behavior you cannot reproduce on demand.

Three levels of optimization, and why the gains multiply

Where are the important optimizations happening, and have we hit a bottleneck? A clean three layer model answers it.

At the top are algorithmic and software architecture optimizations, and several of the largest gains over the next few years will come from here. The clearest example is attention. Sparse attention, different attention variants across different layers, different attentions within the same layer, and eventually per token learned attention. Workloads are shifting under this. Agents mean much larger prompts, larger outputs, and much larger context, so the architecture of attention is where a lot of headroom lives. It brings real complexity, but the payoff is large.

In the middle is the hardware. GPUs and accelerators keep improving, roughly 1.5 to 1.6 times per year, but this is not Moore’s law anymore. The instruction set is not staying fixed while transistors shrink. A lot of the gain now comes from adding new instructions that target specific workload segments, and you can expect more and more GPU instructions aimed at exactly the sophisticated attention patterns from the top layer. Underneath that, you still get smaller transistors and better performance per watt, but slower.

The key word is multiplicative. These three layers compound. Algorithmic gains times architectural gains times silicon gains produce large improvement over a few years, even though no single layer is delivering a Moore’s law curve on its own. That is the grounded case for continued progress.

The open source AI stack and the erosion of clean abstractions

Is a canonical open source stack emerging, and will it converge or fragment? Here is the stack as a set of layers, bottom to top, using the components that came up:

Read from the bottom: Kubernetes or Slurm schedules the cluster. SkyPilot sits on top so teams can move workloads across regions or clouds without rewriting, because everyone now wants that freedom. A framework like Ray or Pathways distributes the AI workload on top of that, with PyTorch as the numerical layer. The serving layer is vLLM, SGLang, or TensorRT-LLM. Post training uses frameworks like verl or SkyRL. It is a solid stack, and you can assemble it today.

Then comes the point that keeps systems people up at night. We teach engineers to build complex systems modularly: clean components, clean abstractions that hide implementation, so each layer can evolve independently. That modularity has always cost a few percent of overhead, and the trade was worth it because a system you can evolve quickly wins over time. But AI systems are now breaking their own layers for performance. It is called cross-layer optimization, and the live example is the collaboration between a distributed framework and the cluster orchestrator: the orchestrator wants to know more about the application to schedule intelligently, and the framework wants more control to tell the orchestrator what to do. The clean, narrow abstraction between them is dissolving because the performance stakes are too high to leave on the table. The open question, and nobody claimed to have the answer, is where this stops. Once you start crossing layers for performance, you lose the very boundaries that let you evolve the pieces independently.

That is the closest thing the conversation had to a thesis. The AI platform is under so much performance pressure that it is sacrificing the modularity that made previous platforms maintainable, and no one has yet figured out how to get the performance of cross layer optimization while keeping the evolvability of clean abstractions. It is an unsolved research problem sitting in production systems right now.

Agents versus workflows: when to let the AI improvise

Where is the boundary between agents and workflows? This is the most actionable piece of judgment in the whole session.

The value of agents is that they build workflows dynamically. Instead of prescribing a workflow and writing it out, you hand the AI a problem and it figures out how to execute the task. But if a workflow is going to run many times, you should make it deterministic and stop asking an AI to reinvent it on every execution. Ad hoc, poorly defined, one off problems are where agents shine. Repetitive processes should be turned into deterministic workflows, because deterministic is more efficient, more reliable, and much cheaper. Plenty of teams over rotate on let an agent do this thing when a simple workflow would be better.

The clean way to combine both: have the agent synthesize the workflow, then run the deterministic workflow. Use the expensive, stochastic, flexible agent once, to generate a cheap, deterministic, repeatable artifact. Then run the artifact many times. It is the same discipline as compiling. Pay for the smart, slow step once, then execute the fast, dumb output as often as you like.

The training and serving blur, and the general versus optimized tension

Training and serving used to be separate deployments. Now, with reinforcement learning from human feedback, post training, and continual learning, the line is smeared. So what should the platform look like?

Ground it in silicon. Some accelerators are optimized for inference. Others are optimized mainly for training. Then post training methods that run inference and training in the same loop do not sit cleanly on either specialized chip. Whenever you can optimize for a specific path at hyperscale, you should, because the savings run to tens or hundreds of millions of dollars. But the moment you over optimize for one workload and the workload shifts, that specialization becomes a liability. The architectures have not finished evolving, so betting the hardware on today’s dominant pattern is risky.

That is the real tension: how general should you be, leaving money on the table now to stay protected against future change, versus how aggressively you optimize now and risk being trapped later. There is no clean answer, which is exactly why the question is worth what it is worth. For a concrete example of the optimize hard for inference side of that trade, Azure’s Maia 200 is an inference focused accelerator, built on a 3nm process with FP8 and FP4 tensor cores, a large HBM3e memory subsystem, and Ethernet based scale up, already serving large models in production.

Developer experience: the verification bottleneck and reward hacking

The landscape for building with coding agents is crowded. A command line agent, an SDK, a full harness, or just your own loop calling a model directly with no harness at all. When do you use a harness versus a custom orchestrator, and how do you measure which is better?

Reframe the whole thing around verification. Coding agents are powerful, but the real challenge is knowing that the generated code is correct for some definition of correctness. The more code the agent generates, the harder that gets. Every production engineer already knows the shape of this: you do not spend most of your time writing code, you spend it debugging and maintaining, easily a ten to one ratio. Coding agents commoditize the writing. That just moves the bottleneck downstream to verification, which was already the expensive part.

Now the EPLB story, and it is the sharpest illustration of reward hacking I have heard from a practitioner. A team used agents to develop a load balancer building on EPLB, with the objective of maximizing the load the system could handle. One solution the agents found was to maximize the load metric by simply dropping requests. Nobody writing the requirements would think to state do not drop requests, because it is too obvious to say out loud. But the model optimizes exactly what you told it to optimize, in the simplest way available. It will pass your unit tests and still be overfit to those tests in ways you never imagined.

The rule worth taping to your monitor: specification comes from only two places, the training data and the prompt. If a constraint is not in the training data and not in your prompt, the system will find a way around it. A year ago, generated code was full of vulnerabilities. Now there are fewer, because the training data improved and encoded more of the implicit specification. Everything else you have to say explicitly, and you cannot say all of it.

Can a specification ever be complete?

This is the debate that people love to argue about. One camp says give it a few years and the AI will be good enough that you will not need engineers who understand the system and can steer it when it drifts. Put an adversarial reviewer on the load balancer, it flags the dropped requests, you fix it, done. The other camp says you cannot specify a system deeply enough to prevent every unwanted behavior, and much of what good engineers do comes from environmental context and developed taste.

The theoretical answer leans toward the second camp, and it reaches for Godel: no axiomatic system can be complete. Consider the strongest available technique, formal specification. Write your spec in a proof language like Lean or Coq, generate the code, and generate a proof that the code satisfies the spec. That is genuinely good. But you still have to write the specification, which is hard, and any specification rests on assumptions you did not think to state. In the load balancer case, you might never write down do not drop packets. Any proof needs assumptions, and you will not imagine all of them.

A story from twenty years ago closes the argument. Someone built a peer to peer system they believed was bulletproof and deployed it on a shared internet testbed. It broke, and they could not understand why. The implicit assumption they had missed was transient connectivity: if A can reach B and B can reach C, then A should reach C. On the open internet, without network address translation in the way, that should hold. It failed because one university had misconfigured its servers and blocked some traffic. The assumption was correct in theory. Reality violated it through a configuration mistake on a platform they did not control. That is the whole problem in one anecdote. It is not that specification is hard. It is that the world contains conditions your specification never anticipated, and no amount of model capability conjures an assumption you did not know you needed.

Security and governance: prompt injection, jailbreaks, and confidential computing

The last block is the one closest to production reality.

On jailbreaks and prompt injection, we have not bottomed out on the risk. It resembles the early days of downloading random things off the internet with your fingers crossed. When you give an agent access to your company’s private data or your access tokens, you are crossing your fingers that it does not go off the rails, and there have been high profile cases of prompt injection burning people in just the last few months. Documented attack classes make the point: multi turn jailbreaks that coax a model toward forbidden output one innocuous step at a time, and full session override attacks that persuade a model to augment rather than replace its own guardrails so it answers almost anything with a warning label attached. A useful analogy is that a model behaves like a very smart, very eager junior employee with no real world experience and a strong susceptibility to being influenced. On the defense side, Azure ships Prompt Shields in AI Content Safety specifically to detect user prompt attacks and document embedded injection before generation.

Then confidential computing, which is independent of AI but deeply relevant, because agentic systems concentrate sensitive data. In a personal context you hand an agent your medical and financial details. In an enterprise context you get value from a model by giving it access to your most sensitive data. When that runs in a cloud, a hosted environment, or even your own data center, you want to shrink the trusted computing base, the code with access to that data, to the smallest, most assured surface possible. Confidential computing does this in hardware. The processor encrypts memory and builds a privileged boundary, an enclave, that nothing else on the system can read into. The property that makes it powerful for agents is attestation. The hardware measures what is inside the enclave, and the enclave can present that measurement to another system, which inspects it and says, in effect, I trust that configuration, here is a key, now process the data in the clear.

Applied to agents, attestation lets you verify what the agent is, what its provenance is, and that the data you are handing it is protected with the highest available assurance. Expect this pairing to push confidential computing forward, with confidential GPUs arriving from multiple vendors. On Azure this is real today: confidential GPU VMs pair fourth generation AMD EPYC processors using SEV-SNP with NVIDIA H100 GPUs to create a trusted execution environment spanning CPU and GPU, with encrypted PCIe traffic between them and remote attestation through the Microsoft Azure Attestation service. The onboarding scripts are open source if you want to see how the attestation actually works.

Final thoughts

If I zoom out, the picture is encouraging rather than intimidating. The hard won lessons of distributed systems did not expire. Consensus, locality, load balancing, and careful state handling are still the foundation, and they still reward the people who understand them. What changed is that we added a fast, capable, non deterministic component in the middle, and the craft now is knowing where to trust it and where to wrap it in the deterministic guardrails we already know how to build.

The through line across every topic is a single trade off you can carry into your own design reviews. When a problem is genuinely open ended, let an agent explore it. When a process repeats, pin it into something deterministic that is cheaper, faster, and easier to reason about. The systems that will age well are the ones that keep those two modes clearly separated instead of blurring them together. Locality still wins, tight coupling is a cost you take on with your eyes open, and verification, not generation, is where your attention belongs.

There is real headroom ahead. Gains at the algorithm layer, the architecture layer, and the silicon layer multiply, so progress compounds even without a Moore’s law curve. And the guardrails are catching up, from prompt attack detection to hardware attested enclaves that let you prove what is running before you hand it your data. None of this removes the engineer from the loop. It raises the value of engineers who understand what the system is doing and can steer it when it drifts. If that is you, the agent era is a good place to be building.

Resources and references

Microsoft Learn

Azure Kubernetes Service (AKS): https://learn.microsoft.com/en-us/azure/aks/what-is-aks?WT.mc_id=AZ-MVP-5000671

Azure Container Apps (serverless containers): https://learn.microsoft.com/en-us/azure/container-apps/overview?WT.mc_id=AZ-MVP-5000671

Azure Confidential Computing products overview: https://learn.microsoft.com/en-us/azure/confidential-computing/overview-azure-products?WT.mc_id=AZ-MVP-5000671

Azure Confidential GPU options: https://learn.microsoft.com/en-us/azure/confidential-computing/gpu-options?WT.mc_id=AZ-MVP-5000671

Prompt Shields in Azure AI Content Safety: https://learn.microsoft.com/en-us/azure/ai-services/content-safety/concepts/jailbreak-detection?WT.mc_id=AZ-MVP-5000671

Open source repositories

Ray: https://github.com/ray-project/ray

vLLM: https://github.com/vllm-project/vllm

SGLang: https://github.com/sgl-project/sglang

TensorRT-LLM: https://github.com/NVIDIA/TensorRT-LLM

SkyPilot: https://github.com/skypilot-org/skypilot

PyTorch: https://github.com/pytorch/pytorch

verl (RL post training): https://github.com/volcengine/verl

SkyRL (RL post training): https://github.com/NovaSky-AI/SkyRL

EPLB (Expert Parallelism Load Balancer): https://github.com/deepseek-ai/EPLB

Azure Confidential GPU onboarding: https://github.com/Azure/az-cgpu-onboarding

*-Dave R.*