Part 4-The LLM Brain: Why Language Models Changed Everything

AI Agents: The Complete Picture · Part 4 of 8

Previous: Part 3:Rationality: What Does It Mean for an Agent to Be Smart?

Next: Part 5: Memory and Knowledge: RAG and How Agents Remember

Before 2017, building a system that could read a legal contract and answer questions about it took a small army of components: a parser, a named entity recogniser, a coreference resolver, a knowledge base, a query engine on top of all of it. Each piece needed its own specialist and months of tuning, and the whole stack was brittle the moment you pointed it at a new domain.

In 2023, you wrote a prompt.

That shift wasn’t AI suddenly becoming intelligent in some vague sense. It was one architecture learning to do all of those jobs from a single training objective: predict the next token. Understanding why that simple objective works-and where it quietly runs out of road-is what every agent engineer needs before building on top of it.

Why Predicting the Next Word Is Harder Than It Sounds

Predicting the next word in a legal contract reliably requires understanding syntax, domain vocabulary, and how arguments are structured in that genre. Doing it for a line of Python requires language syntax, library idioms, and some model of the programmer’s intent. Doing it for a clinical note requires something resembling medical reasoning.

The objective never changes-predict what comes next-but satisfying it forces the model to build internal representations of the structure underneath each domain. That’s why more varied training data keeps paying off: every new document is another constraint nudging those representations closer to how the world actually gets described in language.

As covered in Part 3, the model isn’t solving anything at inference time the way a search algorithm solves a maze. It’s sampling from a learned distribution over plausible continuations, built from billions of examples of competent writing and reasoning. Scale is what makes that distribution rich enough to fake expert-level reasoning across dozens of domains at once.

Tokens, Not Words

LLMs don’t process words-they process tokens, subword chunks produced by Byte Pair Encoding. Common words like “the” survive as a single token; rarer ones get chopped up (“unbelievable” → “un,” “believ,” “able”). Numbers are often split digit by digit.

This explains failure modes people find funny or alarming. A model stumbles counting the letter “r” in “strawberry” because the letters inside a token aren’t individually visible to it-“strawberry” is one or two tokens, not nine letters. Arithmetic wobbles for the same reason: multi-digit numbers split in ways that don’t preserve numeric structure. The model is pattern-matching over token sequences, not running symbolic computation. Hand it a calculator tool and the problem disappears-which is why tool use matters so much for precision tasks, a topic Part 6 covers properly.

How the Model Actually Reads: Attention

Before the Transformer, sequence models read left to right, squashing everything seen so far into one fixed-size vector before moving on. The further back a piece of information sat, the more diluted its influence became. Long-range dependencies got lost.

Attention fixed that bottleneck. For every token, the model computes a weighted sum over every other token, where the weights reflect relevance-and it does this for all tokens, in parallel, regardless of distance.

Take: “The trophy didn’t fit in the suitcase because it was too big.” Resolving what “it” refers to means relating it to both “trophy” and “suitcase,” then using “big” to break the tie. Attention assigns high relevance weights between “it” and the disambiguating words simultaneously-the resolution falls out of the weight pattern, with no hand-coded rule involved.

The Transformer architecture (Vaswani et al., 2017) stacks dozens of these attention layers. Each refines every token’s representation in light of all the others, so by the final layer, a token encodes its meaning in full context. That’s why Transformers handle context so differently from what came before, and why the architecture became the default foundation for essentially every major language model since.

When Scale Produces Surprises

A reasonable guess is that capability improves smoothly with scale. Wei et al. (2022) showed this breaks down for certain skills: some abilities are nearly absent below a threshold size and then appear abruptly once you cross it-emergent capabilities.

Multi-step arithmetic reasoning is a good example. Small models fail at it consistently; cross a certain scale and the ability to chain reasoning steps appears and stays. The leading explanation is that emergent capabilities depend on several simpler sub-skills being present at once-below the threshold, not all the pieces exist yet.

The practical consequence: you can’t safely extrapolate from a small model’s behaviour to a large model’s. Using a cheaper model as a benchmark proxy works for plenty of tasks and fails silently exactly where emergence is possible. Test the capability at the size you’re actually deploying.

The Prompt Is Part of the Program

Ordinary software behaves the same way every time on the same input. An LLM’s behaviour comes from trained weights and the prompt-swap the system prompt and, with identical weights, you get a different agent. The prompt isn’t just input; it functions as part of the program.

That reframes prompt engineering as specification writing rather than wordsmithing. A tightly specified prompt narrows the model’s output toward what you intended; a loose one leaves it guessing from incomplete signal, filling gaps with patterns that may not match what you wanted.

Chain-of-thought prompting illustrates this cleanly. Asking the model to reason step by step before answering measurably raises accuracy on multi-step problems-not because the model changed, but because each reasoning step constrains what can plausibly come next.

The same mechanism that makes prompts powerful makes them a security surface. Anything that injects unwanted content into the prompt-a user message, a retrieved document, a tool’s output-can redirect the model’s behaviour. That’s prompt injection, the single biggest security problem in deployed LLM agents today, covered in full in Part 7.

What LLMs Just Can’t Do

None of what follows is a bug waiting for a patch-these are structural properties of the architecture.

Hallucination is the one people misread most. It isn’t sloppy training; it’s a direct consequence of the training objective. A model trained to output plausible next tokens will produce plausible-sounding text even when nothing in its weights contains the right answer. It has no internal mechanism to distinguish “I actually know this” from “this is what a confident answer looks like here.” From the outside, the two are indistinguishable.

From Language Model to Agent

A language model alone is a very good text completer. It becomes an agent once you add four things: a system prompt defining its role and goals, a context window carrying history and retrieved information, tools it can call, and an outer loop that executes those calls and feeds results back into context.

The ReAct pattern (Yao et al., 2022) made that loop explicit-alternating Thought steps (reasoning about what’s known and needed) with Action steps (tool calls), each result flowing back into context until the agent reaches an answer. The same underlying model performs measurably better on multi-step questions this way than when asked to jump straight to a conclusion.

That’s also where the real vulnerability lives: the outer loop can’t verify a Thought step is correct before acting on it. A compounding reasoning error isn’t just a wrong answer once the agent has access to real systems-it can become an action you can’t undo. That’s why agent architectures need checkpoints, reversibility constraints, and human-in-the-loop gates around consequential actions, which is where Parts 6 and 7 pick up.

What Comes Next

You now have a working model of what’s inside the reasoning engine: a Transformer trained to predict tokens, reading through attention across its context, producing outputs that are fluent, plausible, and limited in specific, predictable ways. The most immediate limit is memory. Part 5 covers the standard fix-Retrieval-Augmented Generation-and where it breaks down.

AI Agents: The Complete Picture · Part 4 of 8

Previous: Part 3:Rationality: What Does It Mean for an Agent to Be Smart?

Next: Part 5: Memory and Knowledge: RAG and How Agents Remember