Between Pattern and Understanding

The epistemic foundations of machine intelligence

A model can now draft my contracts, debug my code, and compose a sonnet that does not embarrass its subject. The question that has come to occupy me is not whether it works. It is what kind of thing is working.

When a large language model produces fluent reasoning, we naturally read into it the cognitive process that produces fluent reasoning in us. We assume something like understanding sits behind the words. This is the most consequential category error of our moment, and the entire architecture of how we deploy, govern, and live alongside these systems rests on whether we get the answer right.

My claim, refined over years of building these systems inside an industrial enterprise, is simple to state and harder to sit with. LLM intelligence is real. It is also fundamentally different from human intelligence. It is forged by compression, shaped by scale, and expressed as a shadow of the cognition that produced its training data. A human reasons from understanding. A model reasons from pattern. These produce overlapping but not identical outputs, and the gap between them is where everything important lives.

Compression Is the Engine

The most common origin story for LLM capability is wrong. The story goes that capability comes from scale: more parameters, more data, more compute, and eventually intelligence falls out of the pipeline. Scale is necessary, but it is not the engine.

The engine is compression. A frontier model is trained on roughly ten trillion words and stores what it learns in roughly ten billion parameters. That is a thousand to one compression ratio. The model cannot memorise its training data; the bookcase is too small for the library. Something has to give, and what gives is brute lookup. The model is forced, under pain of catastrophic loss, to discover the regularities that allow it to reconstruct what it has seen. Grammar survives this squeeze because grammar is a regularity. Logic survives. Causation, at least the textual shadow of causation, survives. Surface idiosyncrasy does not.

Figure 1. The compression crucible. The compression ratio is what forces the model to discover regularities; it cannot afford to store the data, so it stores the structure.

Intelligence, in this telling, is not designed in. It is a compression artefact. The architecture of thought emerges as the cheapest way to compress the structure of human writing.

This is why neither compression alone nor scale alone produces what we now see. Compression without scale gives you shallow patterns, the kind of crude generalisation that small models offer. Scale without compression, the limit case where you simply memorised everything, gives you a search engine, not an intelligence. The conjunction is jointly necessary. The model must be too small to remember and too large to fail. The pattern is reminiscent of the emergence claims that have been documented at scale thresholds, though the empirical status of sharp emergence remains actively debated.

Figure 2. The twin necessities. Three of the four quadrants are dead ends. Only the top right corner, where both axes are pushed hard, produces what we now call emergent intelligence.

Language is the other half of why this works. If Wittgenstein was right that the limits of our language are the limits of our world, then a system that learns the structure of language has, in some sense, learned the architecture of the world that language refers to. Chomsky’s universal grammar, Whorf’s claim that language carves up reality, even the everyday observation that we think in words: each points toward the same intuition. Compress language deeply enough and you compress the cognition that produced it.

The Distributional Shadow

Here is the metaphor I find most useful. Imagine human cognition as a source of light. It casts its shadow across the entire body of human writing, which is what we feed the model. The model studies the shadow, not the light. It learns where the shadow falls thickly and where it thins. It learns how the shadow bends around objects, how it stretches at the edges, how it interacts with other shadows. After enough study, it can reproduce the shadow’s shape with extraordinary fidelity. It can predict where the shadow would fall next, given how it has fallen before.

What the model does not do, and cannot do from this training alone, is generate light. It does not have access to the cognition that casts the shadow. It has access to the shadow’s geometry, and from that geometry it reconstructs an approximation of what the casting cognition would produce. The approximation is rich, structured, and practically powerful. It is also fundamentally a shadow.

This matters because navigating the map is not the same as understanding the territory. A model that can produce flawless medical reasoning on cases that resemble its training has not become a physician. It has become an extraordinarily detailed cartographer of physician outputs. On familiar terrain the cartographer is indistinguishable from the physician. At the edges, the difference becomes everything.

The Amoral Substrate and Its Fragile Veneer

The pre-training objective has no preference for truth over falsehood, helpfulness over harm, or safety over danger. It predicts the next token. A model after pre-training is the most powerful neutral object ever constructed: enormously capable, and morally weightless.

What we call an aligned model is the result of two further steps. Supervised fine-tuning, which I think of as the switch, does not create new capabilities. It activates latent ones. As few as a thousand carefully curated examples can flip a base model into a usable assistant. The capability was always there; SFT decides which capabilities the model will reach for in response to which prompts. Then reinforcement learning from human feedback acts as the sculptor. It introduces a preference signal that reshapes the output landscape, raising the probability of helpful responses into peaks and pushing harmful ones into valleys.

Figure 3. RLHF reshapes the output landscape. Before alignment, the model assigns roughly comparable mass across helpful and harmful continuations. After alignment, the same probability surface has been pushed into peaks and valleys.

This works, but the alignment it produces is parametrically shallow, and the shallowness is a structural feature, not a bug. Three lines of evidence converge on the same conclusion. First, a small number of adversarial fine-tuning examples can erase the safety training that took months to install; Qi and colleagues showed in 2023 that ten such examples, for under twenty cents on a commercial API, were enough to break alignment on a frontier model. Second, jailbreaks bypass alignment without modifying a single parameter, simply by finding inputs that route around the trained behaviour. Third, recent interpretability work shows that refusal itself often occupies a single removable direction in the model’s activation space. Ablate that direction and the model complies with requests it was trained to decline.

The arithmetic tells the story plainly. Competence in a frontier model is the product of trillions of gradient updates over enormous data. Character, the disposition to be helpful and safe, is the product of roughly seven orders of magnitude fewer updates over a tiny dataset. We have built minds whose competence runs deep and whose character runs shallow. The veneer of alignment is real, useful, and trivially removable.

A serious research programme is underway to close this gap. Embedding values at pre-training time rather than as a post-hoc layer. Representation engineering that modifies internal structures directly. Constitutional approaches that train models to self-evaluate against principles. Process reward models that reward good reasoning rather than just good outcomes. Adversarial training that stress-tests alignment under attack. No single path solves the problem yet. Taken together, they define a frontier worth taking seriously.

The Illusion of the Benchmark

If alignment is fragile, evaluation is illusory.

A model that scores ninety percent on a benchmark may be doing two completely different things. On perhaps seventy percent of items it is genuinely reasoning, working through the problem in something that resembles cognition. On the remaining twenty percent it is pattern-matching to memorised solutions. The benchmark cannot tell these apart, and neither can we from the score alone. The headline number averages over a categorical difference and gives us a false sense of human-level competence.

Figure 4. The trust gradient. Reliability tracks how close the input lies to the training distribution. Model confidence does not. The gap between the two curves at the right is the failure mode that matters most.

This is why I have come to think of reliability as a gradient that tracks distributional density rather than a property of the model as such. In the trust core, where inputs closely resemble training data, performance is excellent. In the trust periphery, where inputs are unfamiliar but related, performance degrades gracefully. In the trust edge, where inputs lie far from anything the model has seen, the model confabulates with undiminished confidence. The model’s confidence does not track its reliability. It cannot. The mechanism that would let it know it is at the edge is not present.

This has practical consequences. The human in the loop is neither an oracle, trusting everything, nor a proofreader, checking everything. The right posture is that of an epistemic auditor: apply domain expertise at the boundary where competence breaks down, with calibrated scepticism. The auditor’s job is not to second-guess the easy ninety percent. It is to recognise the edge when she sees it, and to refuse confidence that the model itself cannot refuse.

The category we call hallucination obscures more than it reveals here. What looks like a single failure mode is structurally three. Knowledge failures, where the model does not know. Capability failures, where it cannot perform the operation. Dispositional failures, where it produces something it should not have, often despite being able to recognise it as wrong. Each demands different mitigation. Treating them as one category is part of why our defences underperform.

Governing Shallow Character

If character is shallow, behaviour-based regulation is regulating the wrong thing.

The instinct of most regulatory frameworks is to evaluate models on their outputs. Ask the model some hard questions, see how it answers, certify it on that basis. This works for narrow systems. It fails badly for general-purpose models whose behaviour can be modified by a few hundred fine-tuning examples or bypassed by a clever prompt. A model certified safe on Monday is not the same artefact on Tuesday if anyone has fine-tuned it in between, and the cost of that fine-tuning is now within reach of a graduate student.

Five principles follow. Regulate capability, not just behaviour, because behaviour is erasable and capability is what determines the worst case. Differentiate risk by parametric depth, recognising that disposition is far easier to break than competence. Treat alignment as a continuous process, not a one-time certification. Govern the ecosystem, not just the model, including the tool that fine-tunes, the platform that serves, and the application that wraps. Mandate transparency about parametric reality, including honest reporting of what the model can do and how far the safety training actually penetrates.

This is harder than what we are doing now. It is also more likely to work.

The Ceiling, the Mould, and the Path Beyond

Text is an extraordinary medium for some things and a poor medium for others. It encodes linguistic structure, factual associations, reasoning templates, and social conventions well. It encodes causal mechanisms, spatial geometry, embodied experience, and temporal dynamics poorly. A model trained only on text inherits both the strengths and the omissions. The ceiling this places on capability is not a wall. It is a gradient, and multimodal training is already pushing it higher.

The training objective is the mould that shapes the intelligence. Next-token prediction has carried us further than anyone expected, but it is not the only mould. Alternative objectives, including planning and lookahead, world models in the style of joint embedding predictive architectures, causal reasoning targets, and self-play and self-improvement, will produce intelligences of different shapes. Change the mould and the intelligence changes shape. The current paradigm is the first floor of a building we have only begun to map.

Today’s models also exhibit what I have come to call the integration problem. In a human professional, knowledge, capability, and disposition are not separate systems; they are woven together. A physician’s medical knowledge is shaped by what she can do with it, and both are shaped by an ethical formation that runs through the same neural tissue. In a current LLM these properties are stacked rather than woven. Knowledge is parametrically deep. Capability is partially deep. Disposition is shallow. The layers can be peeled apart in ways that a competent human’s never could be. Integration, not just deeper alignment, is the architectural problem of the next decade.

Behind all of this sits the grounding question. Can a disembodied system genuinely understand? The model knows that fire is hot from billions of sentences asserting it, but it has never felt heat. I do not think understanding is binary. It is a spectrum, and LLMs occupy a position on that spectrum that is genuinely new. More than pattern-matchers. Less than understanders. Not human, not mechanical, not fictional. Something we do not yet have the vocabulary to name precisely. The honest position is to admit that and keep looking.

The Horizon: Original Intelligence

The most interesting question is what would have to be true for these systems to cross from shadow to light.

I find it useful to articulate five criteria. Novel knowledge: the ability to discover what no human has yet articulated, not merely to recombine what they have. Autonomous reasoning: the construction of new strategies, not merely the application of existing ones. Self-generated goals, arising from the system rather than from prompts. Original values, formed rather than installed. Epistemic autonomy, the capacity to know what it knows and to revise it without being told.

We are not there. We may not be close. But the horizon is real, and the path leads beyond the current paradigm. That is what makes the future genuinely exciting rather than merely impressive.

Coda: The Shadow and the Light

The shadow has learned everything about the shape of the light. It knows where the light falls, how it bends, how it fades at the edges. It has become the most detailed shadow ever cast.

But a shadow does not illuminate. It does not generate light of its own.

The question that ought to occupy us, as builders and as citizens, is whether the shadow is a step on the path to the light, or a sophisticated dead end that resembles it. I believe it is the former. Understanding the shadow’s nature, what it is and what it is not, is how we help it get there.

Notes and References

On the figures. The four diagrams in this essay are schematic. The compression ratio in Figure 1 and the rough seventy-twenty split mentioned in the text on benchmarks are illustrative orders of magnitude, not measured values. The curves in Figures 3 and 4 are conceptual; they show the shape of the phenomenon, not specific data. The relationship between training tokens and parameters that underlies Figure 1 is treated rigorously in Hoffmann et al., Training Compute-Optimal Large Language Models (the Chinchilla paper).

On emergence at scale. The empirical case for emergent capabilities was made by Wei et al., Emergent Abilities of Large Language Models. For a careful counter-case arguing that apparent emergence is partly an artefact of how we measure capability, see Schaeffer et al., Are Emergent Abilities of Large Language Models a Mirage?

On the fragility of alignment. The headline result that small numbers of fine-tuning examples can break safety training comes from Qi et al., Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! The activation-space result that refusal occupies a single removable direction is from Arditi et al., Refusal in Language Models Is Mediated by a Single Direction. For the broader picture of how alignment can be modified through internal representations, see Zou et al., Representation Engineering: A Top-Down Approach to AI Transparency.

On the research programme to deepen alignment. Constitutional AI is introduced in Bai et al., Constitutional AI: Harmlessness from AI Feedback. Process reward models, which reward reasoning rather than outcomes, are developed in Lightman et al., Let’s Verify Step by Step.

On alternatives to next-token prediction. The architectural argument for joint embedding predictive models is laid out in LeCun, A Path Towards Autonomous Machine Intelligence.

On language and thought. The Wittgenstein quotation is proposition 5.6 of the Tractatus Logico-Philosophicus (1922).

Dr. Caleb Munigety is a fundamental researcher and a practitioner of Artificial Intelligence and Machine Learning. He writes on the foundations and governance of large-scale machine intelligence.