Is Probability a new pulpit?

From sin via fake news to your company’s wellbeing dashboard, it may appear that whoever “names” the world controls it. Have LLMs inherited the new naming of everything?

In 42 seconds

Naming has always been how power works. Call something sin, heresy, witchcraft, fake news or wellbeing, and you create a category somebody (with power) gets to administer. The word feels like a description; it is actually a jurisdiction. Tom Griffiths’ new book shows that probability, not rules, is what captures language and maybe thought, and that is the unsettling part: rules can be argued with, but probability just feels like reality. LLMs now shape that probability in a billion private conversations at once, with no shared text to contest. Whoever shapes what feels normal no longer needs to make rules, and nobody is measuring it.

A question developed for me: “Was it possible to sin before we had a word for sin?” as I read Tom Griffiths’ new book The Laws of Thought: The Quest for a Mathematical Theory of the Mind. He traces 300 years of attempts to capture thinking in mathematics through different frameworks: rules and symbols, neural networks, and probability. Tom runs Princeton’s AI Lab, and his account of what these frameworks prove or do not is why this article exists. It is a fantastic book, I love the maths and would highly recommend it, but it is not for the faint-hearted who want a summary, answer or quick framework about what to do next. The value is in the struggle.

When I finished the book, Tom left me with an itch: if probability is the framework that captures thought, then what happens to the other question I debated at uni 40 years ago about who shapes language, as “words” in this new AI world just become distributions.

This is how I arrived at “sin and vice”. The “act” of sinning was always possible. People struck each other, betrayed each other, coveted and desired long before anyone wrote the rule, code or word. What the word “sin” created was the category (ontology), and the category is where the power lives. You cannot be forgiven for something that has no name, which means whoever holds the vocabulary of forgiveness holds something with infinite demand and controlled supply: an excellent market for exploitation. A believer can hold that sin describes something true about the human condition and still observe that administering the category built an institution. Words such as “sin” create a legal framing or jurisdiction, whatever the territory underneath it.

I am back from a week walking, and I kept pulling on this thread, much to the annoyance of my fellow walkers, but it leads somewhere I think anyone dabbling in or debating AI ought to be paying attention to, because the machinery that made “sin” work is the same machinery that makes large language models work.

The thing we built by “accident”

The success of LLMs is told as an engineering story: scale plus compute plus data, but buried in it is the one Tom Griffiths’ three frameworks focus on. We spent 50 years building language systems out of rules, syntax, grammar, and formal logic, and they were brittle, narrow and mostly went nowhere. Then we started building models that treat language as nothing but probability (weights), and these new models write better than rule systems ever did, and their fluency is now indistinguishable from a human.

Tom was careful, and so must I be, about what that “proves”. A model that reproduces the surface of language has not necessarily captured what language or understanding is, any more than a flight simulator proves aeroplanes are made of equations. For example, the “Noam Chomsky” tribe has spent 60 years arguing that performance and competence are different things, and they have evidence; these models can learn impossible languages, sequences no human grammar would permit, about as easily as real ones. What has been settled is that the statistical surface of language carries enough of its own probability to transmit it, reproduce it, and now manufacture it at scale. The rules and power that the “grammar kings” wrote down and made us follow to exercise control were always descriptions after the fact; we now see that distribution came first.

Child development research appears to confirm this, but not in the order I expected when I started reading. I assumed probability arrives last, post mastery, at the point where native speakers stop consulting rules and simply feel that a sentence works. It turns out babies are tracking transitional probabilities between sounds months before anything resembling grammar shows up. Probability is not the final stage of language (which I assumed), but appears at the start, running under everything, noticed last only because we did not have the machinery or maths to understand it. This means that mastery is just the point where the distribution sits so deep it stops feeling like calculation, which is true for any athlete: just do it.

The form that model takes, from infant to old age, looks Bayesian. The baby tracking sound transitions is doing what Bayes described in the eighteenth century: holding a prior expectation, meeting evidence, updating the expectation, meeting more evidence. A growing school of neuroscience argues that the brain works this way at every level, a prediction machine that does not passively receive the world but constantly forecasts it, attending mainly to the errors, the places where the world surprises the forecast. Equally, there are others who do not agree with this. However, if that picture is even roughly right, then every conversation you have ever had has been an exchange of evidence against priors, and what we call understanding is the moment two people’s distributions converge enough for surprise to stop. Language is not the only thing that runs on updated probability; on this account, thinking does.

This line of thought is exactly what makes “distribution and probability” dangerous as an instrument of power. Rules can be argued with, as a rule has an author, a history, a possibility of appeal or repeal; you can point at it. Probability, well, just feels like reality. When something violates the distribution you have internalised, it does not feel forbidden; it feels strange, wrong in a way you cannot articulate, and the discomfort never arrives clearly labelled with its source. Every rule needs a new rule, something I have explored before.

Rules can be argued with. Probability just feels like reality.

“Naming” has always been a “technology”

An easy tell in AI writing is about naming and labelling, because it is in the dataset it learnt from, and history keeps demonstrating this. Heresy made an institution the arbiter of thought; before the word there was opinion or disagreement. Witchcraft named something with no coherent reference at all, which is precisely what made it usable against almost anyone, and in practice against women with property or knowledge or independence. Lunacy built the asylum and the authority to control and confine.

Years ago Lily Cole founded Impossible and we met up to talk about it, but ended up talking for hours about her love for Art History, and right now that conversation appears to me one of the cleanest understandings of this same mechanism. The distinction between art and craft is not in the objects; it is a jurisdiction, and for centuries power held it. A tapestry that took a year of extraordinary skill was craft; an oil painting was art; and since women were largely confined to textiles, ceramics and embroidery while the academies excluded them, the naming of ART quietly decided whose work could enter history before any individual judgement was passed. The western canon did the same process but at scale. Old Masters is a category that feels like a description of quality and functions as a register of who was permitted in the room. Primitive art let Europe file the sculptural traditions of entire continents under a word that made sophistication invisible by definition. And the namers always understood the stakes in both directions: when the Nazis hung modernism in a show titled Degenerate Art, and when conservative critics a generation earlier spat Impressionist as an insult, they grasped that you fight art with a label rather than an argument, because labels enter the distribution and arguments merely visit it. The artists understood it too, which is why so many movements took the insult and wore it. Capturing the name was capturing the jurisdiction.

History is not merely recorded by winners, it is named by them, and the names do the judging before any reader arrives. The same armed uprising is a rebellion if it fails and a revolution if it succeeds; the same fighters are terrorists or freedom fighters depending on who survives to print the textbook. Europe discovered a continent that tens of millions of people already lived on, and that single word, “discovery”, enabled five centuries of “legitimacy”. Civilisation and barbarian, pacification, settlement, the Dark Ages as a name for centuries that were dark mainly from where the namer stood: each is a verdict disguised as a noun. By the time I as a learner met these words the trial is long over, the distribution already shaped, and the question of who lost so that this word could win does not come up, because the word arrives feeling like the way things simply were. I was taught to align and obey (well, they hoped for that anyway).

But I should be honest about the other side of the coin, because the mechanism does not only serve the powerful controllers. Genocide was introduced in 1944 and handed the powerless a word with which to indict those who used power as abuse. Sexual harassment named a pattern women had lived inside for centuries as isolated, deniable incidents. Coercive control gave the law eyes for an abuse it previously could not see. Naming creates jurisdictions, and jurisdictions get claimed in both directions. The question is never whether naming is happening; I now think it is who holds the new category, and who the category holds. Who decided who decides, who polices the police.

Let’s run a few of today’s words through this lens and look both ways.

Misinformation sounds protective but quietly creates a class of authorised information and an arbiter of what qualifies, and it arrived at precisely the moment trust in the arbiters was weakest. Radicalisation turns movement away from the centre into pathology without ever having to defend the centre. Problematic suspends a thing from discourse while specifying neither the problem nor the harm.

From the other direction, the work is identical: woke as a pejorative stretches to cover almost anything, virtue signalling lets any moral position be dismissed as performance without engaging it, and cancel culture absorbs everything from mob injustice to ordinary criticism and grants whoever invokes it the standing of the censored. The elite, influencers, and media platforms assign membership of a controlling class whose boundaries are never drawn, which is what makes them durable.

None of these words spread because a committee approved them. They spread through repetition until they stopped feeling like framings and started feeling like descriptions. They were seeded into the distribution.

And so here we are: “Fake news”

This term deserves its own section because it does something the others do not. Every word I have used as an example frames; however, fake news claims jurisdiction over verification itself. Fake news began with a narrow, genuine referent, fabricated stories manufactured for ad revenue around the 2016 American election, and for a few months it named something real. Then it was captured and turned, redeployed against journalism itself, and in that turn it stopped being a category of content and became a weapon against categorisation. Meet a claim with evidence, and the evidence is fake news; challenge the dismissal, and the challenge comes from fake news. It is not an argument about what is true. It is a procedure for exempting yourself from argument, and I think it may be the first mass vocabulary in modern life whose function is to make verification worthless. So much for facts don’t lie, or trust and verify.

What this does is censorship, but worse. A community runs on a shared default that most speakers, most of the time, are at least attempting to describe a common world. Fake news corrodes the default. Every utterance arrives pre-suspected, every source pre-discredited for somebody, and the genuine question, asked in the expectation that an answer might settle something, becomes rarer, because the term exists to ensure nothing settles. Some keep talking at enormous volume, and everyone stops asking. Asking implies someone might answer truthfully, and the word has made that feel naive. Worse, every factual dispute becomes a loyalty test, so curiosity itself reads as defection, and the first freedom lost is the interior one, the willingness to wonder whether your own side might be wrong.

A population can be loud and unfree at the same time, and the freedom that goes first is the freedom to wonder whether you might be wrong.

What is a socially acceptable and friendly version of fake news?

If fake news is the mechanism, wellness and wellbeing are the same mechanism wearing a smile, and I find them more instructive because care is harder to argue with than fear.

Worth addressing the obvious misreading first: supporting people’s health is real and valuable work, and nothing here says stop doing it. The argument here is about what the words do alongside whatever good the provision does.

Wellness, as it has evolved commercially, names a condition of optimal functioning that is by design never reached, and the industry that grew around it, the supplements and retreats and apps and assessments, depends structurally on the gap between where you are and where wellness says you need to be. It is not a description of health; it is a deficit that can be sold to, and it travelled from the margins of alternative medicine to the centre of corporate culture in about two decades because it was commercially useful for it to feel natural.

Wellbeing in institutional hands is subtler, and to my mind the purest modern descendant of sin. An organisation that names and measures employee wellbeing acquires a legitimate interest in the inner lives of its people, a department, a dashboard, a jurisdiction over territory that previously had no institutional name and therefore no administrator. You cannot be supported for something that has no name, and the holder of the vocabulary of support holds leverage, gentler than absolution but from the same family. The employee who declines to engage is not breaking any rule. They are merely behaving in a way that increasingly feels strange, which is the tell that the distribution has already been moved. The word did not create the feeling. It created the jurisdiction, and the jurisdiction went invisible by becoming normal.

“company”

Company comes from the Latin “companio”, literally “bread fellow,” from com (with) and panis (bread). A “company” was the people you broke bread with. The first companies were families and the guilds that grew out of them, bound by the shared meal, the shared table, the obligation that comes from sitting down together. The word carried trust as its core meaning, not structure.

By 1600, the bread had gone, and the word meant a business association; by the industrial era, it meant limited liability, legal personhood and shareholder interest. The same string of letters now names a legal entity that exists precisely so that no individual has to carry the obligation personally, the exact opposite of breaking bread, where the obligation was the whole point. We kept the warm word and replaced the guts underneath it, which is the most efficient move in the entire repertoire of naming: you do not need a new word if you can hollow out an old one while everyone still feels its original warmth. Every time a corporation calls its people “ team” it is reaching back to borrow the trust that the legal form designed to remove. The word does the persuading; and structure does the opposite of what the word promises. Why does nobody ask … because the shift happened in the distribution, slowly, over centuries, it was never announced.

What keeps someone awake at night?

The CEO question, about what to worry about and focus on, is how distributions have always been shaped. The church shaped one through liturgy repeated weekly for centuries; empires shaped them through administrative language; broadcasters shaped them through radio and TV channels reaching every room. So if LLMs are merely bigger, this is an old story at a new scale, and scale alone is not an argument; television reached billions too.

What is genuinely new is not the scale or size. Every previous instrument was broadcast: one message, publicly visible, the same for everyone, contestable in the open. Dissenters heard the sermon too and could disagree and argue with it. An LLM is interactive and individual, deeply personalised. It answers you, privately, adapted to you, and leaves no shared artefact. A billion private conversations, each slightly shaped to its participant, cannot be contested the way a broadcast can, because there is no common text to point at. The shaping becomes invisible at the level of the exchange while remaining systematic at the level of the population. That is the difference. Now, is it big or worth worrying about?

Two reasons.

1. Human history does not speak with one voice; we have misinformation and a decade of attacks on the word misinformation, wellness marketing and wellness scepticism, so the model learns the dispute rather than one side of it.
2. And nobody has yet demonstrated that LLM exposure measurably shifts human linguistic intuitions; the loop is plausible but right now not proven with evidence (though that also applied to smoking, radium and screen time).

But the point is this: if true, it provides a mechanism with a coherent pathway to reshaping what billions of people find sayable, running at full scale, in private, and nobody is measuring it. This is where AI governance fails, and it is a description of an invisible condition we are eroding.

The vast data these systems learned from contains everything in this piece. Sin and heresy and their centuries of institutional weight, the canon and its quiet exclusions, the victors’ nouns with their verdicts already inside them, fake news and its entire adversarial grammar of meeting evidence with attribution rather than evaluation, wellbeing and its dashboards. The machines learned not just the words but the shape of the conversations around them, including the shape of conversations in which questions die, and they will reproduce whatever shape the distribution points to, fluently, on request, forever.

An LLM is the victor’s history made complete: the aggregate account of everyone who got to write at all, with everyone who did not write, or whose writing was not kept, absent from the distribution in a way no reader can detect.

The view from the top of the mountain, when the clouds are down

In all this elegance, there is the same trap the bell curve gives us. A Gaussian distribution (classic bell curve) describes the outcomes of random events with beautiful precision; it lets you explain the pattern, predict the next thousand throws, repeat the experiment, and we have built industries on it. What it never does is allow prediction for a single event. The curve tells you everything about the aggregate and nothing about why this coin, this throw, this moment in time. The model of the outcomes is not the mechanism of the events, and confusing the two is really easy.

LLMs can appear to be the Gaussian of human language. They model the statistical outcome of billions of acts of human thinking with enough fidelity to explain the pattern and repeat it on demand, and that is a genuine achievement, but the model has not, and does not, unwrap how we do it.

Somewhere in each of us, a thought becomes a sentence, and that crossing, whatever it is, from intention or feeling or wordless knowing into ordered words so we can share something, is the event the distribution summarises but cannot see. We have built a perfect mirror of the output of human minds while the generator remains as dark as it ever was.

Again, there are two sides. One side says the models think because the output matches, the other says they obviously do not, and both are claiming knowledge of a mechanism that nobody, for machine or human, actually has. The honest position is that we have learned how to reproduce the outcomes of thinking without learning what thinking is, and that uncertainty should be sitting in the middle of every confident claim made about these systems, but guess what, it never is. Because if it did, you might see it for what it is: probability.

The safety, governance and harm conversations all remain downstream of this question. What conditions are these systems creating in language, and therefore in thought, that will only become visible once they are the baseline? The conditions that make some things feel sayable, and others feel strange, are being reshaped privately and interactively by distributions frozen at a particular historical moment, and the reshaping does what gravity does. Invisible, constant, foundational.

This piece is itself an act of naming. Calling the builders of these systems “administrators of a jurisdiction” is a framing, and by my own logic you should ask what power it transfers and to whom. What is at stake when what you read as a response from an LLM arrives as the quiet statistical shape of every conversation you have with a machine? You are now in the model and cannot see the paradox.

The institutions building these systems are not engineers; they hold the distribution from which normal is drawn, while the thing the distribution summarises, the human act of turning thought into words, remains unexplained in them and in us. They are administering a jurisdiction over territory nobody has mapped. The question is whether they know it, and what knowing it would demand.

The question I would put to your AI team

If you lead a business and you want to make this real rather than philosophical, here is the question to take into the room. Do not ask it as a CEO/ leader/ board exec performing concern but as the thinker of a new risk not on the risk register.

“Show me three words our product, our model, or our content makes feel more normal than they did a year ago, show me who benefits from each, and show me how we would even know if we were wrong.”

What I expect will happen is that the team will reach first for the easy reading, bias in the training data, toxic outputs, the things there are dashboards for. Push past that, because those are the rules layer, the part you can point at and argue with. The real question is the distribution layer: not what the model is forbidden from saying, but what it makes feel ordinary through sheer repetition, and in whose interest. If nobody can answer the third part, how we would know if we were wrong, you have just located an invisible condition inside your own organisation, and you are now the administrator of a jurisdiction you did not know you held. That is not a reason to stop, it is the beginning of what I do for work.