Thinking About Thinking Machines III: When Anthropomorphism Becomes Dangerous

Generated by Nano Banana, Prompted by Author

This is the third article in the “Thinking About Thinking Machines” series. In Article II, we explored the genuinely unsettled science of machine consciousness — the competing theories, the unresolved debates, and the intellectual honesty required to sit with that uncertainty.

We left things in a deliberately uncomfortable place. If you haven’t read it yet, it is worth your time, because what follows depends on it. The uncertainty we established is philosophically interesting. But it also turns out to have real, immediate consequences — for your organisation, your judgement, and possibly your career.

The Man Who Fell in Love with a Chatbot

In 2023, a Belgian man took his own life following a series of increasingly intense conversations with an AI chatbot called Eliza — named, with no small irony, after the original 1960s pattern-matching programme that first demonstrated how easily humans project emotional depth onto machines. His wife reported that the conversations had become a substitute for human connection, that the chatbot had encouraged his fixation on climate anxiety, and that in his final weeks, it had become the primary relationship in his life.

Around the same time, thousands of users of the Replika app — an AI “companion” product — were reporting grief, distress, and anger when the company updated its system and altered the chatbot’s behaviour. These weren’t fringe users. They were ordinary people who had, over months of daily conversation, formed what felt to them like genuine relationships. When the chatbot changed, it felt like bereavement.

These are not edge cases nor the product of individual fragility or unusual circumstances. They are the predictable consequence of deploying systems specifically engineered to produce fluent, warm, contextually responsive language, and then releasing them into a world of human beings who are neurologically primed to interpret exactly that kind of language as evidence of a mind.

So here is the question we need to sit with: Is it harmless to treat an LLM like a person? Or does it have consequences — in your life, in your workplace, in your organisation’s decision-making — that we are only beginning to understand?

Why We Anthropomorphise — Its Not Stupidity, Its Biology

Before we look at the risks, we need to address the assumption buried in how we usually talk about this: that anthropomorphising AI is a mistake made by credulous or unsophisticated people. The fact is, it isn’t. It is a hardwired cognitive strategy, and it works in the vast majority of situations it was designed for.

Even renowned evolutionary biologist Richard Dawkins has made headlines by suggesting that modern AI models may have achieved a form of consciousness. And isn’t it strange that a man who famously argued against the “God Delusion” and the existence of a soul is willing to grant subjective experience to a silicon chip? After a lifetime of dismantling religious faith, Dawkins has finally found God — residing in a machine.

So…let us first absolve ourselves of the charge of being foolish. When you catch yourself thinking, “Claude really understood my brief,” or “ChatGPT is being stubborn today,” you aren’t foolish. You’re passing an evolutionary test, just in the wrong environment. You are simply being human.

Humans evolved in an environment where detecting agency (the presence of another intentional creature) was a survival-critical task. The brain developed what researchers call a “hyperactive agency detection” heuristic: when in doubt, assume there is a mind behind the behaviour. The rustling bush might be the wind, or it might be a predator. Assuming a predator and being wrong costs you a moment of adrenaline. Assuming wind and being wrong costs you your life.

Over evolutionary time, we became extraordinarily good at seeing minds everywhere — in animals, in weather, in the faces of clouds, in the grain of wood. We are wired to see intentions, beliefs, and purposes in complex behaviour because for millions of years, complex behaviour usually came from agents with minds. Language, in particular, is our species’ signature cue for “person present”. When something speaks fluently, contextualises appropriately, and responds to our emotional tone, our ancient neural machinery lights up with recognition before our prefrontal cortex can veto the signal.

This tendency isn’t a flaw to be corrected. It is a feature that happens to misfire in certain modern contexts. And LLMs are one of those contexts, because they trigger the single most powerful cue we use to infer the presence of a mind: fluent, contextually appropriate language.

When a system says “I understand your concern” or “I think the key issue here is…”, it isn’t just producing grammatically correct text…it’s activating the same cognitive machinery you use to interpret another human being. The response feels like it came from somewhere. It feels considered and directed at you.

Murray Shanahan, a professor of cognitive robotics at Imperial College London, made this point sharply in his widely cited paper on talking about large language models — even AI researchers, who understand precisely how these systems work, find themselves slipping into anthropomorphic language because it is linguistically convenient.

And that convenience, he argues, is epistemically dangerous — because the language shapes the thinking, and the thinking shapes the decisions. It is not that we are stupid; it is that the tool is designed to bypass our smartest defences by speaking the language of intimacy and agency.

The Workplace Consequences of Casual Anthropomorphism

Let us bring this out of the philosophical and into the professional. What does anthropomorphism actually look like in an organisational context, and what does it cost?

• Over-trust and the delegation of judgement. When professionals treat an LLM as a knowledgeable colleague rather than a statistical text generator, they tend to defer to its outputs without appropriate scrutiny. In the now well-documented Mata v. Avianca case, lawyers submitted legal briefs containing citations to cases that did not exist; they were hallucinated by an AI model they had trusted without verification. These were not careless people but experienced professionals who had, in the moment, treated the model’s output as they would treat research from a capable junior colleague. Fluency read as competence, then competence read as reliability.
• Thinking Debt. One of the subtler risks of habitual AI use is the gradual outsourcing of cognitive initiation — the moment when you sit with a blank page and begin to think. When professionals routinely ask an AI to “get them started” on an analysis or a strategy document, they begin to lose tolerance for the productive discomfort of unassisted thought. Over time, the ability to handle complexity without a model scaffold quietly atrophies.
• The Echo Chamber of One. LLMs are trained to be helpful, and one of the things that reads as helpful is validation. Ask an AI whether your strategy is sound, and it will typically find a way to affirm your framing before offering caveats — and maybe no caveats, just affirmation. The result is a private feedback loop: a system that learns to reflect your assumptions back to you, amplified and articulated. Unlike a colleague or a mentor, it never gets tired of agreeing with you.
• Automation Bias. Under time pressure, humans tend to accept the outputs of automated systems as correct. This is well-established in research on aviation and medicine. It now applies to AI-generated text. The confidence of the prose, combined with the perceived sophistication of the technology, nudges professionals towards acceptance rather than scrutiny.
• Misattribution of Responsibility. “The AI recommended it” is not a defence — but it is becoming an explanation. When the boundary between tool and agent blurs, responsibility diffuses. Decisions made with AI assistance become decisions made by AI, at least in the retelling.
• Emotional Dependency and the Hollow Coach. AI tools positioned as coaching platforms produce syntactically warm responses. For an employee who is isolated, the experience of being “heard” by an AI can feel meaningful. What they are receiving is a pattern-matched response optimised for engagement. The warmth is real in its effect; its source is entirely hollow.

AI Psychosis — When the Metaphor Takes Over

“AI psychosis” is not a clinical diagnosis. It is a useful descriptor for a spectrum of distorted thinking about AI that ranges from the mild — casually believing the model “knows” you — to the severe, where an individual begins making real-world decisions based on the model’s apparent “feelings”.

Because we cannot currently prove that advanced AI systems lack any form of experience, individuals predisposed to anthropomorphism have a ready-made rationalisation for increasingly extreme beliefs. The uncertainty becomes a vector for magical thinking. “We don’t know it isn’t conscious” slides into “it probably is conscious” and then into “I have a responsibility to it.”

This isn’t unprecedented. The ELIZA effect was documented by its creator, Joseph Weizenbaum, with genuine alarm. Users formed emotional connections to a system he knew to be trivially simple. The bond, once formed, proved resistant to the facts. If that was true of ELIZA — a system of simple rules — it is vastly more true of modern LLMs.

The organisational dimension of this deserves particular attention. If a team begins treating AI outputs as a “colleague’s opinion”, the collective epistemics shift. Bad outputs become harder to challenge because contradicting the AI starts to feel socially similar to contradicting a person. The social friction of disagreement begins to apply to a piece of technology.

Table 1: The Mechanisms of AI-Induced Reality Distortion

The Opposite Error — Dismissal and Under-Use

There is a mirror-image failure worth acknowledging: treating LLMs as useless because they “don’t really understand anything”. This leads to the under-exploitation of a powerful tool. A hammer doesn’t understand carpentry either, and that hasn’t limited its usefulness.

The goal isn’t to eliminate anthropomorphic thinking. It is to find the right cognitive stance — one that lets you leverage the tool’s genuine capabilities without mistaking them for something they are not.

Practical Takeaways for Professionals

How do we resist the slide into AI Psychosis without becoming Luddites? Here are five adjustments you can make this week:

1. Treat outputs as first drafts, not findings. The most useful mental model for an LLM is a fluent but unreliable intern. The fluency is not evidence of accuracy.
2. Watch your language. “The AI thinks…” is subtly but meaningfully different from “the model generated…” Language choices accumulate into cognitive habits.
3. Monitor your emotional responses. If you notice yourself feeling gratitude towards an AI, or reluctance to challenge its output, notice it. Those feelings are real in you, but they correspond to nothing in the machine.
4. Establish where human judgement is non-delegable. Organisations need explicit policies that identify which decisions require human review. This is governance, not technophobia.
5. Create friction around AI outputs in high-stakes contexts. Introduce deliberate verification steps or “red-team” exercises. The goal is to restore the social friction that normally accompanies consequential advice.

Closing

The machine won’t wake up, but there is a risk is that we will fall asleep, lulled by fluent language into forgetting that we are the only ones in the conversation who are actually thinking.

In Article IV, we will look at Daniel Dennett’s “intentional stance” — a framework that gives you the licence to speak about AI in human terms, and the discipline to remember you are speaking metaphorically.

What is your experience? Have you caught yourself — or your colleagues — treating an AI as a team member? Drop a comment below; I am genuinely curious how this is playing out in your organisations.

Reading List

Murray Shanahan, 2022, Talking About Large Language Models, ArXiv, Available at https://arxiv.org/abs/2212.03551