The Silicon Subconscious III: Getting Stuck in the Rabbit Hole

Generated by Flux 2 Pro, Prompted by the Author

This is the third article in a series exploring the parallels — and crucial differences — between human cognition and artificial intelligence. In the previous piece, we examined how both human brains and AI systems can ‘miss the gorilla in the room’. This apparent attention deficit arises from very different mechanisms — and subsequently requires different solutions. Today, we’re tackling a different, arguably more maddening phenomenon: the AI that refuses to let go.

Pattern Locking, Stubbornness, and the ‘Magic Eye’ Effect

Picture this: you are using your favourite Large Language Model (LLM) to outline a complex strategic report. The AI makes a glaring logical error. You politely correct it. The AI responds with that familiar, fawning enthusiasm: “You are entirely right, my apologies for the oversight! Here is the corrected version…”

And then, right there in the very next paragraph, it makes the exact same error again — or perhaps a bizarre, mutated variation of it. Sometimes it even reverts to the logical error as you work on a different part of the report.

It is maddening in a very specific, very human way. It feels like dealing with someone who is protecting their ego. It feels like stubbornness. It feels, if you will forgive the projection, like a colleague who simply refuses to be wrong.

Here is the thing: that feeling is one of the most misleading intuitions you can bring to AI deployment.

The AI has no ego to protect, no emotional investment in its own prior output, and no conscious mind quietly deciding to dig in. What you are experiencing is something far more interesting — and, for your organisation, far more consequential. It is a mathematical phenomenon. Researchers call it pattern locking. If we want to safely deploy AI in our organisations, we have to stop treating these systems like obstinate colleagues and start understanding the mathematical tracks they are laying down in real time.

The ‘Magic Eye’ and the Brain That Won’t Let Go

To understand what is happening inside an AI when it gets stuck, it helps to start with something that happens inside your own head.

Cast your mind back to Magic Eye pictures, those baffling posters of the 1990s, packed with what looked like psychedelic wallpaper, hiding a 3D image that only appeared if you stared in exactly the right way. The trick required you to decouple two visual systems that normally work in lockstep: where your eyes point, and what they focus on. It was deeply unnatural and, for most people, maddeningly difficult.

But here is the interesting part. Once you saw it — once your visual cortex found the hidden dolphin or space shuttle — you could not ‘un-see’ it. The brain had committed. It had found a coherent interpretation of the incoming noise, locked onto it, and constructed a rich, confident, three-dimensional percept from data that was, strictly speaking, two-dimensional nonsense. From that moment, the flat pattern was suppressed. The 3D image was real.

This is the brain doing exactly what it was built to do: find the most coherent interpretation of ambiguous data, and then stay there. Re-evaluating reality from first principles every few seconds is metabolically expensive. Evolution found it far more efficient to commit to a plausible interpretation and update it only when the evidence becomes overwhelming.

The same mechanism shows up in something simpler: the duck-rabbit illusion. One image, two completely valid interpretations. Which one you see first depends enormously on what you were thinking about before you looked. Read a story about ducks, and you see the duck. Attend a lecture on the effect of rabbits on ecosystems, and you see the rabbit. The photons hitting your retina are identical. What changes is your prior — the expectation you brought to the image. Your brain pattern-locks around that expectation with impressive efficiency.

This is what cognitive scientists call priming, and it sits at the heart of a whole family of human cognitive illusions: confirmation bias, the framing effect, and inattentional blindness. We see what we have already decided to see. This isn’t laziness or dishonesty on our behalf, but because that is the architecture, because it is efficient. To save cognitive calories, we lock onto patterns and use expectation as a form of data compression.

Why the Comparison Is Tempting — and Why It’s a Trap

Because we experience reality through this lens of biological bias, we fall into the trap of ‘Ontological Anthropomorphism’ — the seductive assumption that because an AI produces human-like outputs, it must possess human-like internal states. It is natural, at this point, to think: Well, the AI is doing the same thing. It locked onto a wrong answer, and now it can’t let go, just like the human brain.

There is a surface-level truth to that. However, the underlying mechanisms are so different that treating them as equivalent will lead you to entirely the wrong interventions, as we have already discussed in previous posts.

When you argue with a human colleague who has got something wrong, their brain is physically changing. Not metaphorically — literally. Synaptic connections are being re-weighted. New associations are being formed. Memory is being updated in real time. The experience of being persuaded is a biological event. The person who walks out of that conversation is, in a tiny but measurable way, different from the person who walked in.

When you argue with an AI in a chat session, none of that is happening. The model’s core parameters — the hundreds of billions of numerical weights that encode everything it ‘knows’ — are completely frozen. Locked. The conversation you are having is not teaching it anything. It is processing your correction as a new input, running it through a static mathematical structure, and generating the next most probable sequence of tokens. If the architecture of that static structure predisposes it towards a certain type of error, your correction is simply another piece of text to pattern-match against. The weights do not budge.

And here is the deeper point, the one that matters most for anyone making decisions about AI: the model has no metacognition. It cannot step back from its own output and think, “Wait — is my underlying premise actually correct here?” The capacity to doubt one’s own reasoning, to hold a thought at arm’s length and interrogate it, requires something the model simply does not possess: a conscious, reflective self. It has no inner observer watching the process. It only moves forward, one token at a time.

An interesting experiment involving physics and movement illustrates this well. When asked what would happen if a child holding a wooden ruler between both hands let go at one end, the AI consistently stated that the free end would rotate around the grasping hand and the ruler would end up pointing to the floor. This is a bizarre statement; as humans, we grasp the fact that even a young child can hold a ruler straight. Lacking reflective abilities or a conscious self, the AI was unable to identify the self-evident absurdity of its response.

The Mathematics of the Rabbit Hole

So what is actually happening when an AI ‘gets stuck’?

It comes down to how these models generate text. Large Language Models are what is called autoregressive — meaning each word they produce is predicted based on all the words that came before it, including the words they themselves just generated. This is elegant and powerful. It is also, in certain conditions, a trap.

Imagine the AI generates a response that contains a false premise in the first paragraph — let’s say it confidently misattributes a market statistic or conflates two regulatory frameworks. That false premise does not disappear. It becomes part of the context. When the model generates paragraph two, it is now predicting text that is statistically consistent with a world in which that false premise is true. The hallucination becomes the foundation. It is, with complete mathematical honesty, being consistent with its own prior output.

Think of it this way. Picture a complex landscape of hills and valleys, what AI researchers call a loss landscape. When an AI begins generating a response, it is like a ball rolling down that landscape, finding the path of least resistance. The moment it settles into a valley — a specific narrative groove, a particular line of reasoning — the mathematics of the system make it extraordinarily difficult to escape. Your corrections are, in effect, a gentle nudge to a ball sitting at the bottom of a deep well. You would need to lift it entirely out of the well to change its trajectory. But the architecture does not allow for that kind of vertical leap. The context window has collapsed around a localised pattern, and the model is, in a very real sense, stuck.

The ‘Smarter Model’ Trap

At this point, the natural executive instinct kicks in: Fine, but won’t the next generation of models solve this? Won’t GPT-6, or whatever comes after it, be smart enough to catch its own errors?

It is a reasonable question. And the answer is genuinely counterintuitive.

Larger, more powerful models can actually perform worse on certain types of cognitive traps, a phenomenon researchers have documented as inverse scaling. Here is why. A bigger model is better at recognising the dominant pattern in its training data. That is the whole point of scale: more parameters, more nuanced pattern recognition. But if the dominant pattern in the training data happens to encode a misconception, a logical trap, or a flawed heuristic, a larger model will identify and reproduce that pattern with greater confidence.

A small, relatively ‘dumb’ model might stumble into the right answer by accident — its noise and uncertainty occasionally bumping it off the wrong path. A state-of-the-art model will stride confidently and eloquently all the way to the wrong conclusion. The research shows a ‘U-shaped curve’ in performance on these tasks: small models fail randomly, medium models fail systematically, and only the very largest models sometimes recover — but only if the counter-intuitive case was well-represented in their training data in the first place.

More intelligent-sounding, yet more confidently wrong. That is a combination that should give any decision-maker pause.

Admittedly, developers can train these models on specific examples so that, over time, the AI provides the correct response to the ruler experiment. However, this is not a case of the model becoming inherently ‘smarter’; the underlying logic gap remains. The developers have simply ‘patched’ a single variant of the problem by rote. I wonder how much exhaustive training would be required to eliminate every possible permutation of such a misconception.

What This Means for How You Deploy AI

None of this is an argument for abandoning these tools. Pattern locking is a limitation, not a death sentence — and like most limitations, it can be managed intelligently once you understand what you are working with. Here are three principles worth building into your AI governance thinking:

1. Stop arguing with the chatbot. This sounds almost comically simple, but it runs counter to the instinct of most intelligent professionals. When you encounter an error, your natural response is to engage, to explain, to correct, and to reason your way to a better output. Resist this. The longer the conversation continues with a flawed premise embedded in it, the heavier that premise becomes as a mathematical context. Clear the context window. Start a fresh session. Reframe the task entirely.
2. Test for rabbit holes, not just accuracy. Most standard AI benchmarks measure performance on clean, well-defined problems — the kind the models have, in all likelihood, seen variations of during training. They tell you very little about how a model behaves when it encounters a logical trap, a contradictory constraint, or a subtle distractor. Before committing to an AI platform for high-stakes work, stress-test it deliberately. Give it problems with embedded false premises and see how quickly it locks in. Give it tasks where the obvious answer is wrong. The results will tell you something that the vendor’s benchmark scores won’t.
3. Build external metacognition into your workflows. Since the model cannot reflect on its own output in real time, design your processes so that something else does. The most robust approach is to treat generation and evaluation as separate steps, carried out by separate systems. Have one model — or one model instance with a clean context — generate the analysis. Then have a second, entirely independent model evaluate it for logical consistency, factual gaps, and unexamined assumptions. Researchers describe this as the ‘External Hippocampus’ — a verification layer that the generative model lacks within itself. It is not an elegant solution, but it works, and it maps to how good human organisations already function: the person who writes the proposal is rarely the same person who signs it off. The obvious verification step is the expert-in-the-loop.

The Limits of the Machine Mind

There is something almost philosophically clarifying about pattern locking, once you sit with it.

We tend to worry about AI being too human: too clever, too autonomous, too knowing. But AI stubbornness is not a sign of a machine that thinks too much like us. It is, if anything, the most vivid proof of how different it is. A human who repeats the same mistake after being corrected is, at some level, choosing to. There is an ego involved, a narrative self that needs protecting, a conscious agent who could do otherwise. The AI has none of that. It repeats its error because it has no mechanism to do anything else. It is, as the engineers might put it, a runaway process on a track that it is constructing one token at a time.

True intelligence, the kind we value in our best colleagues and leaders, involves the capacity for self-doubt. The willingness to pause mid-thought and ask: Am I sure about this? AI, for all its extraordinary capabilities, cannot do that. Not yet. Possibly not ever, in the ways that matter.

Understanding this is not a reason to distrust AI. It is a reason to deploy it with the same rigour you would bring to any powerful tool with known failure modes. The leaders who will get the most from these systems are not the ones who treat them as infallible oracles, nor the ones who dismiss them as unreliable toys. They are the ones who understand — precisely and practically — where the walls of the rabbit hole are.

In the next article, we turn to something that may be even more consequential for organisational decision-making: why AI doesn’t just fail to disagree with itself, it’s also structurally inclined not to disagree with you. We explore the Sycophant Trap: how the same training dynamics that make AI so fluent and helpful also make it dangerously agreeable, and what that means when you’re using it to pressure-test your most important decisions.

Found this useful? Follow this newsletter for weekly articles in the series.

What has your experience been with AI “rabbit holes” in your daily workflows? Have you found yourself arguing with a machine lately? Drop a comment below, and don’t forget to subscribe to the newsletter so you never miss an instalment!

Further Reading

Fang, L. and Grossberg, S., 2009, From stereogram to surface: how the brain sees the world in depth., Spat Vis., https://pubmed.ncbi.nlm.nih.gov/19055887/

Clark, A., 2015, Conservative versus Radical Predictive Processing, https://philosophyofbrains.com/2015/12/15/conservative-versus-radical-predictive-processing.aspx

Psychology Today, N.D., Priming, https://www.psychologytoday.com/gb/basics/priming

Wikipedia, N.D., Autostereogram, https://en.wikipedia.org/wiki/Autostereogram

Ku, A., et al, 2025, Levels of Analysis for Large Language Models, arXiv, https://arxiv.org/html/2503.13401v2

Mureithi, V., Functionalism, Algorithms and the Pursuit of a Theory of Mind for Artificial Intelligence, Critical Humanities, 2024, https://mds.marshall.edu/cgi/viewcontent.cgi?article=1049&context=criticalhumanities

NSK.AI Community, 2025, Why LLMs Hallucinate: Deep Dive into the Mechanisms, Limits, and Advanced Mitigations, Medium, https://medium.com/@NSK.AI/why-llms-hallucinate-deep-dive-into-the-mechanisms-limits-and-advanced-mitigations-cbe4471428ab

Robinson, G., 2025, The Mathematics of Meaning: How Latent Space Powers Modern Language Models, Medium, https://gregrobison.medium.com/the-mathematics-of-meaning-how-latent-space-powers-modern-language-models-c2235808d1c0

Acciai A, et al, 2025, Narrative coherence in neural language models. Front. Psychol., https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2025.1572076/full

McKenzie, I. R. et al., 2023, Inverse Scaling: When Bigger Isn’t Better, OpenReview.net, https://openreview.net/forum?id=DwgRm72GQF

Wei, J. et al, 2023, Inverse Scaling Can Become U-Shaped, OpenReview.net, https://openreview.net/forum?id=19sGqVUxQw

Jim the AI Whisperer, 2026, The wooden ruler test: What AI medical scribes can’t comprehend about human bodies, Medium, https://medium.com/the-generator/wooden-ruler-test-ai-medical-scribes-abdc5c072bf1