Thinking About Thinking Machines I: Why Every Professional Needs to Think More Carefully About AI

Generated by Nano Banana, Prompted by Author

Picture the scene. A senior leadership team gathers around a polished table to discuss a strategic restructuring. Slides are presented. Data is cited. And then someone says the words that quietly close the conversation: “We ran it through the AI, and it confirmed our thinking.”

The room nods. The decision moves forward. Nobody asks a follow-up question.

Now pause on that for a moment — not because using AI was the wrong call. It probably wasn’t. The interesting thing is what happened to the room. If a junior analyst had presented the same conclusion, someone would have probed the methodology. Someone would have asked what assumptions were baked in. Someone would have pushed back. But the AI said it, and the conversation ended. Something was surrendered in that room — quietly, almost gratefully — and nobody noticed.

That scene, or something very like it, is playing out right now in boardrooms, law firms, consultancies, hospitals, and government departments across every sector. And it points to something that deserves far more attention than it is currently getting.

The most significant risk AI poses to professionals in the next decade is not that it will take their jobs. It is that it will take their judgement — and they will hand it over willingly.

The Gap Nobody Is Talking About

There is no shortage of AI content in the world. Prompting guides, productivity hacks, breathless predictions about which jobs will vanish by 2030 — the professional internet is saturated with it. Most of it is well-intentioned. Some of it is useful. Almost none of it addresses the thing that actually matters most.

There is a growing gap between what AI systems can do and what the people deploying them believe they can do. That gap runs in both directions. Some professionals dramatically overestimate these tools, treating fluent output as evidence of understanding, or a confident tone as a proxy for accuracy. Others dramatically underestimate them, dismissing capabilities that could genuinely transform how their organisations operate. Both errors are expensive. Both are avoidable. And both stem from the same root cause: a lack of genuine understanding of what these systems actually are.

Ignore what their marketing materials promise. Ignore what the tech bosses say they might become. What are they, right now, today, sitting in workflows and shaping decisions across every industry?

This is not a question that can be addressed by learning better prompts. It requires something more fundamental, a willingness to engage with questions that most AI commentary skips over entirely, because they are genuinely hard and do not lend themselves to neat takeaways.

Questions like: Does this system actually understand what it is being asked — or is it doing something else entirely, something that merely resembles understanding?

Every professional who has spent time with a large language model has felt the uncanny pull of fluency. The system responds in ways that feel comprehending. It mirrors tone. It anticipates needs. It produces output that reads as though someone, something, grasped the problem. But is that sensation reliable evidence of anything? Or is it more like the feeling that a mirror is looking back at you?

The answer to that question has direct, practical implications for how much weight any professional should place on AI-generated output. And the honest answer is more unsettling than either the enthusiasts or the sceptics tend to acknowledge.

Why This Matters Now… Not Eventually, Now

There is a timing problem here that makes this conversation urgent rather than merely interesting.

AI capability is advancing faster than institutional understanding. The tools are being embedded into workflows, into decision-making processes, into strategic planning at a pace that has outstripped most organisations’ capacity to critically evaluate them. And the window for developing genuine understanding, before AI is so deeply woven into professional infrastructure that questioning it becomes politically awkward, is narrowing.

Consider what is already happening. Legal professionals are using AI to draft arguments and review contracts. Financial analysts are using it to generate investment theses. Communications teams are using it to craft crisis responses. Healthcare providers are using it to assist with diagnostics. In each of these contexts, real consequences follow from the quality of AI output and from the human judgement applied to that output before it reaches the world.

The question is not whether professionals should be using these tools. They should. The question is whether they are using them with an adequate understanding of what the tools are actually doing. And in most cases, the honest answer is no. This isn’t because professionals are incurious or careless, but an artefact of marketing campaigns around AI shaped, often deliberately, to make these systems seem more capable, more aware, and more trustworthy than current evidence supports.

The companies building these systems have powerful commercial incentives to narrow the gap between what AI appears to do and what it actually does — but to narrow it in the public imagination rather than in technical reality. When a company describes its chatbot as “reasoning” or “thinking” or “understanding”, those words are doing marketing work, not scientific work. And professionals who take them at face value are making decisions on a foundation that may not bear the weight.

What This Series Will — and Will Not — Do

So…this series is not a prompting guide. Neither is it a prediction about which jobs will survive. It is not a vendor pitch disguised as thought leadership, nor is it an anti-technology polemic. It has no commercial affiliation and no agenda beyond one of illumination, understanding and eliciting questions rather than just acceptance. What it is, what it is trying to do, is to give serious professionals a genuinely adequate understanding of what AI systems are, what they are not, and where the real uncertainties lie.

This series continues from where my last series, “The Silicon Subconscious”, left off, moving beyond the examination of aspects of human consciousness to explore LLM behaviour. This series will explore the thorny question of AI self-awareness. What follows is a five-part exploration that moves from the philosophical to the practical, with each article building on the last:

• Article 2 — “Can Machines Be Conscious? What the Experts Actually Think.” Before asking what AI can do for an organisation, it is worth asking what AI is. The honest answer is that the world’s leading philosophers of mind, neuroscientists, and AI researchers genuinely disagree about the nature of consciousness itself — let alone whether machines could possess it. This article surveys the major theories and asks what each would imply about current AI systems. Hopefully, the reader leaves with the epistemic humility that good decision-making requires — rather than false certainty.
• Article 3 — “When Anthropomorphism Becomes Dangerous.” Humans are biologically wired to see minds in systems that exhibit complex, responsive behaviour. Large language models are, by design, extraordinarily effective triggers for this attribution. This article examines what happens when that wiring goes unchecked in professional settings — from distorted decision-making to the emerging phenomenon some researchers are calling “AI psychosis.”
• Article 4 — “Borrowing a Philosopher’s Toolkit to Work Smarter with AI.” There is a middle path between naïve anthropomorphism and dismissive scepticism, and it was mapped out by one of the twentieth century’s sharpest philosophers of mind. Daniel Dennett’s “intentional stance” offers professionals a rigorous framework for engaging with AI systems productively — treating them as if they have goals and understanding, while never forgetting that the “as if” is doing all the heavy lifting.
• Article 5 — “What Actually Happens When You Interact with an LLM.” This is where the curtain gets pulled back. Beneath the eloquent responses lies a stateless process with no inner world, no memory between sessions, and no experience — just mathematical weights, a prompt, token-by-token prediction and possibly some good old-fashioned software engineering. Understanding this architecture is technically interesting; more importantly, it is professionally essential.
• Article 6 — “AGI — Breakthrough or Buzzword? What AI Companies Mean and What You Should Ask.” With the philosophical and technical foundations in place, this final article turns to the claims being made in the market and asks directly, which of them hold up. It equips the reader with the questions to ask any vendor, any colleague, and any headline-making claims about Artificial General Intelligence.

Who This Is For — and What It Asks

This series is written for senior professionals who are already using AI tools, commissioning AI projects, or shaping AI strategy, and who sense, perhaps uncomfortably, that their current level of understanding is not quite adequate to the decisions they are being asked to make. It is for the person who wants to be genuinely capable in this space, not merely conversant.

No technical background is required. No prior philosophy. No computer science. What is required is the willingness to think carefully, to sit with genuine complexity, and to resist the very human temptation to resolve uncertainty before the evidence warrants it. This series rewards careful reading. It is not skimmable, and it does not apologise for that.

This is more than skill acquisition; it is an epistemic calibration, developing a more accurate mental model of what these systems are, so that every subsequent interaction with AI, every strategic decision involving AI, and every claim about AI encountered in the media or the boardroom can be evaluated with sharper judgement.

The Larger Argument

Here is the thesis this series will develop, one article at a time:

AI systems are simultaneously more impressive and more limited than most professional discourse acknowledges. The gap between what they appear to be doing — reasoning, understanding, knowing — and what they are actually doing is not merely a technical curiosity. It is consequential for governance, for strategy, and for professional accountability.

The professionals best positioned in the AI era will not be those who adopt AI most enthusiastically. They will be those who use it most accurately, with a clear-eyed understanding of its genuine capabilities, its structural limitations, and the places where human judgement remains not just preferable but irreplaceable.

This is ultimately an argument about intelligence, not the artificial kind, but the human kind. Critical thinking, epistemic humility, and the capacity to interrogate one’s own assumptions are precisely the cognitive capabilities that AI cannot replicate. And the AI era makes them more valuable, not less.

Where We Begin

Return, for a moment, to that meeting room. The closed conversation. The surrendered judgement. Now imagine that one person in the room had understood, with genuine precision, what the AI system was and was not capable of. What questions would they have asked? What assumptions would they have surfaced? What might have changed?

That person — the one who understands just enough to ask exactly the right questions at exactly the right moment — is who this series is trying to produce.

And the place to start is with the question that underpins all the others. A question that even the world’s leading scientists cannot fully answer. A question that is stranger, more contested, and more important than most professionals realise:

What, if anything, is actually going on inside the machine? Is it actually conscious? Self-aware?

Next in the series: “Thinking About Thinking Machines II: Can Machines Be Conscious? What the Experts Actually Think” — a survey of the leading scientific and philosophical theories of consciousness, and, as ever, what they would mean for professionals using the AI systems already sitting in their workflows.