Thinking Agents, Fast and Slow

The failure modes Daniel Kahneman mapped in human cognition appear almost exactly in AI agents. That makes them predictable. And fixable.

Figure 1: Two operating modes, one mind — the fast chaotic side and the slow deliberate side, with an agent caught in between.

You’ve probably read about cognitive biases in hiring decisions, medical diagnoses, and financial trading. Have you ever applied the same framework to the AI agent you shipped last quarter?

Daniel Kahneman spent decades cataloguing the ways smart people get things consistently wrong. Not stupidity — systematic errors. Predictable ones.

Reading Thinking, Fast and Slow in 2024, I kept stopping to write notes that had nothing to do with human psychology. I was writing about agent architectures. The failure modes Kahneman catalogued map almost exactly onto the failure modes we keep hitting in production. Same structures. Same patterns. Different substrate.

This isn’t a metaphor. It’s a diagnostic lens and a design tool.

Two Systems, One Mind

Kahneman’s central insight is that the brain runs two parallel operating modes. System 1 is fast, automatic, and effortless. It recognizes faces, finishes sentences, detects danger, and makes snap judgments. It runs constantly in the background, brilliant at pattern-matching and terrible at nuance. System 2 is slow, deliberate, and expensive. It does math, follows arguments, weighs tradeoffs, and plans ahead. It requires attention, and most people avoid activating it unless they have to.

Here’s the twist: System 1 handles the vast majority of decisions, including ones that should involve System 2. The two systems don’t collaborate cleanly. System 1 generates an answer first, and System 2 often just endorses it. As Figure 1 shows, the split isn’t clean — the chaotic side bleeds into everything. That’s where the biases come from.

AI Agents Have the Same Problem

Modern AI agents are typically built around a large language model doing what LLMs do best: fast, fluent, pattern-based generation. That’s System 1 behavior. The agent reads the context, activates relevant patterns, produces output.

The engineering challenge is the same one evolution never fully solved: knowing when to shift into deliberate, effortful reasoning, and building systems that actually do it. The agents that fail most visibly aren’t the ones that can’t generate text. They’re the ones that generate confidently wrong text because nothing in the architecture asks them to second-guess themselves.

Let’s go through the biases Kahneman documented and see what they look like in agents.

1. Substitution — Answering the Wrong Question Fluently

This is Kahneman’s most fundamental insight. When System 1 encounters a hard question, it doesn’t struggle — it quietly substitutes an easier question and answers that instead, without flagging the swap. Asked “How happy are you with your life?” people often answer “How do I feel right now?” Asked “Is this investment risky?” they answer “Does this feel familiar?”

In agents: the model is asked a hard question and answers a simpler related one — fluently, confidently, convincingly. This is the mechanistic root of hallucination. The agent wasn’t lying. It answered a slightly different question than the one you asked, and the answer happened to be wrong for yours. A question like “What was the GDP growth rate of Brazil in 2019?” might get answered as “What does a plausible sentence about Brazil’s GDP growth look like?” The outputs are indistinguishable until you check the facts.

Design response: add verification steps after generation. For factual claims, don’t trust the answer — trust the retrieval. Build explicit “did I actually answer what was asked?” checks into reasoning chains. This is the job of the orchestration layer, not the model.

2. Anchoring — Over-Weighting the First Number

When people are exposed to a number — any number, even a random one — it anchors their subsequent estimates. Kahneman’s experiments showed that people gave significantly higher height estimates for a building if they’d first been asked whether it was taller or shorter than 1,000 feet. The anchor doesn’t have to be credible. It just has to appear first.

In agents: the first retrieved document, the first tool result, the first sentence of a long context — all of these anchor the agent’s reasoning in ways that are hard to overcome. An agent doing RAG (retrieval-augmented generation — grounding model output in documents pulled from a search index rather than from model memory) will often over-index on the top result, even if result #3 is more relevant. An agent in a loop will anchor on its initial plan and stretch it to fit new evidence rather than revising. You can watch this happen: give an agent contradictory information where the wrong answer appears earlier in the context, and watch it get the answer wrong despite the correct information being present.

Design response: randomize retrieval order when possible and build explicit re-ranking steps. In multi-step agents, add a planning review phase that forces reconsideration of the initial framing after new information is gathered.

3. Confirmation Bias — Looking for Evidence That Confirms

Once System 1 settles on a hypothesis, it stops searching neutrally. Information that supports the hypothesis is amplified; information that contradicts it is minimized or explained away. That’s efficiency, not cynicism — searching to confirm is cheaper than searching to falsify. The problem is that it produces a very different result.

In agents: once a reasoning chain commits to a direction, subsequent steps tend to find supporting evidence. This is especially visible in multi-hop reasoning tasks, where the agent forms an early hypothesis and then interprets ambiguous evidence as confirmation rather than a reason to pivot. In tool-using agents, it shows up as search query bias: the queries the agent generates are shaped by what it already believes, which means it retrieves evidence that supports the existing trajectory. The loop feeds itself.

Design response: explicitly prompt for steelmanned counterarguments before finalizing a conclusion. Add a “red team” step in critical reasoning flows. Some teams build dual-agent architectures — one agent argues for the conclusion, one argues against, a third adjudicates. Expensive but effective for high-stakes outputs.

4. Availability Heuristic — Judging by What’s Easy to Recall

People estimate probability based on how easily examples come to mind. Plane crashes feel more likely than car crashes because they’re more memorable. Rare but vivid events are overweighted; common but mundane ones are underweighted.

LLMs have a version of this baked in. Patterns common in training data are “easy to recall” in the generative sense — the model has strong priors toward them. Rare but correct answers are harder to surface. The model will confidently produce the common wrong answer over the rare correct one. This shows up sharply in code generation: popular libraries with abundant training examples get accurate completions; niche or newer ones with sparse representation get plausible-looking but broken code. The model isn’t guessing — it’s doing exactly what the availability heuristic does, reaching for what’s easy.

Design response: don’t rely on model memory for precision tasks. Retrieve. Up-to-date documentation, verified code examples, and current data should come from retrieval, not generation. The model’s job is synthesis and reasoning, not recall.

5. Overconfidence — Calibration Is Not a Default

Kahneman documented extensively that humans are systematically overconfident, especially in domains where they’re partially competent. The confidence intervals people report for uncertain estimates are almost always too narrow. They know more than nothing and less than everything, and they’re bad at tracking which side they’re on.

This is probably the most commercially damaging bias in agents. LLMs produce text in a confident, declarative register regardless of how uncertain the underlying inference is. Nothing in the generation process tracks uncertainty and surfaces it. Users end up trusting agent outputs at rates the actual accuracy doesn’t warrant — and the outputs that are most wrong are often stated most confidently, because fluency and accuracy aren’t correlated.

Design response: calibration is a first-class design concern, not an afterthought. Make uncertainty explicit in the system prompt. Add explicit uncertainty scoring for factual claims. Use tool calls to verify before asserting. If you’re building a user-facing product, the agent should communicate its confidence level, and that level should be grounded in something real, not just the model’s tone.

6. Sunk Cost — Doubling Down on a Failing Plan

The sunk cost fallacy is the tendency to continue a course of action because of what has already been invested in it. The rational move is to evaluate only future costs and benefits; the human move is to weight past effort heavily in the current decision.

Agentic systems running long multi-step loops exhibit this strikingly. An agent that has taken 12 steps down a particular path is highly reluctant to abandon it and start over, even when it encounters evidence that the approach is failing. The accumulated context history of a failing plan creates strong priors toward continuing it, because the model is always completing patterns from what already exists in the window.

Design response: build explicit checkpoints where agents evaluate whether the current approach should be abandoned. Context compression that strips the accumulated trajectory while preserving the relevant facts can help reset those priors. Budget limits that force replanning are a blunt but effective intervention.

7. Framing Effects — The Container Changes the Contents

Kahneman showed that logically equivalent problems produce different decisions based solely on how they’re framed. “90% survival rate” and “10% mortality rate” describe the same reality but reliably produce different choices. The frame adds no information — but it shapes the response anyway.

This is one of the best-documented findings in LLM research. The same underlying task, expressed with different framing, structure, or surface-level features, produces measurably different outputs. Prompt engineering is the practice of exploiting framing effects deliberately. The problem is that framing also introduces unintended distortions. A support ticket framed as a complaint will receive a more defensive response than the same content framed as a question. A resume screening prompt framed around “fit” produces different outputs than one framed around “requirements met.”

Design response: normalize inputs before passing them to the model wherever stakes are high. Audit prompts for unintended framing, especially ones that were written quickly and never reviewed. For high-stakes decisions, test equivalent framings of the same inputs and compare outputs. If the outputs diverge, you have a framing problem.

Figure 2: The seven failure patterns, sketched — each one a different way an agent gets it systematically wrong. Keep this as an audit checklist when reviewing an agent’s behavior in production.

The Architecture Question

Kahneman’s later work pointed toward a practical implication: better outcomes come not from eliminating System 1 — which is impossible — but from building environments that trigger System 2 when it matters. For humans, that means checklists, forced deliberation, cooling-off periods, and decision reviews. For AI agents, the translation is direct.

As Figure 3 illustrates, the fast path and the deliberate path both start from the same architecture. The slow one doesn’t emerge on its own — it has to be engineered. Not every agent needs to respond as fast as possible; for high-stakes tasks, multi-step reasoning with explicit verification steps is worth the latency. Build review agents whose job is to scrutinize generation outputs specifically for the bias patterns above — they function as the System 2 your generation layer doesn’t have natively. Keep generation and assertion separate: let the model produce freely, then verify before treating anything as fact.

Make uncertainty legible. An agent that speaks in the same confident register regardless of how certain it is hides information from the user. Calibration is infrastructure, not a UX detail. For agentic loops, design the context structure so pivoting is cheap — the sunk cost pattern is structural, and the fix has to be too.

Figure 3: Fast path versus deliberate path from the same reasoning engine. The glitching trail on the left isn’t a model failure — it’s what happens when you don’t build the right side.

The Fix Is Structural

There’s a tempting reading of this article: agents are too much like System 1, so the goal is to make them more like System 2. That framing is close but misses the point.

AI agents are powerful pattern-completion engines running in a regime optimized for fluency, not for accuracy or calibration. The biases that emerge aren’t accidents — they’re predictable consequences of that optimization target. Kahneman’s work is useful not because it tells us that agents are human, but because it gives us a vocabulary for describing systematic failure modes. When a failure mode is systematic, it’s addressable at the design level. You don’t fix it instance by instance. You fix it structurally.

The biases in this article aren’t exotic edge cases. They’re defaults. Every non-trivial agent you build will exhibit several of them. The question isn’t whether to address them — it’s whether you do it intentionally or whether you discover them through your users.

Kahneman spent his career arguing that awareness of a bias is a first step but not a sufficient one. You can’t think your way out of a cognitive shortcut while relying on the same system that produced it. For agents, the same principle holds: don’t prompt your way out of a structural problem. Build the architecture that makes the right behavior the default.

Marcelo Rosa is a Principal GenAI Architect at GFT Technologies, where he leads a team building generative AI solutions for the financial sector. He focuses on the gap between what AI systems promise in demos and what they deliver in production — and what it actually takes to close it. Connect on LinkedIn

This story is published on Generative AI. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories.

Subscribe to our newsletter and YouTube channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!