Can an AI Learn the Planning Algorithm? Part 4

Learning to Plan

This series looks at how autonomous and adaptive artificial reasoning agents can learn human-like strategies for searching, planning, and reasoning.

This is the last part of four, following an intro article on *Meta-cognition for Reasoning AI*:

• **Part 1: Foundations of Algorithmic Learning**
• **Part 2: RL for the Real World**
• **Part 3: The Biological Perspective**
• Part 4: Learning to Plan (this article)

The first three parts discussed a mix of existing research spanning several different research disciplines. This last part speculates on how those ideas can be drawn together to develop the next generation of reasoning AIs that can continually adapt to changing environments and learn from their mistakes, both for conversational agents and physical robots.

I provide a unifying perspective on the categorization of behavioral control mechanisms, leading to a more realistic way to understand human behavior and a more adaptive solution for AI. I propose a way to combine model-free, model-based, SR, and episodic RL into a single solution. I combine ideas from algorithmic learning, Bayesian inference, and biology to propose a latent representation scheme that enables efficient use of gradient-based planning in complex domains. I discuss potential architectural and functional components of meta-management. And I look at practical ways that a deliberative system can incorporate meta-management operations while avoiding the disruptive effects of context switching.

Contents

· Motivation · Agent Factorization · The problem of Categorizing Control Processes · Control Processes — Re-categorized · Meta-management Operations · Cognitive Architectures · Control Primitives · Latent Space · Disruptive Meta-management · Meta-management via Task Switching · Meta-management via Temporal Hierarchy · Pyramid of Reactivity · Not Covered · Conclusions · Acknowledgments · Citing · References

Motivation

It’s been a while since I laid out the premise from the introductory article, so allow me to briefly summarize it here.

I am interested in understanding how human intelligence works and how we can apply those ideas to AI. In particular, I am interested in how we can incorporate more accurate models of the control structures that support human deliberation. If we are successful in that, I believe our AIs may just perhaps be worthy of the “general” label in General Artificial Intelligence.

A number of principles and assertions guide the way that I have chosen to tackle this question:

• Embodied reasoning agents: I consider problems faced by embodied reasoning agents that can actively observe, explore, and learn from their environments and their own behaviors. Bodies may be physical or virtual. Learning is continual and life-long. However, many of the ideas gleaned from this view can be productively applied to supposedly very different agents — like chatbots that need to learn from their mistakes.
• Trajectories and Meta-management: deliberative processing produces cognitive state trajectories, and meta-management processes are needed to monitor and control those trajectories, both in real-time and over longer-time frames.
• Levels of control: control strategies like model-free / model-based RL etc. are not only applicable to overt action, but also to control of cognitive trajectories. Meta-management can at times be complex enough that it too requires deliberation, with associated needs for model-free, model-based, and other control strategies.
• Resolution of infinite regress: the potential for infinite regress on the number of meta-meta-meta layers providing control over the earlier layers is avoided through collapse of first-order and second-order deliberation into a system that learns to meta-manage itself: auto-meta-management. This is enabled through careful selection of architectural components, inductive biases, and the capacity for learning the planning algorithm.

The first step to building such an architecture is to incorporate a meta-management feedback loop, which derives a high-level summary of internal cognitive state and makes it available as a first-class sensory input like all other senses. This enables the agent to observe, learn, model, and predict its own cognitive processes in the same way as it does for the external environment and the behaviors of its own body.

High-level functional architecture for a deliberative process with meta-management feedback

This architecture creates three feedback loops, with different characteristics and purposes:

• Cognitive loop: RNN-like deliberative processing interacts with its own distributed internal state. State is implicit in the same way that Karmiloff-Smith refers to implicit knowledge (see the Part 3 article; Karmiloff-Smith, 1994): knowledge that is in the system, not for the system, cannot be freely inspected by the system. Here the encoding of state is optimized for object-level goals, as determined by the latent state representations of their sensory observations and the kinds of computations required for action selection.
• Meta-management feedback loop: transforms cognitive state into an object-level sensory observation. Cognitive state that is accessible as an explicit representation can be modelled and manipulated by the system.
• Sensorimotor loop: all overt actions have effects on the environment and the agent’s own body, and the agent subsequently observes those effects (e.g. indirectly through visual, auditory, and tactile outcomes, and directly through proprioceptive outcomes). This enables the agent to build up models of cause-effect relationships.

The meta-management feedback loop essentially completes a sensorimotor loop for cognition. The key difference is that the feedback path is entirely through neuronal connections within the brain.

The next step is to figure out how all that can work. How is that meta-management feedback summary produced? How does meta-management interact with normal deliberation? How does the system learn to do planning over cognitive trajectories?

The rest of this article addresses some of the details behind those questions, offering a starting point for further research.

I start by methodically reviewing past categorization approaches regarding model-free vs model-based, reactive vs deliberation, System I vs System II, object-level vs meta-level, and cognition vs meta-cognition. I present an alternative unifying perspective that, while conflicting with the usual simple juxtapositions, better reflects observations from behavioral science and neuroscience, and offers a more productive path to artificial implementations.

I then delve into practical functional and architectural descriptions of meta-management. The result remains high-level and speculative, but it introduces an idea for combining model-free, model-based, episodic RL, and SR within a single system. By identifying the planning primitives that compose those strategies, and by learning their orchestration. With that approach, the complex and adaptive behaviors that are typical of humans becomes an emergent property of a single system that was not explicitly trained for those behaviors.

I briefly discuss how ideas for latent representation regularization, smoothing, and convexification may be key to efficient control in real-world multi-task settings.

The final sections discuss how to balance an apparent paradox: meta-management is needed to ensure that first-order deliberative trajectories remain on-task and efficient, but the act of doing meta-management can be disruptive to otherwise productive deliberation.

Agent Factorization

How should we model the relationships between environment, body, and cognition in deliberative agents?

In considering only non-deliberative embodied agents, Part 2 described three ways that the relationship between environment and body can be modelled:

• Systems control: The traditional Markov Decision Process (MDP) and Partially Observable MDP (POMDP) formalisms used in RL originate from systems control research that treats the agent and its environment as a whole system. State is a simply a single vector that includes all features that are relevant to control.
• Interaction perspective: An interaction perspective factorizes the system into two parts — the environment and the agent — and considers that they interact with themselves and each other.
• Factorized: This idea of factorizing the system can be taken to further extreme by considering instead the extent of mutual interactions between individual components (e.g., parts of the body, objects in the environment). For example, the agent’s grasping device may have a closer relationship to reachable objects than it does to some component embedded deep within the confines of its frame.

Perspectives on how to model an environment + agent system, from Part 2.

For deliberative agents, we now need to consider three domains.

The system control perspective would be to consider the agent’s whole state as a single system, potentially part of the larger system that it resides within. This perspective can be motivated by the fact that all domains have influence on each other. However, the efficiency analysis of latent state inference from Part 2 informs us that this unfactorized model is a joint probability distribution with many more dimensions than we need.

Interaction perspective on modeling of an embodied reasoning agent

The interaction perspective separates the agent’s body (physical or virtual) from its cognitive state and models those states and their transitions separately. This perspective is convenient for its simplicity because it supports the idea that the carrying out of deliberation can occur partially independent of the physical state of the agent and its environment. This means that deliberation can be studied independently.

The conceptual model of embodied agents still applies to chat-bot AIs (clipart images from rawpixel.com and openclipart.com)

Note also that the embodied model of agents still applies to chat-bot-style reasoning agents. The agent’s environment is made up of the user and the tools that the agent has access to. It’s user interface amounts to a body.

While convenient, this factorization is unable to accurately take into account the very real ongoing interactions between internal cognitive deliberation and physical world of the environment and the agent’s body. The most obvious of which being simply that the situation changes as time proceeds — every moment taken for further deliberation delays action, and that delay has an associated value or cost.

The more extreme factorization perspective looks at how different features of the overall system interact and group according to degree of interaction.

Toy example of applying the factorization perspective to modeling of an embodied reasoning agent

My personal view is that the AI industry is only just starting to take seriously the idea that the cognitive system needs to be explicitly modeled and managed, for example in the work behind recent Reasoning Language Models. We have plenty of room for improvement under that paradigm alone.

However, I expect the most general purpose AI solution would architect the agent to learn the (external and internal) world structure from its own observations and explorations, via latent representations. This approach will naturally result in a clustered factorization where “objects” may map to anything or any combination within real-world environment, agent body, and agent cognitive state.

The problem of Categorizing Control Processes

Complex adaptive deliberative reasoning needs to combine multiple different forms of processing, operating at different levels of description, and with different targets. This motivates us to classify those various forms, levels and targets within human cognition, and to identify how they interact.

Many categorizations already exist and some are commonly used. However, the more I’ve read and thought about this, the more I find such attempts to be fraught with controversy and inconsistency at best, and misleading at worst.

Behavioral Psychology categorization of decision making and behavior

A common categorization used in many theoretical cognitive architectures, is based on a mixture of Dual-process Theory (Evans & Stanovich, 2013) and Meta-cognition research:

• Reactive: habitual control over overt behavior. No noticeable thinking occurs during the control, but we may think (deliberate) over the outcome after the fact. Also known as Type I processing or System I thinking in the context of decision making, as per dual-process theory.
• Deliberative: explicit and prolonged “thinking” that we are consciously aware of. The target of such thinking is either about overt action, about something in the environment, or about abstract concepts that indirectly bare some relation to the environment. Also known as Type II processing or System II thinking.
• Meta-cognitive: inward-focused cognitive processing of one’s own cognitive processes. Includes thinking about thinking, thinking about learning, assessments of memory, and the use of those observations and decisions to control subsequent cognitive processes. Dual-process descriptions often roll this into Type II/System II.

This categorization has a number of problems.

First, dual-process theory is primarily focused on forms of decision making, and leaves open what forms of interaction there may be between decision making and overt action. In practice, both action and cognitive processing can be separately described as under the control of reactive or deliberative processes. Consider these examples:

• Reactive overt action: walking down a road, instinctively ducking to avoid being hit, or suddenly reaching and catching an object before you had time to think about it. These are driven by habitual cognitive processes that are not immediately consciously accessible.
• Deliberative overt action: thinking about which brand to buy in the supermarket, or planning the journey before setting out. The overt action is delayed while the individual mulls over the options. Control of the overt action is deliberative — by extended cognitive processes that are at least partially consciously accessible. However, the cognitive processes themselves are likely a mixture of reactive and deliberative, as the next examples elucidate.
• Reactive cognitive action: many of our entrenched and counter-productive modes of thinking are due to automatized cognitive behaviors (e.g., cognitive biases), but it also includes well-practiced methods for problem solving and memory recall. You probably don’t have to think to know that 2+2 = 4. An experienced mathematician sees a problem and immediately knows that part of the solution involves calculating an integral. These are driven by habitual cognitive actions. Memory recall is probably another example — our ability for memory recall varies through different stages of development and can be improved through practice and different strategies.
• Deliberative cognitive action: working through a novel and complex decision or problem leads us to think extra hard. We may make multiple attempts, using different strategies. Some steps in the reasoning process may seemingly produce instantaneous results, but the overall process may be lengthy. Regardless of whether this is performed in aid of a direct overt goal (like planning the route to a new grocery store) or solving an abstract conceptual problem (e.g., math equations), the cognitive behavior is itself a deliberative process. For example, it includes deliberative decisions about how to proceed through the problem.

A second issue with the linear three-tier categorization is that meta-cognition too appears to be distinguishable into reactive and deliberative. Feelings of knowing and the tip of the tongue effect (Rosenthal, 2012; Shimamura, 2000) appear to our awareness as instantaneous results, without a sequence of events that led towards them. In contrast, thinking about our past mistakes and forming decisions to address them in the future is an act of deliberation.

A third issue is that this categorization makes a poor distinction between cognitive processes directly targeted towards overt-action and those that are more abstract in nature without being meta-cognitive (e.g., solving that math problem). A more nuanced discussion of control levels suggest three tiers: ground-level, object-level, and meta-level (Cox & Raja, 2011). Actual overt actions occur in the ground-level, primary cognitive processes occur in the object-level (including control of ground-level), and cognition of cognition occurs at the meta-level. Under that framing, reactive and deliberative control of overt action is a process that occurs at the object-level but targets ground-level behavior. Abstract thought is an object-level process that targets object-level concepts. And meta-cognition is a meta-level process that targets object-level behavior.

So rather than a simple single-axis categorization, perhaps we need two axes: (1) the target of cognition, and (2) the system of process employed. The target could be at a ground (i.e., overt action), object-level (i.e., abstract thought) or a meta-level (i.e., for meta-cognition). The system of process could be reactive or deliberative.

A two-axes categorization just raises more questions. Where do model-free, model-based, successor representation, episodic, Pavlovian, and reflexive control strategies fit here?

However, a third axis can also be added: whether the control strategy is primarily model-free, model-based, successor representation, episodic, Pavlovian, or reflexive. It’s common to assume that model-free control is reactive, and model-based control is deliberative. But how do successor representations and episodic control fit in? These are both described as middle-grounds between the extremes of model-free and model-based, so it’s not obvious how to associate them with reactive vs deliberative decision making.

Even the model-free/based control connection to reactive/deliberative decision making is not so simple. Consider these counter-examples:

• Model-based reactive control #1: Imagine that you watch someone take a paper cup and a lead weight out of a drawer. They place the weight inside the cup so that you can no longer see it, and then hand you the cup. You will automatically apply knowledge of the fact that lead weights are heavy to predict how much holding force you’ll need to apply when taking the cup. You will most likely do this without knowingly thinking about it (i.e., reactively), but you did just apply model-based control (explicit knowledge of world dynamics).
• Model-based reactive control #2: Gradient-based planning and Active Inference can be used to perform fast model-based control in an automatic way, without deliberation (see discussion of Fast-MB in Part 2).
• Model-free deliberative control: model-free learning is essentially an efficient way to cache highly repeated behaviors so that their control can be carried out more efficiently. Thus it is reasonable to expect that the brain will use model-free mechanisms to control cognitive trajectories wherever possible. Knowing that 2+2 = 4 could be an example. The well-learned mental strategies we use to solve common problems like math equations and cross-word puzzles are further examples. The mechanisms underlying memory recall are probably model-free.

Control Processes — Re-categorized

I see two productive ways to think about all this, by looking at the questions from two different perspectives: control needs, and control mechanisms.

Control needs perspective: which control mechanisms are needed to support overt action, cognition, and meta-management?

The control needs perspective examines what is needed to support control, grouped by the target of control. It produces a categorization as follows:

• Overt action: Control of externally observable behavior via motor function. State trajectories are in physical space (environment + body). It needs both reactive and deliberative control, and it needs all of the control strategies: model-free, model-based, SR, episodic, Pavlovian, reflexes, etc.
• Cognition: first-order or object-level cognitive processing, for the purpose of controlling overt action. Also includes abstract thought. State trajectories are in cognitive space and those trajectories need to be controlled. Needs both reactive and deliberative control. Needs most of the control strategies: model-free, model-based, SR, episodic, and perhaps others. Provides the mechanisms fulfilling the stated needs of overt action and thus runs in parallel and has close mutual interaction with actual overt action.
• Meta-management: second-order or meta-level cognitive processing, for the purpose of controlling cognition. The target of control is cognitive state trajectories, but deliberative meta-management has its own state trajectories in cognitive space, e.g. actively thinking about thinking. These need reactive and deliberative control, and all the same control strategies as needed by first-order cognition. Conceptually, this provides the mechanisms fulfilling the stated needs of Cognition. However, as per my overarching premise, I assume the mechanism is that of a meta-management-feedback loop that enables first-order cognition to meta-manage itself.

The above perspective doesn’t attempt to draw exact boundaries between the categories — in fact, they overlap considerably, explaining some of the problems faced by prior categorization attempts. It’s primary value is in highlighting the fact that the same underlying mechanisms are needed for different purposes.

Control mechanisms perspective: multiple control strategies are available, universally applicable to any target and control outcome. The other forms of categorization describe emergent properties.

The control mechanisms perspective accepts the fact that the same control strategies (model-free, etc.) are needed for different targets, and places those targets at the center of discussion. It suggests that there should be a shared instantiation of the control strategies, and produces an idealized functional architecture as follows:

• Targets of control: The domain of behavior that needs to be controlled. Includes external environment and body. Also includes abstract concepts and the cognitive system itself. The last target being enabled through the meta-management feedback loop and application of modelling of all perceptual inputs. Selection of target is controlled through attention.
• Control strategies: Model-free, model-based, successor representation, episodic, Pavlovian, reflexes, and anything else we are yet to discover. All can be applied to whatever target is selected at the time.
• Control outcomes: Overt action, cognitive action, internal homeostatic control, or any combination thereof. The domain of action depends only partly on the domain of the target. For example, if I notice that I’m having trouble focusing (cognitive space, meta-management), I may choose to go for a walk or make a coffee (environment and body space, overt action).

Essentially, this perspective posits that the brain has a unified way to represent any target of control, so that a single unified set of control strategies can be applied against any target, regardless of the physical or cognitive domain.

No such thing as reactive vs deliberation

Notice that the control mechanisms perspective makes no distinction between reactive and deliberative decision making.

These are useful concepts for simple high-level categorizations of cognitive behavior, and I will continue to use them for that purpose, but I don’t think they exist at the mechanistic level. Certainly not as the strict juxtaposition that research has tended to assume.

Rather, flexible and learned combinations of control strategies produce emergent behavior that under some circumstances can be clustered into the extremes of reactive vs deliberative. Under other circumstances, behavior will be too much of a blend for such a simple categorization.

My reasoning behind this will become clearer shortly.

Meta-management Operations

Now that we’ve understood the larger ecosystem of control, it’s about time I get more concrete with what meta-management entails. Up until now I’ve intentionally avoided listing concrete examples of meta-management. One reason is that I don’t have a clear idea myself. If we had a precise list of underlying human meta-cognitive processes then we’d know a lot more about human intelligence than we do now. My goal is not to define precisely what meta-management is, but to advocate that it’s something worth investigating and to propose a direction for that investigation.

Another reason is that it depends on the capabilities and weaknesses of the deliberative system. A model-free autonomous vacuum cleaner doesn’t need meta-management. A reasoning LLM utilizing Transformers and Monte Carlo Tree Search likely needs a different kind of meta-management to one that utilizes Message-passing GNNs.

Regardless, I will briefly attempt to provide something a little more concrete than what I have stated so far. This is easier now that we have reviewed different ideas across algorithmic theory, RL, bounded rationality, and cognitive theories.

Intuitively, cognitive processes control overt behavior, while meta-management controls cognitive processes. Thus, meta-management can be broadly defined as anything that influences how cognitive trajectories evolve in response to external stimuli (or the lack thereof). And indeed, this will prove to be a very broad definition, and perhaps many will disagree with it, but I like what it produces.

Here is a list of some concrete and some not-so-concrete examples of meta-management:

• Weight updates: the most basic form of meta-management and the most familiar process in AI RL. In humans, weight updating (i.e., synaptic strength) is an automatic process, but its mechanisms have been tuned by evolution and they are also adjusted dynamically from moment to moment within the individual. It is effectively reflexive meta-management (to be clear, I’m alluding here to the motor reflexes, not to self-reflection).
• Reward and value re-evaluations: in RL, the reward and/or value landscape has considerable effect on behavior. These landscapes can be updated through normal objective loss-based learning, but a more advanced agent would modify those landscapes through other means as well. For example, with appropriate latent factorization (see discussion in Part 2), some factors may be represented via associative memory rather than weights. Reactive and deliberative meta-cognitive processing of those memories would adjust the reward/value landscapes and associated behavior.
• Dynamic objective functions: more generally, in more advanced agents, the objective loss functions themselves will be partially guided by the agent’s own learned and cognitively decided metrics. This leads to a fascinating idea of hard-coded meta-objectives that constrain how the agent can tune its own objectives, and invokes ideas from hierarchical RL and bounded rationality.
• Input replay control: from Algorithmic Learning theory (see Part 1), an important requirement for stable and efficient convergence towards decisions is that the original goal is retained and re-injected in each successive cognitive step — known as Replay or Input Injection. However the re-injected inputs also need to be updated when sub-goals arise or overarching goals change. Meta-management may need to monitor and control how long a replayed input is carried over, such as through dynamic gating.
• Bounded resource control: from ideas in Bounded Rationality (see Part 3), the cognitive system must continually trade off the expected advantages of further deliberation vs resource availability and opportunity costs. This influences meta-cognitive decisions around continuation of deliberation and task-switching. Some of that tradeoff may be carried out automatically as learned first-order cognitive behavior. Explicit meta-management may need to step-in when the first-order behavior makes poor decisions, or to review those behaviors after the fact.
• Meta-memory, meta-learning, meta-representations: all of the meta-cognitive theories from cognitive psychology are candidates for meta-management implementation (see Part 3). However, it’s not always clear how to translate the behavioral ideas to low-level mechanisms.
• Feelings of knowing: Feelings of Knowing (FoK) and Feelings of Not Knowing (FoNK) are likely based on a mixture of cue familiarity (without explicit memory recall) and observation of memory retrieval results. For example, if memory retrieval fails but the cue itself is recognized as familiar (FoK), then it is identified as known but not remembered, in contrast with a firm belief of not knowing if the cue was also unfamiliar (Koriat, 1993; Umanath et al., 2023). This suggests an interaction between three systems: a fast and potentially inaccurate system that recognizes familiarity of input cue, a slow and accurate cue-based memory retrieval, and a meta-cognitive ability to observe the behavior of the first two systems and draw conclusions.
• Error explanation: from work on Computational Metacognitive Architectures (Part 3), it is theorized that an agent should seek to explain the cause of its errors in order to resolve them. Work in the context of Algorithmic Learning (Part 1) has proposed how to learn and use causal models that transform out-of-distribution tasks into a unified within-distribution representation (Bevilacqua, Zhao & Ribeiro, 2021; Cappart et al., 2023). That may provide a starting point for causal explanation of errors.
• Counter-factual reasoning: feeds into many adaptive mechanisms within humans, including generation of explanations of past events, setting intentions for future events, modulating emotional response, and making moral judgements of others (Byrne, 2016). It requires the integration of several forms of processing within the brain (Van Hoeck, Watson & Barbey, 2015), and appears to have separate pathways for explicit content-specific intent setting versus emotional affect and mindset changes (Epstude & Roese, 2008). Regret is a closely related process, believed to act as a trigger for counter-factual reasoning (Zeelenberg & Pieters, 2007). This suggests one way in which emotional affect and hindsight, two seemingly very human behaviors, have important meta-management roles in future AI.
• Allostatic regulation: In the body, *allostatic* processes adapt heart rate, blood pressure, hormones, etc. according to changes in energy demand. This occurs both in response to active demand, and preemptively in preparation for anticipated energy needs. These have direct physiological effect on brain function, and include sending increased glucose and oxygen supply to different brain regions depending on need. Similar allostatic processes also take effect more internally within brain function, such as the release of serotonin in anticipation of rewards, which may modulate model-based learning (Taira & Sharpe, 2025). This suggests the need for preemptive meta-management processes in AI systems that prepare energy supply, attention, and cognitive tuning for anticipated workloads.

The broad definition of meta-management reframes the classic RL learning algorithms, placing them as a small but important component of a much larger context. This is the assertion that I suspect be controversial to some. But I think this framing has significant value for the RL community because it highlights the gaps in our current AI control systems and suggests where we will get inspiration to fill those gaps.

Cognitive Architectures

Another question of interest is about the kinds of architectures needed to incorporate meta-management into our AI solutions.

First simple incorporation of meta-management feedback loop. A latent state model using RNN or Transformer and dynamic reconstruction loss against perceptual inputs (solves POMDP problem). A Meta-RL Policy that can emulate both model-free and model-based behavior via internal state dynamics (see Part 2), but subject to problems of Implicit knowledge and Shallow High-level Thought (see Part 3). Meta-management feedback via a highly dimensionally reduced sample of hidden state (e.g., 1% or less), and a second dynamic latent state model (treats cognitive state as a POMDP).

As a first draft, we may discover benefit simply from a meta-management feedback loop designed according to its simplest interpretation. Something like a variational auto-encoder that encodes a dimensionally reduced representation of the internal RNN or Transformer state of the main deliberative loop, and then feeds that representation as an additional input to the same loop.

A further revision would imbue the feedback loop with inductive biases that focus on what the agent needs for its own meta-management. For example, signals needed for learning and making bounded rational decisions that trade-off the cost and accuracy of different deliberative strategies.

However, I expect that we will require two additional components before we see noticeable benefit.

First, the problem domain must be sufficiently complex and fluid to warrant the need for flexible adaptation within the agent. Many of the classic toy problems used in RL research are simply not challenging enough.

A possible example of the minimum kind of complexity needed is for an agent to learn a few Atari games through traditional RL techniques, and then have it learn previously unseen Atari games using only online adaptation techniques and given only a handful of trials (Reed et al., 2022; Lee et al., 2022; Xie et al., 2025). Another more prescient example is for conversational reasoning AIs to adapt and learn new thinking and problem solving skills in real-time.

Toy example of a possible connectionist cognitive architecture with meta-management. States and actions against environment (e), body (b), cognition (c), simulations (s), and memory (m). Modelling primitives used to simulate trajectories to arbitrary future states (s’ₛ) — discussed in next section. Episodic memory automatically captures recent and significant events, queryable at any time (qₘ), with results returned as state inputs (sₘ).

Second, I believe we need to renew focus on the development of Cognitive Architectures (see Part 3). Such ideas have long been at odds with many of the greatest successes in connectionist AI, which often use a single end-to-end trained forward model (single pass or recurrent). These ideas are also at odds with modern neuroscience, which increasingly views the brain as highly distributed and holographic: only weakly functionally organized, and any given region involved in many different functional behaviors.

Despite the issues with cognitive architectures, they provide a much simpler framework for building practical artificial versions of intelligence than if we tried to entirely emulate brain organization. Furthermore, simple cognitive architectures are already a mainstream part of modern deep NN solutions.

For example, the Dreamer architecture has separate components for reward prediction, value prediction, latent state prediction, dynamics prediction, perceptual reconstruction, and policy, all within a well-structured architecture (see Part 2 for a visual breakdown, and Hafner et al., 2020). Reasoning LMs increasingly incorporate explicit components and orchestration code for policy, planning, and both working memory and long-term memory, all within a carefully designed architecture (Besta et al., 2025).

Control Primitives

The Part 2 article described four categories of adaptive behavioral control strategy: model-free, model-based, successor representation (SR), and episodic. This is in addition to even more basic control strategies found in mammals: Pavlovian (stimulus-response), and reflexes (largely innate).

Despite decades of attempts to delineate model-free from model-based control in behavioral studies, and to understand how the brain chooses between them, we still have only a basic idea of how that might be achieved in the brain. The more recent recognition of SR and episodic control only adds to the confusion.

Different control strategies trade-off flexibility and efficiency (adapted from Gershman, 2018)

Another aspect of the problem is that behavior can be achieved through a mixture of control strategies operating at different levels. For example, coarse-grained model-based planning orchestrating automatized model-free fine-grained motor control.

The difficulty of clearly delineating these control strategies in behavioral data is consistent with my suggestion that these control strategies are learned. Systems that learn their behavior through parameter fitting tend to produce more nuanced adaptations than what we might build by hand as researchers.

It is also consistent with the Bitter Lesson (Sutton, 2019) that end-to-end learning is more effective than building in our own misguided assumptions through hand-rolled solutions. However, the No Free Lunch Theory (discussed in Part 1) makes it clear that we must incorporate strong and appropriate inductive biases in order for the system to converge within a reasonable time.

How then should we proceed if we wish to build an artificial system that can learn to perform model-free, model-based, SR, and episodic control, to adaptively combine them in complex ways, while striking the right balance between end-to-end learning and inductive bias?

Control through primitives. Model-free, model-based, etc. may be high-level control strategies that emerge by learning to combine lower-level primitive operations in different ways.

The suggestion I propose is that we look at the low-level primitives that make up the four high-level control strategies. I alluded to one example primitive in the Part 2 discussion on Gradient-based Planning, which gave us a factorization of model-based control into fast-MB (rapid greedy optimization) and slow-MB (domain-aware high-level orchestration). Fast-MB is a planning primitive and needs no learning for gradient-based optimization. Slow-MB is complex and must be learned from experience. Other potential examples include that of value functions, transition models, and policies.

Instead of training an agent according to strict model-free RL, model-based RL, etc., we should architect the system with the union of all primitive modelling and computational capabilities that make up each of the high-level control strategies. And we should train the agent according to flexible objectives that enable them to explore and adapt different ways of orchestrating those primitives to emulate high-level control strategies.

At that point, the high-level control strategies become emergent behaviors rather than being specifically targeted as the training goal.

This can be seen as a generalization of the observation that a meta-learning RNN architecture can produce both model-free and model-based behavior (Duan et al., 2016; Wang et al., 2017; Ritter et al., 2018). So it implies an architecture of low-level primitives, and a meta-controller.

Draft list of control primitives

What might a complete list of primitives include? At a high-level, one would expect to see aspects of each of the four main control strategies (model-free, model-based, episodic, SR). So first we need to break those strategies down into composable parts. We can also take inspiration from a number of other research areas.

An initial suggestion of the kinds of primitives that we might want to include in this solution are:

• From model-free and model-based RL: reward functions (R(s) or more generically R(s,a,s’)), value functions (V(s) and Q(s,a)), transition and reverse transition models (P(s’|s,a), P(s|s’,a)), and policy (𝜋(a|s) or 𝜋(a|s,g)).
• From gradient-based planning: gradient-based fast-MB for trajectory optimization. Fast-MB becomes the primitive, slow-MB becomes the learned orchestration behavior that uses fast-MB.
• From SR: successor features, bottleneck discovery, topological segmentation.
• From episodic RL: associative trajectory memories, trajectory similarity and weighting metrics, task-similarity based recall of past computational state.
• From GNNs: some aspects of search may be included as primitives too. For example, tree and graph-based search can be factorized into separate learned models for node selection, update, expansion, aggregation, pruning, backtracking and others (Besta et al., 2025).
• From hierarchical RL: all of the above, learned and represented over multiple spatial and temporal granularities (P(option|s₁,sₖ))
• From multi-task RL: methods for detecting task context, clustering task skills.
• From research rational analysis: systems for proposing reward-maximizing goals (see Part 3), inclusion of time and opportunity cost in value estimates.

A number of overarching principles help to focus how these primitives can be incorporated into the larger solution:

• Localized losses: the application of end-to-end learning does not require that learning objective losses are only applied to the end result. Many recent successes in RL show the importance of applying component-level localized losses in addition to global objective losses (e.g., Ha & Schmidhuber, 2018; Hafner et al., 2020;). For example, new observations can be incorporated into the transition and reward models regardless of the control strategy employed by the agent at the time. In the ideal, each of the primitives described above would have its own local loss, while additionally taking part in the full end-to-end objective.
• Curriculum learning: the developmental sequencing of learning in humans as their mature is a clear indication of the importance of adjusting learning strategies to the level of learning already achieved. While architectural structures are an effective way to induce certain inductive biases, a learning curriculum provides an alternative avenue for application of inductive bias. And the more flexible the architecture, the more we need the inductive biases. AI training now includes many different training strategies, from supervised learning, semi-supervised learning, imitation learning, teacher-instruction learning, to full trial-and-error RL. The system above will likely work best with a learning curriculum that starts with the most strict techniques like supervised learning and slowly transitions towards allowing the agent to perform their own exploration against their own intrinsic rewards.
• Intrinsic reward: with so many different options for computing behavior, and with many options producing comparable overt results much of the time, the system must produce its own intrinsic reward signals so that it can learn the optimal choice of control primitive for different scenarios. Lessons from Resource Rational Analysis play a significant role here, suggesting that intrinsic signals should incorporate metrics of effort, opportunity cost, uncertainty, and the “stakes” associated with the task. The Free Energy Principle and Active Inference (Da Costa et al., 2020; Parr, Pezzulo & Friston, 2022) suggest that the system should seek to explore how different control primitives contribute to outcomes, provided that such exploration never exceeds its tolerance for the risk of harmful outcomes.
• Meta-management: the complexity associated with flexible emergence of multiple control strategies is a strong case for why meta-management is needed. It is with this level of flexibility that we must consider additional human cognitive behaviors of counter-factual reasoning, hindsight, regret, and even emotional affect. These are the mechanisms by which we observe and change our overt and cognitive behaviors over the long term, learning from what worked and what didn’t. Auto-meta-management is what enables those meta-cognitive processes to be executed and to take effect.

There are many potential problems to be faced by anyone attempting such an elaborate architecture. The solution described here is essentially a radically scaled out meta-version of the Options framework for Hierarchical RL, with all of the associated problems (Sutton, Precup, & Singh, 1999; Pateria et al., 2021; Hutsebaut-Buysse, Mets, & Latré, 2022). Regardless, this is already the direction we are moving in. The distinction between model-free and model-based solution has become increasingly blurred in RL research. It is only a matter of time before SR and episodic RL are merged in too.

Latent Space

Planning is hard, in part because it is essentially a task of identifying a trajectory through a highly non-linear system with complex dynamical interactions. Any given choice of action at one time can have a significant and complex effect on the outcomes and available choices at a later time.

For example, one action can place the goal within line of sight, while another may induce a maze-like subsequent trajectory. When multiple control strategies are also available, the range of choices explodes and the system needs metrics and mechanisms for comparing their relative merits.

One avenue for reducing the combinatorial complexity and dynamical system effect is through shaping of the state representations. Part 2 discussed how latent state representations are more efficient than raw perceptual representations. It also described how latent representations can be further shaped or transformed to focus on task-relevancy. Part 3 discussed how biology appears to perform smoothing and straightening of its state representations via the Temporal Straightening Hypothesis. Part 1 introduced Path Independence and Input Re-injection with associated training and regularization techniques that encourage iterative processing to behave as if its navigating a convex manifold.

In the extreme ideal, all state or value representations for all problems would be transformed to a linear or convex manifold, so that all iterative processing (e.g., deliberation and planning) is simply a matter of greedy hill-climbing. That’s not practical in the general case, but perhaps there’s some truth in the idea for localized tasks.

Todorov introduced the idea of a Linear MDP, that reframes the planning problem with a linearized form of value function (Part 2; Todorov, 2006; Todorov, 2009). The approach has a number of problems for real-world RL. One issue has been a lack of general purpose techniques for learning linearized latent spaces.

I suspect that the Temporal Straightening Hypothesis is part of the answer. Another part of the answer lies in the convex objective functions produced by Path Independence and Input Re-injection. None of these approaches can hope to produce globally smooth and convex surfaces for planning, but they can do it locally for specific tasks.

A hypothetical 3-tier approach for representation and planning.

This leads to an overall approach that combines techniques:

• Multiple representations
• Smoothing and linearizing
• Task-dependent transformations.
• Fast and slow planning.

First, perceptional representations are transformed to a domain-agnostic latent space with regularization for local temporal smoothing (as per the Temporal Straightening Hypothesis). This representation space is not convex for any given typical task, but by being temporally smoothed it is easier to optimize subsequent transformations.

Second, the domain-agnostic representation is transformed to a task-conditioned value or value-equivalent space, where most behavioral control takes place. While the representation itself is task-specific, work on Episodic RL and Meta-learning shows us that processes in relation to it can be done by a task-agnostic similarity kernel (Gershman & Daw, 2017; Ritter et al., 2018; Botvinick et al., 2019; Zheng et al., 2025). Essentially, the agent can rapidly learn new tasks and associated task-representations while still benefitting from slow-learned general-purpose inductive biases for comparing options and outcomes.

Perhaps the task-agnostic similarity kernel may prove insufficient for rapid adaptation to new skills, so an extension is to follow it by a third transformation via fast-learned value metrics such as from ideas associated with Successor Representations or more directly via methods for rapid value updating (Pritzel et al., 2017).

Any given complex problem is likely still not convex in full, but most of its subtasks can be. And the simplest problems do become convex in full. Localized sub-task value spaces can be easily planned over via fast-MB, while slow-MB orchestrates the overall process with heuristics, search, and backtracking where needed — leading to the variety of decision heuristics discussed in Part 3.

A common consensus from Philosophy of Mind research is that we cannot infer much from our introspective experience. Just because our perception of our mind feels a certain way, does not mean that the neurons truly carry out their processing in that way. Regardless, some techniques in traditional RL are clearly nothing like how our minds work. When we plan through a problem we don’t consciously roll out thousands of potential trajectories. We don’t predict the outcome of every single tiny muscle movement that makes up the motion of the arm and hand as they will later grasp the coffee mug once the jug has boiled.

Likewise, for the vast majority of planning tasks that we do from day to day, the mental landscapes that we navigate do not twist and weave like a complex dynamical system. Rather, planning usually feels like a couple of concrete decisions and the rest is as easy as riding a slide to the bottom. I believe the representational transformations described above, and the smooth and convexified manifolds they produce, is much more aligned to how our minds really work. This is the direction we need to go.

Disruptive Meta-management

So we’ve got a collection of learnable low-level primitives, we’ve got multiple levels of domain-general and task-dependent representations, and we’ve got high-level controllers that learn to orchestrate all of the above. This is precisely the kind of complex system that needs meta-management — the high-level controllers need feedback, reward shaping, and meta-review of their behavior. It’s time to tease out how deliberation and meta-management should interact.

Ideas from Bounded Rationality and Resource Rational Analysis (see Part 3) provide a good formalism for analyzing the interactions, however the terminology used there conflicts somewhat with the meaning of “meta” in discussions of meta-cognition and meta-management. So to try to avoid confusion, let me lay out the terminology clearly.

Layers of MDP for deliberation. An object-MDP with states (s₁, s₂, …), transitioning due to actions (a₁, a₂, …). A first-order meta-MDP with belief states (b₁, b₂, …), transitioning due to cognitive actions (c₁, c₂, …), and producing an object-level action as its terminal output. The object-level reward is supplied as the reward for the meta-MDPs terminal state, and some intrinsic mechanism computes credit assignment to intermediate belief-state transition rewards.

Bounded Rationality and Resource Rational Analysis model the evolution of states for overt action and deliberation as a hierarchy of MDPs. The object-level MDP (object-MDP) models state and actions of environment & body. The meta-level MDP (meta-MDP) models state and actions in the deliberative system. The ultimate goal is a sequence of actions against the object-MDP, but prolonged decision-making about those actions require execution of the meta-MDP.

Meta-cognitive processes are not represented under that framework, so in this and the next two sections I shall identify meta-management as a second-order process or task at the meta-level (e.g., second-order meta-MDP). Normal deliberation (e.g., in relation to overt action) is a first-order meta-level process or task.

In summary:

• object-level: direct control of overt action
• first-order meta-MDP: deliberation
• second-order meta-MDP: meta-management.

The proposed hierarchical relationship between object-MDP and meta-MDP reveals an important lesson about the allocation of resources across different domains. An agent can internally simulate external situations with significantly less time and energy than would be expended performing the situation in the external environment. While deliberation does consume time and energy, for many situations this expenditure is almost negligeable. Thus, we can consider that there is a partial separation between the domains of external dynamics and internal cognitive dynamics.

More concretely, it is reasonable to take the simplifying assumption that the object-MDP is frozen in state and time during the execution of the meta-MDP for deliberation. Colloquially, we assume that the mind is fast enough that the outside world is effectively frozen while the mind figures out what to do. Once the meta-MDP has completed its computations and decided upon the best object-level action, that action is then carried out with respect to the object-level state as it was before the meta-MDP process began. The meta-MDP is side-effect free: the execution of its intermediate computations do not change the object-level state (under our simplifying assumption).

Meta-management via explicit hierarchy. An object-level MDP with states (s₁, s₂, …), transitioning due to actions (a₁, a₂, …). A first-order meta-MDP with belief states (b₁, b₂, …), transitioning due to cognitive actions (c₁, c₂, …), and producing an object-level action as its terminal output. A second-order meta-MDP (meta-meta-MDP) with higher-order belief states (h₁, h₂, …), and producing a first-order meta-level action as its terminal state.

How so for meta-management? If a meta-MDP is required for deliberative consideration of overt actions, and meta-management is deliberative consideration of cognitive actions, then a naive extension is that a third layer is needed in the hierarchy of MDPs: a meta-meta-MDP.

This raises three problems. The first is that time proceeds at the same rate when simulating cognitive processes (while executing the meta-meta-MDP) as when actually carrying out cognitive processes (while executing the meta-MDP). So the cost/advantage of simulation is very different at this level than it is when simulating object-level trajectories. Secondly, and for the same reason, the state of the meta-MDP does not “freeze in time” because time clearly proceeds. Lastly, as I laid out in the intro article, I don’t believe it is sustainable or biologically plausible to have a separate architectural level for meta-management. Instead, it must be performed by the first-order cognitive system against itself, as a second-order task within the meta-MDP. This necessarily directly affects its own state.

Consequently, the meta-MDP’s state is not “frozen” in any way during execution of the second-order meta-MDP task. Meta-management has a side-effect problem: every second-order meta-MDP computation directly and indirectly influences the immediate state of the first-order meta-MDP. By the time that meta-management computations are complete and it has chosen some suitable action to take, the reference state has changed and so the output of meta-management is no longer accurate.

By way of analogy, imagine you are trying to decide to turn left or right at a fork in the road. However, you have no map, no knowledge of either route, and no way to heuristically guess. So you explore the whole of one route, then return and explore the whole of the other route. By the end of the second route you now have all the information needed and you know that you should have turned left….except you’re no longer at that fork, so the left-turn decision is not directly applicable to your current state. The act of investigating the options at a particular state changed the state.

I see four approaches to tackle this issue:

• Architectural hierarchy: a separate architectural component devoted to meta-management. I include this for the sake of completeness, and so I can re-state the reasons why I don’t think this is suitable. While tempting as a way to enable second-order meta-MDP state to evolve independently of first-order meta-MDP state, it creates a problem of infinite regress (how should the second-order meta-MDP be trained and controlled?), and raises questions of how domain knowledge can be efficiently shared between the levels.
• Ignore the problem: maybe the side-effect problem is not so much of an issue. Perhaps the agent simply learns to anticipate and correct for the fact that first-order state changes as a result of second-order processing.
• Task switching: treat meta-management as an additional task in a system that knows how to manage the state of multiple tasks and to freely switch between them without loosing their respective states.
• Temporal hierarchy: rather than actively processing meta-management concerns during first-order deliberation, do that meta-management processing during the “off-peak times” when first-order deliberation is either not occurring or is involved with only inconsequential matters.

Let’s discuss the task switching and temporal hierarchy solutions in more detail.

Meta-management via Task Switching

In humans, task switching enables us to break problems into subtasks, and to work through those subtasks without losing track of the “big picture”. It also enables us to cope with distractions that force us to attend to completely unrelated contexts and then to return to our previous task. A key feature of task switching in humans is the ability to recall our previous working memory state when we return to a prior task.

Meta-management via task-switching: instead of a separate hierarchical layer, second-order and first-order task share the same meta-MDP.

This enables an auto-meta-management compatible solution whereby meta-management becomes a sub-task that is identified and carried out by the meta-MDP, before it returns to its previous task.

Mechanisms for human-like task switching in artificial systems are often discussed in the context of Cognitive Architectures, however practical implementations of such ideas are usually based on hand-rolled symbolic systems rather than trainable neural networks (e.g., ACT-R, Borst & Anderson, 2015). There is significant neuroscientific and behavioral investigative work into task switching, particularly in terms of attempts to explain difficulties associated with task switching (Herd et al., 2014).

Alternatively, task-switching can be viewed in the context of Hierarchical RL and solved, for example, via the use of the Options framework or its variants. The Meta-learning approaches associated with Episodic RL research typically use some form of associative slot-memory to recall context-appropriate past states (Ritter et al., 2018; Botvinick et al., 2019). The latter approach could act as an episodic memory of the past task context, enabling it to be retrieved back into working memory once the meta-management task finished, with a gating mechanism to update the past task in light of the meta-management conclusion.

Meta-management via Temporal Hierarchy

Notice, however, that task switching is not free of its own costs. Firstly, time spent on meta-management is time not spent on the object-level goal, which may be time sensitive. Secondly, the longer the agent focuses on an auxiliary task, the worse the risk that they forget important details of the primary task they were attempting to carry out. In short, meta-management, like all other examples of task switching, is disruptive to other cognitive processes.

All of this takes the simplifying assumption that deliberation consumes most of the cognitive resources of the biological or artificial cognitive system. Thus there can only be one “active” deliberative process at any given time. However, some “passive” processes can continue to influence active deliberation. Examples of possible passive processes include pre-emptively setting priors on inferencing processes, and automatized “background” predictive detection of error states during active deliberation.

There is also a second angle of consideration: when the meta-management processes are carried out relative to the deliberative process that they manage.

Categories of meta-management: meta-management processing can take different forms relative to the target first-order deliberation at some immediate moment.

This gives us two axes of consideration for when and how meta-management processes influence the behavior of first-order deliberation:

• Active vs passive: whether the explicit processing of meta-management outcomes at any moment in time is the focus of the meta-MDP (i.e., it consumes the primary focus of cognition), or only influences the meta-MDP through previously calculated results or automatized background processes. This could also be conceived as foreground vs background.
• Time: time of meta-management processing relative to the first-order deliberative process that it needs to manage. This has three categories: pre-emptively affecting first-order meta-MDP behavior in anticipation of expected outcomes; directly affecting first-order meta-MDP behavior at the given immediate moment; or post-hoc review and learning from past outcomes.

Active forms of meta-management have real-world examples in human experience, offering insight for how they could be useful in artificial systems:

• Active immediate meta-management takes over execution of the meta-MDP, so that no first-order deliberation can be carried out at that moment. It is fully disruptive, but may be necessary in some extreme cases, such as when something completely unexpected has occurred and the normal deliberative processes are unable to behave correctly.
• Active preemptive meta-management corresponds to meta-management deliberation that is carried out in preparation for some anticipated future task. For example, physically or mentally reviewing notes in order to load them into working memory just before a difficult exam.
• Active post-hoc meta-management corresponds to “mulling” over past behaviors and outcomes. This results in learning from counter-factual reasoning, hindsight, and regret. An AI example of this is found in MuZero Reanalyze, which is able to learn by re-planning past episodes (Schrittwieser et al., 2020).

Passive forms of meta-management also have their correlates in human behavior, and suggest areas for future RL research:

• Passive immediate meta-management includes “background processes” that continually monitor the ongoings of first-order deliberation and produce alarms that trigger a change in processing when unexpected or problematic events occur (Sloman, 1998).
• Passive preemptive meta-management corresponds to allostatic processes.
• Passive post-hoc meta-management corresponds to automatic learning from past events. In other words, it’s just another name for RL. However, it alludes to the need for intrinsic RL — where all RL learning mechanisms are embedded within the system and carried out automatically as part of life-long learning (Parisi et al., 2019; Kudithipudi et al., 2020).

This reframes our current RL mechanisms for learning as part of a larger ecosystem of multi-timescale learning and adaptation. Most current RL learning mechanisms fit within the passive post-hoc box.

Pyramid of Reactivity

The disruptive effects discussed above are not limited to meta-management. All cognitive processing carries opportunity costs — for example in terms of energy expenditure that might have been better spent in the future, and other potential processing that is delayed, skipped, or forgotten as a result of the current focus. The more complex and involved the processing at hand, the greater the potential for opportunity costs.

Thus the brain should generally prefer forms of processing that take the least amount of time and fewest resources while still obtaining sufficient accuracy. I like to think about this as a Pyramid of Reactivity.

The least disruptive forms of processing should be carried out the most frequently — the widest part of the pyramid. Examples include fully automatized model-free overt behavior — such as walking. These consume the fewest cognitive resources, and can potentially be executed “passively”, so that deliberative processes for some unrelated task can be carried out simultaneously.

More disruptive forms of processing should be carried out according to their respective needs and costs. Habituated model-free deliberation is faster and consumes fewer resources than model-based deliberation, which involves planning and simulation of multiple trajectories. But model-based deliberation may be more accurate and could converge towards a decision faster for more complex problems.

The most disruptive forms of processing, such as active meta-management, should be carried out least often — the tip of the pyramid. Passive meta-management should generally be preferred over active meta-management, specially when the current task is important.

This kind of tradeoff strategy can already be seen in results from Resource Rational Analysis of human searching and planning behaviors (see Part 3). Humans tend to employ fast and inaccurate search heuristics when the stakes are low or when they have insufficient time to consider their decisions more carefully. In contrast, they tend to employ slower and more accurate search heuristics when the stakes are high enough, when they have enough time for consideration, and when they are not significantly distracted by other competing tasks.

Overall, the brain attempts to (approximately) solve a meta-optimization problem that balances the immediate and opportunity costs against accuracy for different deliberative tasks and strategies. Selection of meta-cognitive processing is just part of that continual balance.

Not Covered

I have to step here. Well, I have to stop somewhere, and this is as good a place as any. There are too many potential discussion points I have not touched. So let me just pick the three that come to mind.

Lifelong Learning. Today’s AIs execute short episodes and their dynamic state is then reset. Meta-management can have a part to play in proactively controlling the trajectories of these episode-style reasoning AI, but many of the longer-term meta-management processes can be simply collapsed into the training regime. Meta-management makes more sense in the context of lifelong learning agents, where models of the world must be gradually revised from experience and where weight updates are generated from learned intrinsic motivations.

There is extensive research into lifelong learning, but it remains a difficult challenge to avoid problems of task interference and catastrophic forgetting. For reviews, see that of Parisi et al. (2019) and Kudithipudi et al. (2020).

Stability and Convergence. I raised the issue of stability in the introductory article and I haven’t offered a concrete solution. Any system that uses its own models to assess its own outcomes is prone to confirmation bias at best and to exponential run-off from positive feedback at worst. The short answer is that I don’t have one yet — this needs practical experiments to classify the kinds of instabilities and to discover potential solutions. The slightly longer answer is that I suspect the solution at least partly lies in combining ideas discussed in earlier parts.

Firstly, Deep Equilibrium Models (Part 1) suggest that deliberative architectures should include Recall of current task, and should include training regimes and regularizations to encourage path independence. These directly attack the problem of convergence at the individual task level. The meta-controller + RL primitives architecture is essentially an instantiation of the Options framework, and there is plenty of research discussing its stability issues and solutions. I’m big fan of prediction-based schemes such as that of Predictive Coding and Active Inference for their inherent convergence — they natural reduce prediction error over time.

Arbitrarily queryable models: this has puzzled me for a long time. Humans posses an ability to take a concept, to spin it around in our minds, to take it apart, to put it back together in different ways, and generally to investigate that concept from multiple angles. This seems eminently useful. It should be how the model is represented in model-based RL. It should definitely be how a auto-meta-managing reasoning AI models itself.

What kind of data structure supports that? Knowledge graphs? Maybe. But the brain is a neural network and our current AI connectionist models have nothing of that kind of arbitrarily queryable capability. I’m sure that there must be a thorough discussion of this in the literature. However, aside from vague references to this idea scattered around, I’m yet to find a coherent and thorough discussion that relates arbitrarily queryable representations to connectionist implementations. More research needed here.

Conclusions

Attempting to understand the inner working of our own mind and to reproduce that in artificial form is the ultimate form of meta-cognition: a worldwide group of minds attempting to understand themselves. Speaking in 2014 about the difficulty of understanding consciousness, David Chalmers said: “Faced with this, radical ideas might be needed. We might need one or two ideas that might seem crazy.” (Chalmers at TEDTalks, 2014). Understanding the functional mechanisms of human deliberation and meta-cognition may be officially classified as an “easy problem” by Chalmers (see Hard Problem of Consciousness), but it too may need some ideas that initially seem crazy.

I hope that auto-meta-management seems crazy. It certainly did to me when I first drew that conclusion. I hope too that merging model-free, model-based, SR, and episodic RL via their primitives and learning the planning algorithm seems a little crazy. Countless papers and blog posts have described frustration in the disparity between the human experience of deliberation and current RL techniques. Crazy is what we need.

The ideas I have presented are also practical. Most of the individual components already have a long history of experimentation. The few ideas without extensive experimentation all have hints scattered throughout the literature that can be found if you are lucky enough.

And that brings me to what these articles have been about. My original interest was in the philosophical and functional aspects of Philosophy of Mind, but I later realized the relevance to understanding cognition more generally. I started this series with an idea of what meta-management is and how it relates to human deliberative thought, and I wanted to share that idea. But the framework I have presented has been inspired by a wide range of research disciplines. Behavioral psychology, developmental psychology, neuroscience, computer science, philosophy, and AI. Many of the ideas are fairly widely known, but not all.

In the end, this series has been my attempt to share those less widely known ideas. These are ideas that I believe form the hidden fundamentals for understanding and reproducing intelligent thought.

I have not developed a complete theory of meta-management. And I have not given strong proof for where and what circumstances it is needed. That was not the goal. I did not seek to provide a detailed solution, but to motivate investigation, and to set some directions.

Hopefully I have shown (1) why meta-management is probably needed, (2) the kinds of architectures that are worth investigating, (3) why meta-management and learning to plan are intimately related, and (4) directions for further research.

I hope to be a part of that research.

Acknowledgments

Unless stated otherwise, all diagrams and images are the work of the author, with AI help for the images.

Citing

For attribution, please cite this work as:

Lett, M. (2026, Aug). Can an AI Learn the Planning Algorithm? Part 4: Learning to Plan. AI Advances. https://medium.com/@malcolmlett/can-an-ai-learn-the-planning-algorithm-part4-f5a142af3c6f

References

• Besta, M., Barth, J., Schreiber, E., et al. (2025). Reasoning Language Models: A Blueprint. ArXiv. https://arxiv.org/abs/2501.11223
• Byrne, R. M. J. (2016). Counterfactual Thought. Annual Review of Psychology, 67, 135–157. https://doi.org/10.1146/annurev-psych-122414-033249. Full text: https://www.researchgate.net/publication/282126362_Counterfactual_Thought
• Bevilacqua, Y. Zhou, and B. Ribeiro (2021). Size-invariant graph representations for graph classification extrapolations. PMLR, 139, 837–851. https://proceedings.mlr.press/v139/bevilacqua21a.html
• Borst, J. P., & Anderson, J. R. (2015). Using the ACT-R cognitive architecture in combination with fMRI data. In B. U. Forstmann & E.-J. Wagenmakers (Eds.), An introduction to model-based cognitive neuroscience (pp. 339–352). Springer, New York. https://doi.org/10.1007/978-1-4939-2236-9_17. Full text: https://act-r.psy.cmu.edu/wordpress/wp-content/uploads/2014/01/BorstAndersonInPress.pdf
• Botvinick M, Ritter S, Wang J, et al. (2019). Reinforcement Learning, Fast and Slow. Trends in Cognitive Sciences, 23, 408–422. https://www.cell.com/trends/cognitive-sciences/fulltext/S1364-6613(19)30061-0
• Cappart, Q., Chételat, D., Khalil, E., et al. (2023). Combinatorial optimization and reasoning with graph neural networks (ArXiv preprint released 2021). JMLR, 24 (130), 1–61. https://jmlr.org/papers/v24/21-0449.html
• Cox, M. T., & Raja, A. (2011). Metareasoning: An Introduction. In Michael T. Cox, Anita Raja, Eds, Metareasoning: Thinking about Thinking. MIT Press. https://doi.org/10.7551/mitpress/8069.003.0003. Full text: https://www.researchgate.net/publication/284348455_Metareasoning_An_Introduction
• Da Costa, L., Parr, T., Sajid, N.,, Veselic, S., Neacsu, V., & Friston, K. J. (2020). Active inference on discrete state-spaces: A synthesis. Mathematical Psychology, 99, 102447. https://doi.org/10.1016/j.jmp.2020.102447
• Duan, Y, Schulman, J., Chen, X., et al. (2016). Rl2: Fast reinforcement learning via slow reinforcement learning. ArXiv. https://arxiv.org/abs/1611.02779
• Epstude, K., & Roese, N. J. (2008). The Functional Theory of Counterfactual Thinking. Personality and Social Psychology Review, 12(2), 168–192. https://doi.org/10.1177/1088868308316091
• Evans, J. S., & Stanovich, K. E. (2013). Dual-Process Theories of Higher Cognition: Advancing the Debate. Perspectives on psychological science: a journal of the Association for Psychological Science, 8(3), 223–241. https://doi.org/10.1177/1745691612460685
• Gershman, S. J. (2018). The Successor Representation: Its Computational Logic and Neural Substrates. The Journal of Neuroscience, 38(33), 7193–7200. Full text: https://gershmanlab.com/pubs/Gershman18_SR.pdf
• Gershman, S. J., & Daw, N. D. (2017). Reinforcement Learning and Episodic Memory in Humans and Animals: An Integrative Framework. Annual Review Psychology, 68, 101–128. https://doi.org/10.1146/annurev-psych-122414-033625
• Herd, S. A., O’Reilly, R. C., Hazy, T. E., Chatham, C. H., Brant, A. M., & Friedman, N. P. (2014). A neural network model of individual differences in task switching abilities. Neuropsychologia, 62, 375–389. https://doi.org/10.1016/j.neuropsychologia.2014.04.014
• Hafner, D., Lillicrap, T., Ba, J., Norouzi, M. (2020). Dream to Control: Learning Behaviors by Latent Imagination. ArXiv. https://arxiv.org/abs/1912.01603
• Hutsebaut-Buysse, M., Mets, K., & Latré, S. (2022). Hierarchical Reinforcement Learning: A Survey and Open Research Challenges. Machine Learning and Knowledge Extraction, 4(1), 172–221. https://doi.org/10.3390/make4010009
• Karmiloff-Smith, A. (1994). Précis of Beyond modularity: A developmental perspective on cognitive science. Behavioral and Brain Sciences, 17(4), 693–707. https://doi.org/10.1017/S0140525X00036621. Full text: https://www.researchgate.net/publication/264001979_Precis_of_Beyond_modularity_A_developmental_perspective_on_cognitive_science
• Koriat A. (1993). How do we know that we know? The accessibility model of the feeling of knowing. Psychological review, 100(4), 609–639. https://doi.org/10.1037/0033-295x.100.4.609
• Kudithipudi, D., Aguilar-Simon, M., Babb, J. et al. (2020). Biological underpinnings for lifelong learning machines. Nature Machine Intelligence, 4, 196–210. https://doi.org/10.1038/s42256-022-00452-0. Full text: https://www.researchgate.net/publication/359427017_Biological_underpinnings_for_lifelong_learning_machines
• Lee, K.-H., Nachum, O., Yang, M., et al. (2022). Multi-game decision transformers. NeurIPS. https://doi.org/10.48550/arxiv.2205.15241
• Nolte, R., Pomerlan, M., Janssen, A., et al. (2025). How Metacognitive Architectures Remember Their Own Thoughts: A Systematic Review. ArXiv. https://arxiv.org/abs/2503.13467
• Parisi, G. I., Kemker, R., Part, P. H., Kanan, C., & Wermter, S. (2019). Continual lifelong learning with neural networks: A review. Neural Networks, 113, 54–71. https://doi.org/10.1016/j.neunet.2019.01.012
• Parr, T., Pezzulo, G., & Friston, K. (2022). Active Inference: The Free Energy Principle in Mind, Brain, and Behavior. MIT Press. https://doi.org/10.7551/mitpress/12441.001.0001
• Pateria, S., Subagdja, B., Tan, A.-H., & Quek, C. (2021). Hierarchical Reinforcement Learning: A Comprehensive Survey. ACM Computing Surveys, 54(5), 1–35. https://doi.org/10.1145/3453160
• Pritzel, A., Uria, B., Srinivasam, S., et al. (2017). Neural Episodic Control. PMLR, 70, 2827–2836. https://proceedings.mlr.press/v70/pritzel17a.html
• Reed, S., Zolna, K., Parisotto, E., et al. (2022). A generalist agent. Transactions on Machine Learning Research, 11/2022, https://doi.org/10.48550/arxiv.2205.06175
• Ritter, S., Wang, J. X., Kurth-Nelson, Z., et al. (2018). Been There, Done That: Meta-Learning with Episodic Recall. PMLR. https://proceedings.mlr.press/v80/ritter18a/ritter18a.pdf
• Rosenthal, D. M. (2012). Higher-order awareness, misrepresentation and function. Philosophical Transactions of the Royal Society B: Biological Sciences, 367, 1424–1438. https://doi.org/10.1098/rstb.2011.0353
• Schrittwieser, J., Antonoglou, I., Hubert, T. et al. (2020). MuZero: Mastering Go, chess, shogi and Atari without rules [Blog post]. Google DeepMind. https://deepmind.google/blog/muzero-mastering-go-chess-shogi-and-atari-without-rules/
• Shimamura, A.P. (2000). Toward a Cognitive Neuroscience of Metacognition. Consciousness and Cognition, 9(2), 313–323. https://doi.org/10.1006/ccog.2000.0450
• Sloman, A. (1998). Damasio, Descartes, Alarms and Meta-management. SMC’98 Conference Proceedings. IEEE International Conference on Systems, Man, and Cybernetics, 3, 2652–2657. https://doi.org/10.1109/ICSMC.1998.725060. Full text: https://www.cs.bham.ac.uk/research/projects/cogaff/Sloman_smc98.pdf
• Sutton, R. S. (2019). Bitter Lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html
• Sutton, R. S., Precup, D., & Singh, S. (1999). Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112, 181–211. https://doi.org/10.1016/S0004-3702(99)00052-1. Full text: http://www-anw.cs.umass.edu/~barto/courses/cs687/Sutton-Precup-Singh-AIJ99.pdf
• Taira, M., & Sharpe, M. (2025). Complementary roles of serotonin and dopamine in model-based learning. Current Opinion in Behavioral Sciences, 61, 101464. https://doi.org/10.1016/j.cobeha.2024.101464
• Todorov, E. (2006). Linearly-solvable Markov decision problems. NIPS. https://papers.nips.cc/paper_files/paper/2006/hash/d806ca13ca3449af72a1ea5aedbed26a-Abstract.html
• Todorov, E. (2009). Efficient computation of optimal actions. PNAS, 106 (28), 11478–11483. https://doi.org/10.1073/pnas.0710743106
• Umanath, S., Coane, J.H., Huff, M.J. et al. (2023). Ecological validity of don’t remember and don’t know for distinguishing accessibility- versus availability-based retrieval failures in older and younger adults: knowledge for news events. Cogn. Research, 8, 2. https://doi.org/10.1186/s41235-022-00458-7
• Van Hoeck, N., Watson, P. D., & Barbey, A. K. (2015). Cognitive neuroscience of human counterfactual reasoning. Frontiers in human neuroscience, 9, 420. https://doi.org/10.3389/fnhum.2015.00420
• Wang, J. X, Zeb, K.-N., Dhruva, T., et al. (2017). Learning to reinforcement learn. ArXiv. https://arxiv.org/abs/1611.05763
• Xie, Y., Ma, Y., Lan, S., et al. (2025). Play to generalize: Learning to reason through game play. ArXiv. https://doi.org/10.48550/arxiv.2506.08011
• Zeelenberg, M., & Pieters, R. (2007). A Theory of Regret Regulation 1.0. Journal of Consumer Psychology, 17(1), 3–18. https://psycnet.apa.org/doi/10.1207/s15327663jcp1701_3
• Zheng, Y., Wolf, N., Ranganath, C., O’Reilly, R. C., McKee, K. L. (2025). Flexible Prefrontal Control over Hippocampal Episodic Memory for Goal-Directed Generalization. ArXiv. https://arxiv.org/abs/2503.02303