Mapping Reality: From Ancient Navigation to AI’s Spatial Innovation

“We assert that the central office itself is far more like a map control room than it is like an old-fashioned telephone exchange. The stimuli, which are allowed in, are not connected by just simple one-to-one switches to the outgoing responses. Rather, the incoming impulses are usually worked over and elaborated in the central control room into a tentative, cognitive-like map of the environment.”

• Edward Tolman (whose building I spent some great years working in)

I’m sure you’ve experienced waking up in the middle of the night and automatically making it to the bathroom, somehow accurately navigating in complete darkness. It turns out that our brain’s ability to construct and maintain detailed internal maps of the world around us are also the key to intelligence. Our nervous system must constantly convert the messy flood of sensory signals of changing patterns of light, sound, touch, smell, and movement into stable, three-dimensional representations that allow us to navigate, recognize objects, and interact with our environment. Neuroscientists have been studying these representations for decades and are just now driving advances in artificial intelligence. From evolution’s elegant solution to the Thousand Brains Project’s Monty, intelligence is based on reference frames, which are the brain’s coordinate systems and the foundation of intelligent behavior, transforming raw sensory information into meaningful, navigable maps of our reality.

What Are Reference Frames? The Foundation of Spatial Cognition

A reference frame is “a cognitive reality in which they compute their environment based on a reference point”. It’s the coordinate system your brain uses to organize spatial information and make sense of where things are in relation to each other, just like your bedroom and bathroom locations. They are the invisible scaffolding on which our spatial understanding is built. The brain primarily operates with two fundamental types: egocentric reference frames, where locations are defined relative to your own body (the coffee shop is “to my left,” the door is “behind me”), and allocentric reference frames, where spatial relationships exist independently of your position (the library is “north of the park,” the red car is “next to the blue one”). We are constantly using both types of reference frames. When you’re searching for your car in a crowded parking lot, you might initially use egocentric cues (“I parked near the entrance where I walked in”) but switch to allocentric references (“it’s two rows east of the shopping cart return”) if beneficial. When you give directions to someone, you might choose between self-centered instructions (“turn left at the stop sign”) versus world-centered landmarks (“head west toward the Bay”). Your brain switches between these coordinate systems depending on the context and what information proves most useful.

Allocentric and egocentric organization of spatial and nonspatial conceptual domains.

Spatial cognition emerges from a network of brain regions working together, each contributing specialized processing capabilities to our understanding of space. The parahippocampal place area (PPA), retrosplenial complex (RSC), and occipital place area are important nodes in the spatial navigation system, with each region playing a role in how we encode and retrieve spatial information. Research using functional magnetic resonance imaging has demonstrated that different brain areas are involved in viewer-, object-, and landmark-centered judgments about an object’s location. This finding suggests that our brains maintain multiple, specialized systems for different types of spatial reasoning. Spatial memory appears to be supported by multiple parallel representations, including egocentric and allocentric representations, meaning your brain doesn’t rely on a single “master map” but instead uses several concurrent spatial models that can be used depending on the task at hand. This redundancy explains why you can still navigate even when some spatial cues are missing, and the flexibility that allows you to seamlessly switch between different spatial perspectives as needed.

The Evolutionary Journey from Simple Navigation to Complex Cognition

The story of reference frames begins millions of years ago with the challenge of staying alive in a complex, ever-changing environment. For our earliest mammal ancestors, spatial cognition was a matter of life and death. Successfully tracking the locations of food sources, remembering where predators had been spotted, and finding the way back to safe shelter required sophisticated mental mapping abilities that could operate reliably across seasons, weather conditions, and landscape changes (more than simple steering could provide). Grid cells in the hippocampal formation create spatial maps, while place cells fire when animals are in specific locations, together forming a biological navigation system that has helped our ancestors survive for millions of years. However, evolution didn’t stop there. The “phylogenetic continuity hypothesis” suggests that neural mechanisms supporting spatial navigation have been adapted to organize concepts and memories through spatial codes, meaning the same neural machinery that once helped our ancestors track mammoths across the savanna now helps us navigate everything from social hierarchies to mathematical concepts.

What makes human cognition special is how evolution reused these ancient spatial systems for new domains. Scientists have discovered that grid cells in the hippocampal complex can encode abstract reference frames, such as the conceptual space of an object’s properties, basically using the same neural infrastructure for both physical and conceptual navigation. This expansion changed cognitive maps from just representations of physical environments into sophisticated frameworks for organizing any kind of relational information, from family trees to musical scales to mathematical theorems. The evidence for this spatial foundation of thought can be seen in our language itself: we say, “moving forward with plans,” “getting around problems,” “reaching conclusions,” and “grasping concepts,” deeply spatial metaphors that structure our abstract reasoning. When you’re “trying to wrap your head around” a difficult idea or feeling “lost” in a complex argument, you’re illustrating how your brain uses spatial processing mechanisms to navigate conceptual terrain, organizing abstract relationships using the same reference frame principles that guide us through physical landscapes.

Spatial terminology from reference frames.

The evolutionary success of reference-frame cognition is based on several computational advantages that provided our ancestors with decisive survival benefits. Spatial memory is supported by multiple parallel representations rapid decision-making because pre-computed spatial relationships could be accessed instantly without recalculating distances and directions from scratch each time. That difference is important when facing a charging predator or competing for limited resources. The flexibility offered by multiple reference frames allowed adaptation to very different contexts: the same cognitive architecture that tracked seasonal migration patterns could be redeployed for understanding social relationships or planning tool construction sequences. This system’s hierarchical organization provided efficiency, reducing computational load by organizing information at multiple scales without overwhelming the brain’s processing capacity. Spatial principles demonstrated strong generalization power, transferring across domains and enabling the cognitive flexibility that would eventually give rise to language, mathematics, and abstract reasoning. Evolutionarily, organisms with more sophisticated spatial cognition could better exploit environmental areas, engage in more complex social behaviors, and ultimately develop the cultural and technological innovations that distinguish human intelligence.

The Thousand Brains — Monty’s Breakthrough in 3D World Representation

The Thousand Brains Project departs from conventional artificial intelligence approaches by using a sensorimotor learning framework based on the Thousand Brains Theory of the neocortex that reimagines how intelligent systems should be built. Named after Vernon Mountcastle, who proposed cortical columns as a repeating functional unit across the neocortex, the project’s first implementation, lovingly called Monty, that rather than processing information through massive, monolithic neural networks like ChatGPT, Claude, and Gemini, the brain achieves its capabilities through thousands of semi-independent processing units working in parallel. The project’s key insight changes how we think about machine learning: each learning module operates as a semi-independent unit that can model entire objects, represents information through spatially structured reference frames (why they’re so important), and both estimates and effects movement in the world. This approach is quite different than today’s deep learning systems, which require massive datasets and struggle with continual learning and catastrophic forgetting, by creating a framework where intelligence emerges from the coordinated activity of many smaller, specialized modules rather than from brute-force computation.

Monty’s architecture mirrors the modular organization of the mammalian neocortex through four interconnected components that work together to create robust world representations via reference frames. At the heart of the system are Learning Modules (LMs), where each LM works as a stand-alone unit and can recognize objects on its own providing the redundancy and specialization that makes biological intelligence work. Sensor Modules serve as the system’s interface with the world, converting raw sensory data from cameras, touch sensors, or other inputs into a common spatial language that all learning modules can understand and process, just like all our sensory inputs are just electrical spikes to our brain. Motor Systems enable the crucial active exploration component, allowing Monty to move, rotate, and manipulate objects to test hypotheses and gather the multi-perspective information necessary for building complete 3D models. Tying everything together is the Cortical Messaging Protocol (CMP), a communication system where everything is expressed as features at poses relative to a common reference frame such as the body, ensuring that information can flow seamlessly between modules while maintaining spatial coherence across the entire system.

Monty is different from other AI systems in its deep commitment to spatial representation as the foundation of all learning and inference. All models have an inductive bias towards learning objects within a 3-dimensional space, complemented by a temporal dimension, meaning the system builds genuine spatial models that capture how objects exist and move through space over time. The framework uses body-centric coordinates that serve as a common reference frame for spatial computations and provide a stable foundation that allows different sensors and learning modules to coherently integrate their information. This spatial approach proves remarkably flexible: the system can handle any kind of graph structure necessary which is ≤ 3D space (strings, graphs defined by edges, or 3D point-clouds), enabling Monty to represent everything from simple geometric shapes to complex hierarchical objects. By grounding all representations in spatial relationships, Monty can potentially extend beyond physical objects to represent abstract concepts using the same fundamental framework.

Monty embodies the principle that true intelligence emerges through active interaction with the world, just like we see with children exploring to learn about the world around them. Objects are learned through movement and interaction, with the system building comprehensive models by actively exploring different perspectives, testing hypotheses, and integrating information over time. This active approach enables advanced capabilities through parallel processing and multiple perspectives to enhance both speed and accuracy, combining multiple LMs to speed up recognition (e.g., recognizing a cup using five fingers vs. one). The learning process is also very efficient, characterized as a quick, associative process, similar to Hebbian learning in the brain, which allows rapid adaptation without the extensive training periods required by deep learning systems. This sensorimotor approach captures a basic feature of how biological intelligence works by actively manipulating them, rotating them, and building multi-sensory models through direct interaction, exactly what Monty and human children do as they explore their environment.

Monty’s approach has four key advantages that directly address the limitations of current AI systems. First, its biological inspiration means it mimics how cortical columns process information in living brains, utilizing millions of years of evolutionary optimization rather than trying to reinvent intelligence from scratch. Second, the use of multiple independent models provides inherent robustness, because if one learning module makes an error or encounters corrupted input, others can compensate, creating a system that degrades gracefully rather than failing catastrophically. Third, modular architecture enables remarkable scalability, allowing complex hierarchical representations to emerge naturally as modules combine their knowledge, while new modules can be added without disrupting existing capabilities. Finally, Monty achieves true continual learning where there is no clear distinction between learning and inference, just like how we are always learning and making inferences. Monty is a system that continuously adapts and improves through experience, much like biological intelligence does throughout an organism’s lifetime.

The Benefits of Reference Frame Intelligence

Reference frame intelligence gives us some important computational advantages that provide a new avenue for artificial systems to understand and interact with the world. Spatial structure reduces search space exponentially — instead of considering every possible relationship between objects, the system can immediately eliminate vast numbers of impossible configurations based on spatial constraints, making recognition and reasoning orders of magnitude more efficient. This spatial ability enables predictive power, allowing systems to anticipate hidden properties of objects based on partial observations: seeing one side of a coffee cup immediately suggests the existence of a handle, an interior volume, and specific affordances for grasping and drinking. Spatial principles learned from one context can generalize across objects and situations, meaning a system that understands how to grasp a bottle can immediately apply that knowledge to cups, tools, and countless other objects with similar spatial properties. Reference frames also enable true compositional understanding, where complex objects are naturally understood as hierarchical arrangements of simpler parts, allowing systems to build sophisticated world models from spatial building blocks rather than requiring separate training for every possible object configuration.

In robotics, systems like Monty allow more natural interaction with physical environments by building spatial models rather than relying on pre-programmed responses. This ability allows robots to adapt fluidly to new objects and situations while maintaining spatial awareness across complex manipulation tasks. Autonomous vehicles will benefit from better understanding of 3D spatial relationships, moving from simple obstacle detection to truly comprehending the spatial dynamics of traffic flow, pedestrian behavior, and complex parking maneuvers. Augmented and virtual reality applications can use reference frame intelligence to create more intuitive spatial interfaces that understand how users naturally think about and manipulate objects in three-dimensional space, eliminating the awkward disconnect between digital tools and spatial cognition. Medical imaging could get enhanced 3D visualization and analysis capabilities that help surgeons better understand complex anatomical relationships, enable more accurate diagnoses through spatial pattern recognition, and support the development of personalized treatment plans based on individual spatial anatomy.

Understanding reference frame intelligence also shows how we can use our brain’s spatial foundations to enhance almost every aspect of mental performance. Spatial organization improves memory recall because information encoded within spatial frameworks becomes more accessible and resistant to forgetting, such as the ancient “method of loci” used by memory champions that uses these reference frame principles to achieve extraordinary recall abilities. Spatial thinking enables remarkably creative problem-solving by allowing us to mentally manipulate relationships, visualize solutions, and explore “what-if” scenarios in our mind’s eye before committing to actions in the real world. Reference frames provide powerful scaffolding for accelerated learning, helping us organize new knowledge within familiar spatial structures and creating conceptual frameworks that make complex information more comprehensible and retainable. Spatial thinking underlies everything from mathematical reasoning to social cognition, with insights into how we can optimize learning and problem-solving across all domains of human activity.

Conclusion

Reference frames may be the universal language of intelligence, the central computational foundation that underlies everything from the simplest navigation tasks to the most abstract reasoning processes. Systems like Monty push us towards this spatial future, showing us how biologically-inspired AI built on reference frame principles can achieve robust learning and flexible reasoning that current deep learning approaches can’t match. In the near term, we can expect improved robotics and spatial AI applications that interact more naturally with physical environments. In the medium term, we’ll likely see the emergence of truly intuitive human-computer interfaces that understand and utilize our spatial cognition. And in the long term, artificial general intelligence may finally emerge from systems built on these spatial foundations rather than brute-force computation. Realizing this vision will require collaboration between neuroscience, AI, and cognitive science, to tackle the deep questions about how spatial intelligence emerges and operates. This research suggests we need to better understand our own spatial cognition, a practical tool for enhancing human learning, problem-solving, and interaction with the increasingly intelligent systems that will shape our future.