The AI Behaviors That Scientists Still Can’t Fully Explain
Grokking, in-context learning, emergent abilities, superposition, the reversal curse five behaviors that work, that have…
The AI Behaviors That Scientists Still Can’t Fully Explain
Grokking, in-context learning, emergent abilities, superposition, the reversal curse five behaviors that work, that have been measured, that have been replicated, and that no one in the field can yet account for at a mechanistic level. A deep technical walkthrough of what we see, what we don’t understand, and why the gap matters.
Artificial intelligence visualized as a digital human face integrated with glowing circuit board patterns. Image Credits: https://scitechdaily.com/how-scientists-are-finally-revealing-ais-hidden-thoughts/
The Most Honest Sentence In Modern AI Research
Some time in late 2023, in a conference talk that was widely shared in the AI safety community, Neel Nanda then at Anthropic, now at Google DeepMind opened with a sentence that captures the current state of the field as cleanly as any sentence in any paper has. “One of the core mysteries and annoying things about neural networks is that they’re very good at what they do, but that by default, we have no idea how they work.” The speaker was an AI researcher whose entire job is to understand how neural networks work. The audience was a room full of people whose entire job is to understand how neural networks work. The room agreed with him.
This article is about the specific places where that agreement holds the empirical phenomena in modern AI systems that have been documented, replicated, sometimes published in Nature and Transactions of the Association for Computational Linguistics, and which nonetheless remain unsolved at the mechanistic level. We are not talking about hypothetical concerns or speculative behaviors. We are talking about real, measured, reproducible effects that any researcher with the right equipment can demonstrate, and that no one including the people who built the systems can yet fully account for. The list is not short. The structural pattern across the list is the most important fact about modern AI: the gap between what we can build and what we can explain is growing faster than the gap is closing.
The five behaviors we will walk through in this article are grokking, in-context learning, emergent abilities, superposition, and the reversal curse. Each is a different kind of mystery. Each has been observed across multiple architectures, multiple research groups, and multiple model scales. Each has spawned its own subfield of investigation with dozens of follow-up papers. Each remains genuinely open. By the end of the article, you should have a clear technical picture of where each behavior sits, why it resists explanation under current frameworks, and what the convergence of these five mysteries says about the state of mechanistic understanding in AI as of 2026.
“These weird brains work differently from our own. They have their own rules and structure.” — Neel Nanda, Quanta Magazine, “How Do Machines ‘Grok’ Data?”, April 12, 2024
Behavior 1: Grokking — When Networks Suddenly “Get It”
The story starts in late 2021 at OpenAI. A research team led by Alethea Power was training small transformers on modular arithmetic tasks the model takes two numbers, learns to predict their sum modulo some prime, and is evaluated on a held-out test set. The team’s interest was not the math itself; modular arithmetic was a controlled testbed for studying generalization. They expected the standard pattern: the model would either learn the underlying rule and generalize, or it would memorize the training data and fail on the test set. Either way, the answer should arrive within the first few thousand training steps.
What actually happened was something nobody predicted. A colleague went on vacation and forgot to stop a training run. When the team came back, the model had done something genuinely strange. For the first several thousand steps, training accuracy had climbed to 100 percent pure memorization while test accuracy stayed at random-chance levels. The model was overfitting in the textbook sense. Standard practice would have been to stop the training at that point and conclude that the model could not generalize. But because the run was left running, the team watched the test accuracy do something it was not supposed to do. After roughly two orders of magnitude more training steps long after any reasonable researcher would have given up the test accuracy suddenly began to climb. And then it climbed to 100 percent. The model had not just learned the training data; it had discovered the underlying mathematical structure of the task and could now solve any modular arithmetic problem given to it. Power and her colleagues called the phenomenon grokking, borrowing the term from Robert Heinlein’s Stranger in a Strange Land, meaning to understand something so deeply that you become part of it.
Figure 1: Training curve for a 1-layer Transformer trained on modular addition mod 113, demonstrating clear grokking. The chart, published in Neel Nanda’s August 2022 mechanistic interpretability analysis (later expanded into the ICLR 2023 spotlight paper with Tom Lieberum), is the canonical visualization of the phenomenon. The model’s training loss descends rapidly during the memorization phase while test loss stays at random-chance levels; only after tens of thousands of additional training steps does test loss suddenly drop, marking the transition to a generalizing solution. The mechanism remains an active research topic across at least a dozen labs. Image credits: Neel Nanda and Tom Lieberum, “A Mechanistic Interpretability Analysis of Grokking,” August 15, 2022. Source: https://www.alignmentforum.org/posts/N6WM6hs7RQMKDhYjB/a-mechanistic-interpretability-analysis-of-grokking · arXiv: https://arxiv.org/abs/2301.05217
The first puzzle is that grokking should not have happened at all under the prevailing theory of neural network generalization. Classical statistical learning theory predicts that once a model has memorized its training set, it has essentially used all of its parameters to fit the data, and further training will either do nothing or actively make things worse through overfitting. Grokking violates this prediction in the most direct way possible. The model is, in some sense, more committed to the memorization solution after the first few thousand steps than it is to the eventual generalizing solution. The fact that gradient descent, applied for long enough, finds its way from one to the other is genuinely mysterious. The fact that it does so abruptly going from random-chance test accuracy to perfect test accuracy in a small fraction of the total training run is even more mysterious.
The second puzzle is what the model is actually computing once it generalizes. Neel Nanda, working from 2022 onward, set out to reverse-engineer a small transformer that had grokked modular addition. What he found was so unexpected that the result has become a touchstone in mechanistic interpretability research. The grokked model was not doing addition in any way that resembled how a human would do addition. It had instead discovered a representation of numbers as points on a circle and was performing modular arithmetic through trigonometric identities and discrete Fourier transforms. Specifically, the model would map each input number to a representation involving sines and cosines of multiples of that number, multiply and rearrange the resulting expressions using standard trig identities, and recover the answer through an inverse Fourier transform on the output. The algorithm was elegant. It was mathematically correct. It was also nothing any human teacher would have suggested.
Figure 2: Norm of rows of the embedding matrix after applying a Discrete Fourier Transform to the input space, for a 1-layer Transformer that has grokked modular addition. The sparsity pattern is the empirical fingerprint of the algorithm Nanda reverse-engineered: the model maps integer inputs into a small set of frequency components, applies trigonometric identities to compute the modular sum in the frequency domain, and reads off the answer through an inverse transform. The algorithm was learned purely by gradient descent and was not predicted or suggested by any of the human researchers involved. Image credits: Neel Nanda and Tom Lieberum, “A Mechanistic Interpretability Analysis of Grokking,” August 2022. Source: https://www.alignmentforum.org/posts/N6WM6hs7RQMKDhYjB/a-mechanistic-interpretability-analysis-of-grokking
The mechanistic interpretation Nanda established for this small case has been validated by other research groups, including the Gromov 2023 paper that provided closed-form analytic expressions for the weights of a network that has grokked modular arithmetic. But the success of mechanistic interpretation on the toy case has not generalized to a clean theory of when and why grokking occurs. The phenomenon has been documented in modular arithmetic, sparse parity learning, group operations, greatest common divisor learning, and image classification. The unifying explanation across all these cases is still a matter of active debate. Some researchers point to phase transitions in the loss landscape. Some point to the role of regularization. Some point to the lottery ticket hypothesis and the emergence of “winning” subnetworks late in training. None of the explanations cleanly account for all the observed cases. The phenomenon is real, it is reproducible, and at the level of “why does this specific network grok at this specific moment,” we do not have a complete answer.
Behavior 2: In-Context Learning — Learning Without Learning
The second open question is the one that defines essentially every modern LLM product. In-context learning is the ability of a large language model to perform a task it has never been explicitly trained on, given only a few examples of the task in its prompt without any weight updates, without any gradient steps, without any training in the conventional sense. You write a few examples of English-to-French translation in the prompt, then write a new English sentence, and GPT-5 or Claude or Gemini produces a French translation. The model was not trained on this task. The model’s parameters have not changed. The “learning” happens entirely inside the forward pass, in some way that involves the model reading the examples and inferring the implicit rule from them.
This is, when you stop and think about it, genuinely strange. A standard machine learning model can only “learn” by updating its weights through gradient descent. In-context learning looks, from the outside, like the model is doing some kind of fast learning inside its forward pass but the weights are frozen. Where is the learning happening? What is the mechanism that allows the model to extract the rule from the examples and apply it to a new input? This question has been one of the central preoccupations of mechanistic interpretability research since the 2022 Brown et al. paper introducing GPT-3 first documented the phenomenon at scale.
The current best theoretical account, developed by Anthropic’s interpretability team and collaborators, involves a specific type of attention head called an induction head. An induction head is a learned circuit in the model that, when it sees a token X followed by a token Y earlier in the context, will predict Y after seeing X again later in the context. In other words, induction heads implement a simple pattern-completion behavior at the attention level: “this thing came after that thing before, so it will come after that thing now.” Anthropic’s 2022 paper In-Context Learning and Induction Heads showed that induction heads emerge during training at a specific moment and that this moment of emergence coincides exactly with the moment when the model’s in-context learning ability suddenly improves. The emergence of induction heads is a phase change in the model’s training, and it is the proximate mechanism for the simplest form of in-context learning.
Figure 3: Loss curve for predicting repeated subsequences in a 2-layer attention-only transformer, demonstrating a phase change in the model’s behavior. The chart, from Nanda and Lieberum’s analysis, shows the same delayed-generalization pattern that defines grokking, applied to a different task. The structural similarity between this curve and Figure 1 supports the hypothesis that grokking and induction-head formation are instances of a more general phase-transition phenomenon in neural network training. The mechanistic interpretation of what is happening during the transition remains an active research topic. Image credits: Neel Nanda and Tom Lieberum, “A Mechanistic Interpretability Analysis of Grokking,” August 2022. Source: https://www.alignmentforum.org/posts/N6WM6hs7RQMKDhYjB/a-mechanistic-interpretability-analysis-of-grokking
The induction-head story is a real piece of progress, but it is also incomplete in a specific way. Induction heads explain the most basic form of in-context learning pattern completion based on previously-seen examples. They do not explain the more sophisticated forms. GPT-5 and Claude Opus can, given an in-context demonstration of a novel mathematical operation defined by examples, infer the operation and apply it to new inputs. They can be given a code example in a programming language they were trained on, plus a few examples of how to translate code from that language into a hypothetical language nobody has ever used, and they will produce reasonable translations. They can be given a complex multi-step reasoning task with three demonstrations and complete the fourth instance. None of this is straightforwardly explained by pattern completion. The proximate mechanism in these cases is something more like implicit Bayesian inference the model treats the in-context examples as evidence about the underlying task and updates its prior distribution over possible tasks but at the mechanistic level, no one has shown which circuits are doing this work or how they implement the inference.
The deeper puzzle is that in-context learning emerges from a training process that is not explicitly optimized for it. GPT-5, Claude, Gemini, Llama none of these models is trained with a loss function that rewards in-context learning. They are trained on next-token prediction. Yet the capability emerges, and improves with scale, and works on tasks the training data does not contain. The 2022 Xie et al. paper An Explanation of In-Context Learning as Implicit Bayesian Inference provides a theoretical framework, but the empirical question of which specific circuits implement which forms of in-context learning across modern frontier models is still being mapped one feature at a time.
Behavior 3: Emergent Abilities — Phase Transitions or Statistical Artifacts?
The third open question is the most contested in the field. In 2022, a team of researchers led by Jason Wei at Google published Emergent Abilities of Large Language Models in the Transactions on Machine Learning Research. The paper documented 137 capabilities across language models, identifying a subset that exhibited a particular pattern: the capability was essentially absent in models below a certain scale (showing near-random performance) and present in models above that scale (showing strong performance), with a sharp transition between the two regimes. Three-digit addition, modular arithmetic, transliteration, multi-step reasoning, several language-understanding benchmarks all showed this pattern. Wei et al. called the abilities emergent: capabilities that were not present in smaller models, that appeared abruptly at a certain scale, and that could not have been predicted by extrapolating a scaling law from below the threshold.
The claim was enormously consequential. If emergent abilities are real, then the field’s central scaling-laws framework is incomplete capabilities can appear discontinuously rather than smoothly, and we cannot reliably predict what will emerge at the next scale. This has direct implications for AI safety: dangerous capabilities might emerge without warning. It also raised the prospect that scaling is the right path to artificial general intelligence that simply making the models bigger would unlock qualitatively new abilities, repeatedly, without any structural changes to the architecture.
In 2023, a paper by Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo at Stanford titled Are Emergent Abilities of Large Language Models a Mirage? challenged the entire framework. The paper’s argument, published at NeurIPS 2023 and now one of the most-cited works in the area, makes a sharp empirical claim. The emergent abilities documented by Wei et al., the Stanford team argued, are an artifact of the metrics used to evaluate the models. When the researchers measured performance using nonlinear or discontinuous metrics like exact-match accuracy or multiple-choice grade metrics that give zero credit for partially-correct answers and full credit for fully-correct ones they observed sharp transitions. But when they re-measured the same model families on the same tasks using continuous, linear metrics like token edit distance or Brier score, the transitions disappeared. The underlying per-token error rate was decreasing smoothly with model scale. The apparent emergence, the Schaeffer paper argued, was the result of researchers choosing a metric that mapped a smooth underlying improvement to a discontinuous-looking output curve.
Schaeffer et al. quantified the effect: more than 92 percent of the emergent abilities Wei had identified in BIG-Bench appeared under just two metrics, exact string match and multiple choice grade. When evaluated with continuous metrics on the same model families, the same tasks produced smooth, predictable improvement curves. The paper’s policy implication was equally sharp: if emergence is a metric artifact, then capability prediction at the next scale is much more tractable than the Wei framework suggested, and the AI safety community can plan for incremental rather than sudden capability gains.
The field has not converged on a resolution. Wei and collaborators have responded that the Schaeffer critique is correct as far as it goes emergent abilities are easier to see under discontinuous metrics but that this does not eliminate the underlying phenomenon. Some abilities, they argue, genuinely cannot be predicted by extrapolating smaller-model performance, regardless of metric choice. The 2025 Sun and Haghighat paper Phase Transitions in Large Language Models and the O(N) Model reformulated the Transformer as a physics-inspired statistical model and identified two distinct phase transitions corresponding to temperature and parameter scaling, providing theoretical support for the emergent-phase-transition view. The 2024 Hu et al. PASSUNTIL framework provided a continuous metric that recovers smooth scaling for some tasks but not others. The empirical and theoretical debate is genuinely live as of 2026.
What is uncontested is the underlying observation. Some specific tasks really do show large performance jumps at specific model scales. The question is whether these jumps are fundamental properties of the scaling process or whether they reflect how researchers chose to measure them. The answer matters for AI safety, for capability forecasting, for resource allocation in frontier model training, and for the basic theoretical question of how scaling produces capability. As of this writing, scientists genuinely do not know.
Behavior 4: Superposition — When Networks Pack More Features Than They Have Dimensions
The fourth open question is the one that breaks the most basic intuition about how neural networks work. The intuition is that each neuron in a trained network corresponds to a single concept neuron A activates for “red”, neuron B activates for “round objects”, neuron C activates for “dog snouts”, and so on. This intuition is appealing because it would make interpretability easy: to understand what a model is doing, you would just need to label the neurons. The intuition is also, for any modern large language model, mostly wrong.
In 2022, a team at Anthropic and Harvard published a paper titled Toy Models of Superposition that documented a phenomenon they called, accurately, superposition. The paper showed, in carefully controlled toy networks, that neural networks routinely represent more features than they have dimensions. A network with 100 neurons in a layer does not represent 100 features. It represents 1,000, or 10,000, or possibly more encoded as overlapping linear combinations across the 100 neurons. Each neuron, instead of corresponding to a single concept, ends up responsive to many unrelated features. This property is called polysemanticity, and the paper’s main contribution was showing that polysemanticity emerges naturally from superposition, which emerges naturally when the model needs to represent more features than it has dimensions.
The mechanism is intuitive once you see it. If a network has 100 neurons and needs to represent 1,000 features, it cannot do this in any orthogonal basis there simply are not 1,000 orthogonal directions in 100-dimensional space. But if the features are sparse (most of them are zero most of the time), the network can encode them in nearly-orthogonal directions and rely on the nonlinearity of the activation function to filter out the interference between overlapping features. The result is a compression: the network packs many features into few dimensions, accepts some loss from interference between them, and ends up with a representation that is more efficient than any orthogonal encoding would be. The toy models paper showed this mechanism explicitly in controlled networks and identified the conditions under which it occurs specifically, that features need to be sufficiently sparse, and that there is a phase transition between regimes where features are represented separately and regimes where they are represented in superposition.
The implications for interpretability are severe. If a frontier LLM represents millions of features in a few thousand neurons, then “look at neuron X to understand what the model is doing” is not a viable strategy. Every neuron will respond to dozens of unrelated concepts, and the actual feature representation lives in the combination of neuron activations, not in any single neuron. This is the structural reason that mechanistic interpretability is hard. It is also the structural reason for the wave of sparse autoencoder work that has dominated interpretability research from 2023 onward the Anthropic Scaling Monosemanticity paper, the OpenAI superposition-and-dictionary-learning work, the academic interpretability community at Stanford, MIT, and ETH Zurich.
What scientists do not yet understand is the precise relationship between superposition and the model’s downstream behavior. The toy models paper established that superposition exists and characterized when it appears. The scaling monosemanticity work showed that sparse autoencoders can recover interpretable features from production models. The Anthropic 2024 Mapping the Mind of a Large Language Model paper extracted 34 million features from Claude 3 Sonnet. But the question of how those features compose to produce specific behaviors why a particular input produces a particular output through a particular circuit of activated features is still being worked out one feature at a time. The forward pass of a frontier LLM is, even with the latest tools, mostly opaque. Superposition is a documented mechanism for why this is hard. It is not yet a complete explanation of what the model is actually computing.
Behavior 5: The Reversal Curse — When Logic Fails
The fifth and last behavior we will cover is the simplest to state and, in some ways, the most disorienting. In 2023, Lukas Berglund and a team including researchers from Vanderbilt, Apollo Research, NYU, the University of Sussex, and Oxford published a paper titled The Reversal Curse: LLMs trained on “A is B” fail to learn “B is A”. The result is in the title. If you train an autoregressive language model GPT-3, Llama, any of the standard architectures on the sentence “Olaf Scholz was the ninth Chancellor of Germany,” the model learns this fact. Ask the model “Who was Olaf Scholz?” and it will tell you. But ask the model “Who was the ninth Chancellor of Germany?” and it will not return “Olaf Scholz.” It will return some other plausible-looking name, with no higher probability assigned to the correct answer than to a random alternative.
This is, by any normal standard, a failure of basic logical deduction. If “A is B” is true, then “B is A” follows by the symmetry of identity. Every formal logic system handles this. Every knowledge graph handles this. Most humans handle this without conscious effort. The Reversal Curse documents that autoregressive LLMs, fine-tuned on fictitious facts to control for memorization effects, fail to make this inference. The failure is not subtle. The probability the model assigns to the correct answer in the reverse direction is statistically indistinguishable from random, even after extensive training, even with hyperparameter sweeps, even with data augmentation strategies designed to encourage symmetry. The paper documents experiments across GPT-3, Llama-1, and additional model families, with consistent results.
The mechanism, as documented in the follow-up research, is that autoregressive language models learn directional associations rather than symmetric ones. When trained on “A is B,” the model updates the probability distribution P(B | A) that is, given A, predict B is likely. It does not, in any explicit way, update P(A | B) given B, predict A is likely. The two are different conditional probabilities in the model’s parameter space, and training on one does not automatically update the other. From the model’s perspective, “Olaf Scholz” and “the ninth Chancellor of Germany” are not symmetric labels for the same entity; they are two text strings that happen to co-occur in a specific order in the training data. The model learns the order. It does not learn the underlying identity.
The unexplained part is why the model does not learn the underlying identity given the enormous quantity of “B is A” patterns that appear in its pretraining data alongside “A is B” patterns. The training corpus for any frontier LLM contains both orderings of every fact that appears in it Wikipedia, for instance, will mention “Olaf Scholz was the Chancellor” and “the Chancellor was Olaf Scholz” both, in many articles, in many contexts. The model has the data. The model has the capability to learn either direction independently when trained on each separately. Yet the model does not learn to generalize across the two directions even when presented with enormous quantities of both. The Reversal Curse is, in this sense, a failure of generalization at the most basic logical level the field can measure.
The paper sparked an active research subfield. The 2024 follow-up work documented that the curse persists across model sizes, across training regimes, across alternative architectures including diffusion language models, and even across explicit attempts to engineer around it. Some researchers have argued that the curse can be partially mitigated by reversing the training data and presenting both orderings explicitly but doing so doubles the training cost and does not fully eliminate the asymmetry. Others have proposed that the curse reflects a more fundamental limitation of the autoregressive objective itself, and that breaking the curse would require moving to bidirectional or causal-symmetric training objectives that current frontier LLMs do not use. The mainstream frontier models in 2026 GPT-5, Claude Opus 4.6, Gemini 3 all still exhibit a measurable Reversal Curse when tested on fictitious facts.
What makes this an unexplained behavior is the gap between the model’s apparent intelligence on many tasks and its inability to make this simple logical inference. A model that can explain the symmetry of identity in formal logic, that can reason about counterfactuals across thousands of tokens, that can produce passable mathematical proofs that same model cannot reliably infer “B is A” from “A is B” when the relation has not appeared in both directions in its training data. The phenomenon is real, it is reproducible, it is documented across hundreds of follow-up experiments, and at the level of “why this asymmetry persists despite massive scale,” we do not have a complete answer.
What All Five Mysteries Have in Common
Step back from the specific behaviors and a pattern emerges. Every one of the five mysteries we have walked through has three structural properties in common.
The first is that the behavior is reproducible and measured to high precision. We know exactly when grokking occurs on modular arithmetic. We know the model size at which induction heads form. We know the metric-dependence of emergent abilities. We can count the features in superposition. We can measure the reversal curse with statistical confidence. These are not handwaving observations; they are quantitatively pinned-down phenomena.
The second is that the mechanistic explanation is partial or contested. Grokking has multiple competing explanations and no consensus. In-context learning has the induction-head story for the simplest case and nothing solid for the more sophisticated cases. Emergent abilities have an active debate between the phase-transition and metric-artifact frameworks. Superposition is well-characterized in toy models but not in production-scale networks. The reversal curse has a directional-conditional-probability story that explains the symptom but does not predict the cure. In each case, we are in the position of knowing what the model is doing without knowing why it is doing it.
The third is that the gap has consequences. If we cannot predict when grokking will occur, we cannot rely on it for production training runs. If we cannot characterize in-context learning beyond induction heads, we cannot predict which novel tasks a model will handle well. If we cannot resolve the emergent-abilities debate, we cannot reliably forecast frontier capabilities at the next scale. If we cannot fully account for superposition, we cannot audit the internal computations of deployed models. If we cannot solve the reversal curse, we cannot trust LLMs to perform basic logical deductions in safety-critical contexts. Each unexplained behavior maps onto a specific practical limitation in how we can build, deploy, and trust AI systems.
The convergence of these three properties is the most important fact about the state of mechanistic understanding in AI as of 2026. We have systems that work well enough to deploy at scale, that produce genuinely useful capabilities across hundreds of domains, that are commercially worth tens of billions of dollars per quarter and we cannot, at the mechanistic level, fully explain how any of it is happening. The field is in the position of someone who has built a working engine without yet understanding combustion. The engine runs. The thermodynamics is genuinely unsolved.
“It would be very convenient if the individual neurons of artificial neural networks corresponded to cleanly interpretable features of the input. Empirically, in models we have studied, some of the neurons do cleanly map to features. But it isn’t always the case that features correspond so cleanly to neurons, especially in large language models where it actually seems rare for neurons to correspond to clean features.” — Elhage, Hume, Olsson, Schiefer et al., Toy Models of Superposition, Anthropic & Harvard, September 14, 2022
Why These Mysteries Are the Frontier of AI Research
The list of behaviors we have walked through is not the full list. Other genuinely unexplained phenomena include the success of chain-of-thought prompting (why does “think step by step” produce measurably better outputs?), inverse scaling (why do some capabilities get worse with scale?), the persistence of hallucinations (why can models not reliably distinguish what they know from what they do not?), and the structural similarity between deep network training dynamics and physical phase transitions (why does the same mathematics describe both?). Each of these is its own active research field, with its own published papers, its own competing theoretical frameworks, and its own gap between empirical observation and mechanistic explanation.
The reason these particular mysteries are the frontier of AI research, rather than a side discussion within it, is that the practical capability of AI systems is now decoupled from our theoretical understanding of them. The models are getting more capable faster than the theoretical framework is catching up. Anthropic’s Mapping the Mind paper extracted 34 million features from Claude 3 Sonnet a remarkable feat but Claude 3 Sonnet itself has been superseded twice since publication, and the next generation of models will require new feature extractions to interpret. The field’s interpretability tools are running a race against the field’s capability tools, and the capability side is winning.
This is the structural reason that mechanistic interpretability is now considered one of the highest-leverage research directions in AI. The Anthropic interpretability team, the OpenAI superalignment work (before its 2024 restructuring), DeepMind’s safety team, and the academic interpretability community at universities across the world are all racing to build tools that can look inside a model and explain what each forward pass is actually computing. Sparse autoencoders. Activation patching. Circuit analysis. Feature steering. These are the techniques that will determine, over the next decade, whether deployed AI systems are auditable or opaque. The work is real. The progress is meaningful. It is also nowhere near complete, and every new frontier model release widens the gap between what we have built and what we can explain.
For the wider technical community, the practical implication is that trusting an AI system requires a different kind of evidence than trusting traditional software. Traditional software has source code that humans can read; if you want to know why the program behaves a certain way, you trace the code path. AI systems do not have an equivalent. The “code” is a tensor of billions of floating-point numbers, and tracing the path through it requires the kind of interpretability work we have been discussing. Until interpretability catches up to capability, the only way to know whether a deployed AI system will behave correctly is to test it extensively in the conditions you care about, accept that the test will not be exhaustive, and architect your systems around the assumption that the model will sometimes produce outputs that nobody not you, not the lab, not the engineers who wrote the training code can explain in the moment.
What This Means in Technical
For engineers, founders, researchers, and serious operators in the AI economy, the implications of the five unexplained behaviors translate into four operational principles.
First, treat capability prediction as inherently uncertain. The emergent-abilities debate is unresolved, which means that the next generation of frontier models may or may not unlock specific capabilities at predictable scales. Plan your roadmap around the assumption that capability arrival times are stochastic rather than deterministic, and build systems that can absorb sudden capability gains rather than depending on smooth, predictable improvement.
Second, do not trust models on logical-symmetry tasks without verification. The Reversal Curse is the most direct example of a class of basic-logic failures that frontier LLMs exhibit despite being able to discuss the relevant logic at length. Any production system that depends on the model correctly inferring “B is A” from “A is B” entity disambiguation, knowledge base completion, fact verification, any task requiring symmetric reasoning over identity needs explicit verification. The model knows the symmetry exists in principle. The model fails to apply it reliably in practice.
Third, invest in observability rather than only in capability. The structural gap between what AI systems can do and what we can explain means that deployed systems will produce surprising behavior in ways their builders did not anticipate. Logging, monitoring, anomaly detection, and the ability to inspect model behavior at runtime are the practical compensations for the absent mechanistic understanding. Teams that build observability infrastructure for their LLM-powered applications will catch problems that teams without it will not.
Fourth, follow the interpretability research. Mechanistic interpretability is the most strategically important research direction in AI, and the papers are public. Anthropic’s transformer-circuits.pub publications. OpenAI’s interpretability work. DeepMind’s Gemma Scope project. The academic interpretability community at Stanford, MIT, ETH Zurich. Reading these papers not just the abstracts, but the figures and the methodology is the highest-leverage learning any technical reader can do in 2026. The community is small enough that a serious individual can become a meaningful contributor within twelve to eighteen months of focused study.
The Bottom Line
Five behaviors. Grokking: networks suddenly learning the task long after they should have failed. In-context learning: models acquiring new abilities from a handful of prompt examples without any weight updates. Emergent abilities: capabilities that appear sharply at specific scales, or depending on whom you ask that appear smoothly when measured with the right metrics. Superposition: networks packing far more features into their dimensions than orthogonal encoding would allow. The Reversal Curse: LLMs failing to make the most basic logical inference about identity. All five are documented. All five are reproducible. All five remain unexplained at the mechanistic level. All five define the current frontier of what serious AI researchers are working on.
The temptation, when reading a list like this, is to land on either of two narrative endings. The first is that AI is fundamentally mysterious in ways that make it dangerous, and that we should slow down until we understand it better. The second is that these are technical curiosities that will be solved by routine engineering effort, and that we should ignore them. Both endings are wrong. The behaviors are real, technically specific, and being investigated at high effort by serious people. The work is producing genuine progress Nanda’s grokking analysis, the induction-head story for in-context learning, the Anthropic feature extraction work, the Schaeffer mirage critique, the Berglund reversal curse follow-ups. The progress is also nowhere near closing the gap, and the gap is growing as the frontier models grow.
The single most strategic technical investment any reader of this article can make is to read the primary sources. The Power et al. 2022 grokking paper. The Brown et al. 2020 GPT-3 paper. The Wei 2022 emergent abilities paper. The Schaeffer 2023 mirage paper. The Elhage et al. 2022 toy models of superposition paper. The Berglund 2023 reversal curse paper. The Nanda 2022 grokking interpretability analysis. Each is freely available. Each is well-written. Each rewards close reading with intuitions that will outlast the specific products and benchmarks of any given year. The asymmetry between people who have read these papers and people who have not is now one of the most strategically important asymmetries in technology.
The future of AI is being written by the people who can close the gap between what these systems do and how they do it. The papers are public. The math is precise. The window for serious engagement is open. The mysteries are not the obstacle to building better AI; they are the doorway to it.
If this piece helped clarify the actual technical situation for you, share it with the engineer, founder, or curious colleague who still thinks AI is “just predicting the next token.” The next-token prediction story is correct as far as it goes, and it explains essentially none of what makes these models interesting. The five behaviors above are why the field is genuinely alive right now, why the labs are spending tens of billions of dollars on interpretability research, and why the conversation about what comes next is still wide open.
References
메타데이터
- post_id
- b91ca3345f17
- slug
- the-ai-behaviors-that-scientists-still-cant-fully-explain-b91ca3345f17
- url
- https://medium.com/data-science-collective/the-ai-behaviors-that-scientists-still-cant-fully-explain-b91ca3345f17
- canonical_url
- https://medium.com/data-science-collective/the-ai-behaviors-that-scientists-still-cant-fully-explain-b91ca3345f17
- author_url
- https://medium.com/@hayanan
- status
- ok
- fetched_at
- 2026-06-09 15:37:30
