The Mathematics of “Grokking”

A network memorizes everything, understands nothing, then suddenly it does.

Training accuracy climbs almost immediately, while validation accuracy stays near chance through a long plateau — then jumps to near-perfect roughly a thousandfold later. Original diagram by the author.

Picture a single curve on a graph, plotted against a logarithmic time axis.

One line, tracking how well a neural network performs on its training data, shoots up almost immediately. Before a thousand steps of training have passed, the network has essentially memorized everything it was shown. It is perfect on the material it has seen.

A second line, tracking how well that same network performs on data it has never seen, stays flat. Pinned to chance. The network has memorized, but it has not understood. It can recite, but it cannot generalize. This is the textbook picture of overfitting, the failure mode every machine learning student is taught to fear.

And then something happens that should not happen.

Long after the network has overfit — not slightly later, but a thousand times later, near a million training steps — the second line suddenly lurches upward and races to near-perfect accuracy. The network that had only memorized abruptly generalizes. It “gets it.”

This phenomenon is called grokking, and it is one of the strangest and most revealing discoveries in modern machine learning. In the original 2022 experiment, training accuracy reached near-perfect levels before 1⁰³ steps, while validation accuracy only reached that level near 1⁰⁶ steps — a gap of roughly a thousandfold between memorizing and understanding.

The obvious question is the one that makes grokking worth a long look: what is happening inside the network during that vast, silent plateau, where nothing seems to be improving and then, suddenly, everything does? The answer turns out to connect small transformers doing modular arithmetic to the statistical physics of phase transitions, and — if a careful, still-speculative thesis holds — possibly to the human experience of insight itself.

What We See

The word itself comes from science fiction. Grok was coined by Robert A. Heinlein in his 1961 novel Stranger in a Strange Land, where it is a Martian verb meaning to understand something so thoroughly that the observer becomes part of the observed. It is a fitting name for a phenomenon about the difference between holding information and genuinely absorbing it.

The phenomenon was introduced to machine learning in a paper with a precise and unglamorous title: “Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets,” by Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. The setup was deliberately minimal. They trained small decoder-only transformers — two layers, width 128, four attention heads, around 4x10⁵ non-embedding parameters — on the kind of task that has a single correct answer and no ambiguity: binary operation tables for arithmetic modulo a prime number, p = 97, along with composition in the symmetric group S₅.

The canonical result, the one reproduced in Figure 1 of that paper, was for division modulo 97 using half the possible data for training. Training accuracy became near-perfect before a thousand steps. Validation accuracy stayed near chance through an enormous plateau and only climbed to near-perfect near a million steps. The single most effective lever for producing this late generalization was weight decay, a standard form of regularization that penalizes the network for using large internal weights.

From this came a clean operational definition of grokking, three conditions in sequence: training accuracy reaches roughly 100% early; test accuracy stays near chance through a long plateau; and then test accuracy rises sharply to near-perfect, long after the point of overfitting.

That delay is the entire mystery. In ordinary learning, generalization and memorization rise together, more or less in step. In grokking, they are separated by orders of magnitude of training time. The network spends the overwhelming majority of its training in a state that looks, from the outside, like complete stagnation. Understanding what fills that apparent emptiness is the work that the rest of the research does.

What’s Inside

For a while, grokking could have been dismissed as a curiosity of the loss landscape, a quirk with no interpretable cause. That changed when researchers managed to open the network up and read, in full, the algorithm it had secretly learned.

This is the achievement of Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt, who fully reverse-engineered a one-layer ReLU transformer trained on addition modulo 113. The model had a dimension of 128, four attention heads, and 512 MLP neurons, trained on 30% of the data for 40,000 epochs. What they found inside is the centerpiece of the whole grokking story.

The network had not learned addition the way you or I would. It had not built a lookup table, and it had not learned to carry digits the way a schoolchild does. Instead, it had quietly reinvented trigonometry.

To see what that means, picture a clock face. When you add five hours to nine o’clock, you don’t land on fourteen — you wrap around to two. The numbers live on a circle, and addition becomes rotation. “Modular arithmetic,” the formal name for the kind of wrap-around math the network was trained on, is exactly this: arithmetic where the numbers loop back on themselves like the hours on a clock. The numbers the network was working with don’t sit on a straight line stretching off to infinity. They sit on a circle, and adding two of them is just a matter of rotating around that circle by the right amount.

This is the structure the network discovered entirely on its own. Nobody told it about clocks or circles. Given only thousands of examples of wrap-around addition, it built an internal representation in which each input number is placed at a specific angle around a circle, and the act of adding two numbers becomes the act of rotating from one angle to another.

To do the rotating, the network uses sines and cosines — the mathematical tools that describe positions on a circle. Each number gets mapped onto a set of these wave functions, the network combines the waves using standard trigonometric identities, and then it reads off the answer by checking how far around the circle it has landed. In effect, the network rotates the circle by the negative of the result, and the correct answer is the spot where everything lines up. The whole machine is built out of interfering waves, which is why the researchers describe it as computing with Fourier features — the same mathematics that lets a prism split light or an audio app separate a song into frequencies.

What makes the finding so striking is how economical the network’s solution turned out to be. Out of all the frequencies it could have used, it relied on just five key ones, corresponding to specific values the researchers identified as k = 14, 35, 41, 42, and 52. The reliance was so clean that when they deliberately removed the parts of the network operating at other frequencies, performance actually improved, with the loss dropping by around 70%. The network had found a sparse, elegant, almost crystalline algorithm — and hidden it inside what looked, from the outside, like a black box.

The network learns to place numbers around a circle rather than along a line, so adding two of them becomes a rotation around the clock face. Pass the top and the count wraps back to zero — exactly like hours on a clock, and exactly like arithmetic modulo a prime. Original diagram by the author.

The metaphor the researchers used, and the one that has stuck, is a clock. Adding two numbers modulo a prime is exactly like adding hours on a clock face: 9 o’clock plus 5 hours lands on 2, because the clock wraps around. The network discovered this wrap-around structure on its own and built a continuous, rotational, wave-based machine to exploit it — sparse, economical, and entirely self-taught.

Reading the internal mechanism also let the researchers see what was happening during the silent plateau, and this is perhaps the single most important finding for dispelling the magic. They identified three phases. First, memorization, from roughly epoch 0 to 1,400. Second, circuit formation, from about 1,400 to 9,400, during which the network gradually builds the Fourier machinery. Third, cleanup, from about 9,400 to 14,000, during which the leftover memorization components are pruned away, largely by weight decay.

The crucial insight is this: the internal progress measures change continuously throughout, even though the test accuracy jumps discontinuously. The network is not idle during the plateau. It is steadily constructing its generalizing circuit the entire time. The sudden leap in test accuracy is not the moment the understanding is built; it is the moment the already-built understanding finally outweighs the memorized noise enough to show up in the output. The “aha” we observe from outside is the visible surfacing of a long, hidden, continuous process.The metaphor the researchers used, and the one that has stuck, is a clock. Adding two numbers modulo a prime is exactly like adding hours on a clock face: 9 o’clock plus 5 hours lands on 2, because the clock wraps around. The network discovered this wrap-around structure on its own and built a continuous, rotational, wave-based machine to exploit it — sparse, economical, and entirely self-taught.

Reading the internal mechanism also let the researchers see what was happening during the silent plateau, and this is perhaps the single most important finding for dispelling the magic. They identified three phases. First, memorization, from roughly epoch 0 to 1,400. Second, circuit formation, from about 1,400 to 9,400, during which the network gradually builds the Fourier machinery. Third, cleanup, from about 9,400 to 14,000, during which the leftover memorization components are pruned away, largely by weight decay.

The crucial insight is this: the internal progress measures change continuously throughout, even though the test accuracy jumps discontinuously. The network is not idle during the plateau. It is steadily constructing its generalizing circuit the entire time. The sudden leap in test accuracy is not the moment the understanding is built; it is the moment the already-built understanding finally outweighs the memorized noise enough to show up in the output. The “aha” we observe from outside is the visible surfacing of a long, hidden, continuous process.

While training accuracy is high from the very start, the network moves through memorization, then circuit formation, then cleanup — and the generalizing circuit is built continuously across all three. The sudden “aha” in test accuracy is only the moment that circuit finally outweighs the memorized noise. Original diagram by the author.

This reframes grokking entirely. There is no instant of insight inside the machine. There is a gradual reorganization that produces a sudden-looking result. The discontinuity is in our measurement, not in the underlying dynamics.

What Kind of Event It Is

Once you accept that grokking is a structured transition rather than a fluke, the natural scientific question is: what kind of transition? Here the most productive language has come from physics, and specifically from the theory of phase transitions — the mathematics that describes how water becomes ice, or how a magnet abruptly loses its magnetism at a critical temperature.

Several independent theoretical treatments converge on the idea that grokking is a phase-transition-like event, driven by regularization, by competition between a memorizing solution and a generalizing solution, and by a rise and then fall of internal complexity.

One foundational account came from Ziming Liu and collaborators, who proposed an effective theory identifying four distinct learning regimes: comprehension, grokking, memorization, and confusion. In their picture, generalization emerges only within a “Goldilocks zone” of conditions, a balance they evocatively called “intelligence from starvation” — the network is forced toward a general solution precisely because it is denied the resources to simply memorize. They visualized this with phase diagrams and with learned embeddings that organize themselves into a circle, the same circular structure that the Fourier clock would later explain mechanistically.

The phase-transition framing has since been sharpened into quantitative physics. Bojan Žunkovič and Enej Ilievski derived analytic expressions for critical exponents, for the probability of grokking, and for the distribution of grokking times. These are the signatures physicists look for when they want to claim that something is genuinely a phase transition rather than merely resembling one.

But the field has a genuine and unresolved disagreement about exactly which kind of phase transition grokking is, and intellectual honesty requires putting it plainly on the table.

On one side, Noa Rubin, Inbar Seroussi, and Zohar Ringel analyzed grokking in two-layer networks and concluded that the grokked state is analogous to the mixed phase that follows a first-order phase transition — the abrupt, discontinuous kind of transition, like water freezing, with a barrier separating the two states.

On the other side, Xiaotian Zhang, Yue Shang, Entao Yang, and Ge Zhang reframed grokking as a computational glass relaxation. In their account, memorization corresponds to the network rapidly cooling into a non-equilibrium glassy state, and generalization corresponds to the slow relaxation out of that state toward a higher-entropy, more stable configuration. Critically, they reported finding no entropy barrier of the kind the first-order picture implies — a direct contradiction of the earlier framing.

One reading of grokking: the network first settles into a fast, shallow memorization basin, then slowly relaxes into a deeper, more stable generalizing state. Whether a barrier separates the two is still debated. Original diagram by the author.

This is an active, unresolved controversy, and it has not been settled. It matters because the two pictures make different claims about whether there is a barrier between memorizing and generalizing, and therefore about the fundamental nature of the transition. A reader should hold both accounts in mind without prematurely choosing, because the researchers themselves have not converged.

Other work fills in the surrounding structure. Vikrant Varma and colleagues explained grokking through circuit efficiency: grokking occurs when a generalizing solution is slower to learn but ultimately more efficient than memorization, a framing precise enough to predict and then confirm two new phenomena they named ungrokking and semi-grokking. Branton DeMoss and collaborators identified a characteristic rise and then fall of intrinsic complexity — the network’s internal complexity climbs during memorization and then falls as it discovers the simpler generalizing pattern.

The network’s internal complexity rises while it memorizes, reaches a peak, and then falls as it discovers the simpler general rule. This rise-and-fall is one of the clearest hints that grokking behaves like a phase transition. Original diagram by the author.

That rise-then-fall is itself a strong hint of a phase transition, because it mirrors the way order parameters and correlation lengths behave near criticality in physical systems.

There are also important results showing grokking is neither fragile nor confined to toy problems. Lucas Prieto and collaborators found that without regularization, grokking tasks push the softmax function to the edge of numerical stability — a failure they called Softmax Collapse — and introduced techniques, StableMax and ⊥Grad, that enable grokking even without regularization. And Ahmed Imtiaz Humayun, Randall Balestriero, and Richard Baraniuk demonstrated that grokking appears well beyond small arithmetic networks: a ResNet18 begins grokking adversarial robustness after around 1⁰⁴ steps and reaches it around 2×1⁰⁵ steps, long after its ordinary accuracy has settled. The phenomenon, in other words, is general.

Even in a large image classifier, ordinary accuracy is mastered early while adversarial robustness “grokks” only much later — showing the delayed-generalization pattern well beyond toy arithmetic. Original diagram by the author.

It is worth drawing one clean distinction here, because it is easy to blur. Grokking is a training-time phenomenon: a single model, trained longer, suddenly generalizes. This is different from the “emergent abilities” of large language models, which are described as a function of model size. That size-based notion of emergence has itself been challenged — Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo argued that 25 of 29 metrics showed no genuine emergence when measured continuously rather than with sharp thresholds. Importantly, that critique targets size-emergence, not grokking, whose internal mechanism has been directly observed. Later work by Leonardo Berti and colleagues has connected these related threads.

The 2025–2026 Frontier

Grokking is not a closed chapter. The most recent work extends it into territory that makes the phenomenon feel less like a laboratory artifact and more like a fundamental feature of how learning systems behave.

In reinforcement learning of reasoning, researchers have reported what they describe as a striking grokking phase transition: after an extended period with near-zero reward, RL-trained models abruptly climb to near-perfect accuracy, in one run jumping after roughly 450 steps of apparent stagnation.

In reinforcement learning of reasoning, reward can sit near zero for a long stretch and then climb sharply to near-perfect accuracy — here after roughly 450 steps. The same silent-plateau-then-jump signature appears far outside the original setting. Original diagram by the author.

In the study of model safety, grokking-like dynamics have appeared in an unsettling place. Work on emergent misalignment during fine-tuning found that the behavioral transition to misalignment occurs later than the peak in gradient norm — a hidden build-up followed by a delayed surfacing, directly analogous to grokking’s silent plateau and sudden jump. The same early-warning structure that governs benign generalization may also govern the emergence of undesirable behavior.

And the physics has continued to deepen, with feature-emergence scaling laws derived from grokking dynamics, an end-to-end proof of grokking for over-parameterized ridge regression, and treatments framing grokking as a transition between competing basins through the lens of singular learning theory. The trend across all of it is the same: grokking is increasingly described not with hand-waving but with the rigorous mathematical machinery of statistical physics.

The Predictive Thesis

Everything to this point is established, sourced, and reproducible. What follows is different in kind. It is a hypothesis — an original synthesis built on established parts but reaching beyond what any single result proves. It should be read as a falsifiable prediction, not a settled finding.

The hypothesis: if sudden machine understanding has a computable phase-transition structure, then human insight — the “Eureka” moment — may share the same mathematical substrate.

The case for taking this seriously rests on three established pillars and one speculative bridge.

The first pillar is that human insight is genuinely sudden and has a measurable neural signature. Work by Mark Jung-Beeman, John Kounios, and colleagues using fMRI and EEG found that insight solutions are accompanied by increased activity in the right anterior superior temporal gyrus, and by a burst of gamma-band activity around 40 Hz beginning roughly 0.3 seconds before a person reports the solution, itself preceded by a rise in alpha-band activity over the visual cortex. Insight, in the brain, is not a smooth ramp. It is a sudden reorganization with a detectable electrical fingerprint.

A schematic of the insight signature: gamma-band activity stays near baseline, then bursts sharply about a third of a second before a person consciously reports the solution. Original diagram by the author, based on the finding in Jung-Beeman et al. 2004.

The second pillar is that this suddenness reflects representational change, not gradual accumulation. The “representational change theory” of Stellan Ohlsson, with experimental support from Günther Knoblich and colleagues, holds that insight requires discrete restructuring of how a problem is represented — through constraint relaxation and chunk decomposition — rather than the steady accrual of partial progress. This is conceptually close to what the grokking network does when it abandons its memorized lookup table for a restructured, circular, Fourier representation.

The third pillar is the critical brain hypothesis, which holds that neuronal networks operate near a phase transition. The evidence is the existence of neuronal avalanches — cascades of activity whose sizes follow a power-law distribution, with exponents near α ≈ 3/2, exactly the kind of scale-free signature that physical systems display at criticality.

A schematic of neuronal avalanches: cascade sizes fall on a straight line on log–log axes, the scale-free signature that physical systems show when poised right at a phase transition. Original diagram by the author, based on Beggs & Plenz 2003.

The speculative bridge is this. Both biological and artificial learning are increasingly described using the same mathematical language: the statistical physics of disordered systems. That shared language is real and documented. The claim that human insight and machine grokking are governed by the same underlying equation is not yet demonstrated. It is our speculation, built on these established pieces.

A network’s test accuracy stays flat through a long plateau and then jumps; a human brain signal stays quiet through incubation and then spikes at the moment of insight. The thesis asks whether the same phase-transition mathematics governs both — a speculative synthesis, not yet a proven equivalence. Original diagram by the author.

What lifts this from analogy to science is that it makes specific, testable predictions. If human insight is genuinely a phase transition, then one should expect to observe three things in the moments before an aha.

First, critical slowing-down: as the brain approaches the insight transition, its dynamics should show rising autocorrelation and variance in the EEG signal — the same early-warning logic seen in the misalignment work, where a measurable quantity builds before the behavioral jump.

Second, power-law-distributed incubation times: the waiting periods before insight should follow a heavy-tailed power law rather than a memoryless exponential distribution, which is the statistical fingerprint of critical rather than random dynamics.

Third, an identifiable order parameter: there should exist some measurable quantity, some function of brain state, that jumps discontinuously at the moment of insight, playing the role that test accuracy plays in the machine.

What Would Collapse the Thesis

A hypothesis is only worth stating if it can be killed. Here is what would reduce this one from science back to metaphor.

If careful EEG studies of human problem-solving find no critical slowing-down before insight — if the autocorrelation and variance of the signal stay flat right up to the aha — then the phase-transition framing fails on its most direct prediction.

If incubation times before insight turn out to be exponentially distributed rather than power-law distributed, then the dynamics are memoryless and random rather than critical, and the analogy to grokking’s phase structure breaks.

If no order parameter can be found — if there is no measurable quantity that jumps at the moment of insight the way test accuracy jumps in the network — then there is no transition in the physical sense, only a subjective feeling of one.

Confirming the three predictions would turn an analogy into a shared science. Failing to confirm them would leave grokking as a beautiful machine phenomenon and human insight as a separate mystery that merely resembles it. Either way, the experiments are now specifiable, which is the most a thesis at this stage can ask for.

What grokking has already given us is solid and remarkable: the demonstration that a learning system can spend the vast majority of its training in apparent stagnation while silently constructing understanding beneath the surface, and that the moment of sudden comprehension is the visible crest of a long, continuous, hidden wave. Whether the same equation that describes a transformer learning to tell time on a Fourier clock also describes the instant a human being finally sees the answer is, for now, an open question.

It is, at least, a question we finally know how to ask.