Recursive Refinement as Approximate Hopfield Dynamics: How Iterating Through Your Own Output Sharpens Representations

by Mikey Bonsignore

There is a relationship hiding in plain sight between two ideas that developed independently. One is the recursive refinement technique used in tiny language models, where the output of a forward pass is fed back as input for another pass. The other is the dynamics of modern Hopfield networks, where a state vector is iteratively updated until it converges to a stored pattern. They are not identical, but the attention mechanism at the core of the transformer is mathematically the same as the Hopfield update rule, and this connection explains much of what we observe when recursive refinement works.

This paper lays out the relationship, identifies where the analogy holds and where it breaks down, and proposes experiments to distinguish the mechanisms at play.

The Established Equivalence

In 2020, Ramsauer and colleagues published “Hopfield Networks is All You Need,” establishing that the attention mechanism used in transformers is mathematically identical to the update rule of a modern Hopfield network with continuous states. The update rule is:

new state = X times softmax of (beta times X-transpose times current state)

Here X is the matrix of stored patterns, the current state is the query, and beta is an inverse temperature parameter that controls how sharply the softmax peaks. When beta is low, the softmax distributes attention broadly across all stored patterns. The network averages. When beta is high, the softmax concentrates on the single most similar stored pattern. The network commits.

In standard transformer attention, beta equals one over the square root of the key dimension. It is fixed. The network gets one shot to retrieve the relevant pattern from a single application of the update rule.

This equivalence between attention and Hopfield retrieval is mathematically proven. It is not an analogy. A single attention computation is a single Hopfield update step.

What Recursive Refinement Does Empirically

In a series of experiments on tiny recursive models conducted over the past several months, I found that feeding a model’s output back as its own input for multiple forward passes produced dramatic improvements. A model with roughly one million parameters achieved 99.2 percent improvement over its single-pass baseline on word-level language modeling. It did this without overfitting, even when training loss approached zero. The model appeared to be learning a meta-skill of iterative refinement rather than memorizing sequences.

The key empirical observations were these. The first one or two recursions produced the largest gains. Subsequent recursions showed diminishing returns. The model generalized well despite very low training loss. And the technique required sufficient model scale to work: small models with inadequate representational capacity could not benefit from recursion in the same way.

The Hopfield Interpretation

Because each attention computation is a Hopfield update, it is natural to ask whether recursive refinement is doing something like iterated Hopfield retrieval. If the attention mechanism dominates the recursive dynamics, then each pass through the network approximates another retrieval step, progressively sharpening the representation toward a stored pattern.

The theoretical framework for understanding this comes from Koulischer and colleagues at Ghent University, who analyzed phase transitions in modern Hopfield networks. They showed that for stored patterns with a given norm and separation angle, the dynamics of the network are governed by an effective inverse temperature:

beta-effective = beta times the squared norm of the patterns times (one minus cosine of the separation angle)

There is a critical value of beta-effective below which the energy landscape has only a single global attractor. Above this critical value, pattern-specific energy minima appear. The network transitions from averaging over everything to committing to specific stored memories.

If recursive refinement approximates iterated Hopfield retrieval, then each pass could increase the effective sharpness of the retrieval by amplifying salient activations and increasing the effective separation between competing patterns. The first pass identifies the most relevant features. The second pass receives a representation where those features are already highlighted and can sharpen further. Each recursion would push the system closer to the pattern-specific regime.

There is direct empirical evidence for this. In experiments with Hopfield-style attention, the optimal value of beta turned out to be 2.0 — double the standard transformer scaling of one over the square root of the key dimension. The standard scaling is too cold. The attention distribution is too diffuse, spreading probability mass across too many patterns when it should be committing more decisively to the most relevant ones.

This connects directly to recursive refinement. Two passes at the standard beta empirically produce a comparable reduction in attention entropy to one pass at beta equals 2.0. Each recursion effectively doubles the signal from the salient features, because those features appear in the input twice: once from the raw signal and once from the refined output of the previous pass. The noise does not double because it was not preserved through the refinement step. Recursion and higher beta are two different paths to the same destination — getting the attention distribution past the critical sharpness threshold where the network commits to the relevant pattern rather than averaging over candidates.

This interpretation is consistent with the empirical observations: rapid improvement in the first few passes (crossing the effective phase transition), followed by diminishing returns (settling into an attractor basin), and resistance to overfitting (convergence toward learned structure rather than memorized noise).

A Toy Demonstration

These dynamics can be observed directly in a pure Hopfield network simulation. Store three patterns in 100 dimensions — a cross, a square outline, and a diagonal — then present the network with a noisy version of the cross and observe retrieval at different temperatures.

At low beta (0.1), the attention weights distribute broadly across all three patterns. The output is a ghostly overlay of all stored memories. The network is too cold to commit, so it returns a fuzzy mixture. This is the averaging regime that Koulischer describes below the critical temperature.

At standard beta (1.0), the weights shift decisively toward the correct pattern (0.999 probability on the cross). The output is clean. In a standard transformer, this is the single one-shot retrieval step.

At high beta (5.0 and above), the network commits with effectively 100 percent probability to the nearest stored memory. The noise is gone. The pattern is retrieved perfectly.

The critical observation is that recursive refinement mimics the effect of higher beta. Even with beta fixed at 1.0, feeding the output of one pass back as the input to the next pass squashes the attention distribution further, pushing the system into the sharp retrieval regime without changing the temperature parameter. Each pass makes the weights more peaked, the commitment more decisive.

This toy demonstration uses well-separated patterns in high-dimensional space, where even a single pass at standard beta achieves near-perfect retrieval. The real test comes when patterns are harder to distinguish — similar patterns, overlapping clusters, ambiguous inputs. That is the metastable regime where multiple patterns compete and a single pass cannot resolve the ambiguity. It is also much closer to the reality of natural language, where token representations are clustered, overlapping, and context-dependent. In that regime, the standard scaling leaves the network unable to commit on a single pass, which is why beta equals 2.0 helps and why recursion helps. They push the system through the phase transition on ambiguous inputs where one-shot retrieval fails.

And this is where the engram earns its place. In the simulation, if you add a bottleneck between passes, the noisy activations that do not correspond to any stored pattern are discarded by the compression. When the cleaner signal loops back in, the signal-to-noise ratio improves faster than simply increasing beta, because the bottleneck is selectively removing noise rather than indiscriminately sharpening everything.

Where the Analogy Breaks Down

It is important to be precise about the limits of this interpretation. A full transformer forward pass is not a pure Hopfield update. The attention mechanism is the Hopfield component, but a transformer also includes value mixing, residual connections, MLP nonlinearities, layer normalization, positional encoding, and feedforward expansions. These additional components break the formal energy function guarantees that make Hopfield theory rigorous.

In a pure Hopfield network, there is a conserved energy function that decreases monotonically with each update step. This guarantees convergence to fixed points and provides formal attractor dynamics. In a recursive transformer pass, there is no such guarantee. The MLPs can increase energy. The layer normalization renormalizes activations, which means the “norms increase with each recursion” argument does not hold in its simple form. The residual connections add the input back to the output, modifying the trajectory in ways that don’t correspond to pure energy minimization.

So the claim is not that recursive refinement is mathematically equivalent to iterated Hopfield dynamics. The claim is that the attention mechanism, which is the primary associative retrieval mechanism of the transformer, performs Hopfield-style pattern retrieval, and that recursive refinement gives this mechanism multiple opportunities to sharpen its retrieval. The other components of the transformer modify the convergence dynamics in ways that may help or hinder, and characterizing those effects is an empirical question.

What Definitely Contributes

Even setting aside the Hopfield interpretation, there are well-understood mechanisms that explain much of recursive refinement’s behavior.

Weight sharing across iterations is a powerful implicit regularizer. When the same parameters are used for every recursion, the model gets deeper effective computation without additional parameters. This is equivalent to a very deep network with tied weights, which is known to regularize well.

Iterative denoising is a general principle that applies regardless of the energy landscape interpretation. Each pass through a learned function removes inconsistent features from the representation. This is the same principle that makes diffusion models work: repeated application of a denoising step progressively sharpens the output.

Entropy reduction through repeated softmax operations produces sharper attention distributions even without any energy landscape argument. Applying softmax to an already-softmaxed distribution concentrates probability mass further. This alone produces the appearance of a temperature drop.

Fixed-point convergence is a standard dynamical systems result. If the network approximates a contraction mapping near good solutions, repeated application converges to a fixed point. No energy landscape is required for this.

These mechanisms are real and sufficient to explain convergence behavior. The question is whether the Hopfield interpretation adds anything beyond what these simpler explanations provide.

What the Hopfield Lens Adds

I believe the Hopfield interpretation is more than just a relabeling of iterative denoising for one specific reason: it provides a geometric theory of what the fixed points are and why the model converges to them.

The denoising explanation says the model removes noise with each pass. But it doesn’t say what counts as signal versus noise, or why the model converges to useful representations rather than arbitrary fixed points. The Hopfield interpretation says that the fixed points are implicit stored patterns encoded in the weights, that these patterns correspond to the structures the model extracted during training, and that convergence toward them is convergence toward learned knowledge rather than arbitrary states.

The phase transition result from Koulischer adds a prediction that pure denoising doesn’t make: there should be a critical number of recursions where behavior changes qualitatively, corresponding to the effective temperature crossing the critical threshold. Below this threshold, the model averages. Above it, the model commits. This is testable.

The scale dependence also gets a specific explanation. Small models have poorly separated stored patterns (low effective separation angle), so their beta-effective starts low and recursion doesn’t push it past the critical threshold. Large models have well-separated patterns, start closer to the threshold, and cross it with fewer recursions. This predicts a specific scaling relationship between model size and optimal recursion depth.

Connection to Dual-Objective Training

Recent experiments combining next-token prediction with sentiment classification as a dual training objective showed that the combined loss acts as a regularizer, improving generalization beyond what either objective achieves alone. At the seven-billion parameter scale, the dual-objective model reached 96.79 percent accuracy on sentiment classification versus 96.22 percent for the single-objective baseline.

In the Hopfield framework, dual-objective training shapes which patterns are stored in the energy landscape. A model trained only on next-token prediction stores patterns optimized for predicting the next word. Adding a sentiment objective creates additional attractor basins corresponding to evaluative states. The energy landscape now has structure along both linguistic and evaluative dimensions.

If recursive refinement were applied to a dual-objective model, each recursion would sharpen both linguistic and evaluative patterns. The model would refine not just what the text says but how the text feels, through the same iterative retrieval mechanism. This points toward an architecture where iterative self-evaluation emerges from recursive refinement through a network trained with competing objectives — a kind of computational appraisal process that deepens with each pass.

Whether this is best understood as “iterated Hopfield retrieval in a landscape shaped by dual objectives” or simply “iterative denoising in a representation space with evaluative structure” is, at some level, a choice of vocabulary for the same underlying dynamics. But the Hopfield vocabulary provides specific predictions about phase transitions, scaling, and the geometry of the attractor landscape that the denoising vocabulary does not.

Amplification Versus Filtering: The Engram Hypothesis

There is a further distinction worth drawing. Raising beta and recursive refinement both sharpen the attention distribution, but they do so differently. Higher beta amplifies everything indiscriminately — both signal and noise get sharper attention weights. Recursion through a bottleneck does something more selective.

In the engram architecture, the output of a forward pass is compressed through a low-dimensional bottleneck before being fed back as input to the next pass. Whatever survives that compression is by definition the most salient structure in the representation. The noise is discarded not by being attenuated but by being geometrically excluded from the bottleneck’s representational capacity. When this compressed representation is combined with the raw input for the next pass, the model receives the original signal plus a filtered version where only the important features remain. The signal is doubled. The noise is not.

This is strictly better than raising beta, which sharpens the attention distribution over all features equally. Higher beta makes the network commit harder to whatever it attends to, whether that’s signal or noise. The engram makes the network commit harder specifically to what survived compression, which is the learned structure.

The predicted experimental signature is precise. On tasks involving independent inputs like single-sentence sentiment classification, the engram should provide only a marginal improvement over optimal beta, perhaps around one percent. This is because most of the work is done by getting into the right sharpness regime, and the engram’s filtering adds only a small refinement once you’re already there. But on tasks with temporal depth — document-level classification, legal case outcome prediction, any task where the model must accumulate evaluative state across many chunks of input — the engram becomes the dominant mechanism rather than the marginal one. Beta sharpens each individual retrieval. The engram carries forward what mattered from previous retrievals. They solve different problems that only diverge when there is sequential structure to exploit.

This leads to a clean factorial experiment. Test four conditions: standard beta with no engram, optimal beta (2.0) with no engram, standard beta with engram, and optimal beta with engram. If the engram at standard beta matches or exceeds optimal beta without engram, the filtering mechanism is doing the same work as amplification. If optimal beta plus engram provides only a marginal improvement over optimal beta alone, the two mechanisms are largely redundant for independent inputs, confirming that the engram’s real payoff is on sequential tasks where it carries context forward rather than merely sharpening a single retrieval.

The RAG Asymmetry: Strongest Evidence for Weight-Stored Attractors

There is one empirical finding that does more to distinguish between competing explanations than any of the theoretical arguments above. The engram improves performance only on tasks relying on knowledge internalized during training, and provides no measurable benefit for retrieval-augmented generation. This dissociation is extremely diagnostic.

If recursive refinement were simply adding computational depth, it would help RAG as much as learned tasks. More depth is more depth regardless of where the information comes from. If it were generic denoising, it would smooth noise in retrieved context just as well as noise in learned representations. If it were just extra memory capacity, RAG performance would improve because RAG is literally a context bottleneck problem.

None of these alternative explanations predict the asymmetry. Only one mechanism does: associative retrieval over patterns stored in the model weights.

Knowledge internalized during training forms clusters in representation space. Those clusters carve attractor basins. Recursive refinement pulls the representation downhill into those basins, and the engram helps by filtering the signal toward the basin floor. But RAG context was never encoded in the weights. There are no attractor basins for unseen, externally retrieved text. The energy landscape is flat in those directions. Recursion has nowhere to converge to, so it provides no benefit.

This behavior matches the qualitative predictions of associative memory models such as modern Hopfield networks. The network retrieves stored patterns. It cannot improve retrieval of patterns that were never stored. If a pattern was not memorized into the weight matrix during training, there is no basin of attraction, and the dynamics produce only averaging over arbitrary directions.

The selectivity of the improvement — learned knowledge benefits, external context does not — is precisely what an attractor memory would produce, and precisely what generic denoising, depth, or capacity explanations would not. This negative result on RAG is stronger evidence for the Hopfield interpretation than any of the positive results on learned tasks, because it discriminates between mechanisms rather than merely being consistent with one of them.

Proposed Tests

The difference between “approximate Hopfield dynamics” and “generic iterative denoising” is empirical, and several experiments could distinguish between them.

First, measure attention entropy across recursive passes. If the Hopfield interpretation holds, attention distributions should become monotonically sharper with each recursion — entropy should decrease at each step, corresponding to the system moving toward a pattern-specific fixed point. If entropy sometimes increases between passes, the Hopfield interpretation is weakened and the dynamics are better described as generic fixed-point iteration.

Second, track representation norms across recursions, measured after layer normalization. The simple version of the effective temperature argument predicts increasing norms, but layer normalization should prevent this. If the Hopfield-like sharpening still occurs despite stable norms, then the mechanism operates through attention weight concentration rather than norm amplification, and the beta-effective formula needs modification.

Third, measure the cosine similarity between representations at successive recursion depths. If the system is converging to an attractor, successive representations should become more similar to each other (the trajectory slows down as it approaches the fixed point). Plotting the rate of convergence across recursions would reveal whether the dynamics resemble attractor convergence or something else.

Fourth, the factorial experiment described above: standard beta versus optimal beta (2.0) crossed with engram absent versus present. This directly tests whether the engram’s filtering mechanism is redundant with or complementary to temperature-based amplification. The interaction term in this two-by-two design is the key result. If beta and engram are redundant, the interaction is negative (combining them adds less than either alone). If they’re complementary, the interaction is positive. Run this first on independent sentences where we predict near-redundancy, then on sequential document-level tasks where we predict complementarity.

Fifth, examine whether recursive refinement benefits scale with model size in the specific way the Hopfield interpretation predicts: larger models should need fewer recursions to achieve the same benefit, because their stored patterns are better separated and beta-effective starts closer to the critical threshold.

What This Means

The relationship between recursive refinement and Hopfield dynamics is not a strict mathematical equivalence. A transformer is more than an attention mechanism, and the additional components break the formal energy guarantees that make Hopfield theory rigorous. But the attention mechanism is the core retrieval operation, it is genuinely a Hopfield update, and giving it multiple iterations through recursive passes approximates the kind of iterated retrieval that Hopfield networks perform when converging to stored patterns.

This interpretation is valuable not because it proves something new but because it generates specific, testable predictions about phase transitions, scaling relationships, and the geometry of learned representations. It connects an empirical finding (recursive refinement works surprisingly well) to a theoretical framework (energy-based associative memory) that has been developed over decades and is well understood mathematically.

The simpler explanation, that recursive refinement is just iterative denoising with weight sharing, is probably sufficient for engineering purposes. But the Hopfield lens suggests that what’s being denoised toward is not arbitrary. It’s the set of patterns stored in the learned weights, organized into an energy landscape with specific geometric properties. Understanding that landscape, its phase transitions, its attractor basins, and how training objectives shape it, could inform the design of architectures that exploit recursive refinement more efficiently.

The stored patterns are the knowledge. The recursion provides additional opportunities for the attention mechanism to perform associative retrieval. Higher beta is committing harder. The engram is committing to what matters. And the theory predicts that there’s a critical depth of recursion where the representation snaps from uncertain averaging to confident commitment — a phase transition in thought. What the engram adds is not just sharper commitment but wiser commitment, filtered through a bottleneck that preserves the structure worth committing to.

References

Ramsauer, H., Schafl, B., Lehner, J., et al. (2020). “Hopfield Networks is All You Need.” International Conference on Learning Representations, 2021.

Koulischer, F., Goemaere, C., Van Der Meersch, T., Deleu, J., Demeester, T. (2023). “Exploring the Temperature-Dependent Phase Transition in Modern Hopfield Networks.” NeurIPS 2023 Workshop on Associative Memory and Hopfield Networks.

Vaswani, A., Shazeer, N., Parmar, N., et al. (2017). “Attention is All You Need.” Advances in Neural Information Processing Systems 30.

Krotov, D. and Hopfield, J. (2016). “Dense Associative Memory for Pattern Recognition.” Advances in Neural Information Processing Systems 29.

Bonsignore, M. (2024–2025). Tiny Recursive Model experiments. GitHub: MikeyBeez/trm.