Growing Logic: Why Training Neural Networks Is Becoming a Materials Science

For years, the mechanistic interpretability community has watched the phenomenon of “grokking” with a mixture of awe and frustration. We have treated this sudden leap — from memorizing training data to generalizing underlying logic — as a ghost in the machine, a passive event we wait for with crossed fingers. But as we move from the era of discovery to the era of reliable engineering, we must stop simply observing these transitions. We must start engineering them.

Recent research into inducing specific algorithmic structures, such as Strassen matrix multiplication, suggests a fundamental shift in perspective. We should treat neural networks not as abstract mathematical entities, but as high-precision “materials” that we are attempting to “crystallize” into specific logical structures. In this view, training is less about optimization in a vacuum and more about trajectory engineering — steering a model into a discrete, rigid attractor.

Algorithms as Material Properties

This shift from “theory” to “materials engineering” is a radical departure from traditional AI research. In standard computer science, an algorithm is a sequence of instructions; here, an algorithm is a material property induced through a precise “recipe.”

The analogy is drawn directly from semiconductor manufacturing: we do not need to resolve the complexities of quantum mechanics to produce high-quality silicon; we simply need a reliable, documented process for growing crystals. As the research abstract notes:

“I now frame this work as materials engineering. We are not discovering fundamental laws. We are developing recipes for producing specific material properties in neural networks. The analogy is semiconductor manufacturing: we do not fully understand electron mobility at the quantum level, but we know exactly how to grow silicon crystals for chips.”

Central to this “crystallization” is the breaking of Gauge Symmetry. In their continuous state, bilinear models possess symmetries where coefficients can be rescaled (α,β) without changing the output. However, forcing weights to a discrete lattice of {−1,0,1} breaks these symmetries. This transition from a fluid, symmetric state to a rigid, discrete structure is the essence of algorithmic crystallization.

The Batch Size Enigma: Beyond Hardware Limits

One of the most robust findings in this new materials science is the existence of a “Goldilocks zone” for batch sizes. The research identifies a critical range between 24 and 128 that is essential for inducing the Strassen structure.

Initially, it was hypothesized that this range was a byproduct of hardware — specifically, that batches in this range fit within the CPU’s L3 cache, minimizing memory thrashing. However, a rigorous memory analysis definitively debunked this: even a batch size of 1024 requires only 321.1 KB of memory, fitting comfortably within a standard 1MB L3 cache.

The batch size effect is not a hardware constraint; it is a property of the training dynamics. While still theoretically unexplained, it appears to relate to gradient noise geometry. In the [24, 128] range, the noise facilitates a “coherent regime” that allows the model to land in a discrete attractor, whereas larger batches lack the necessary stochasticity to escape generalizing local minima.

The 0.001 Fragility: Narrow Basins of Truth

Extreme Sensitivity to Noise The logic we seek to induce exists in a state of extreme fragility. Data from the source shows that adding Gaussian noise with a magnitude of only 0.001 to the weights causes a 100% failure rate in discretization.

The Narrow Basin Problem This suggests that algorithmic solutions inhabit incredibly narrow basins of attraction in weight space. If a training trajectory drifts by even a fraction, the model might still generalize (32% of runs reached generalizing local minima), but it will never “crystallize” into the rigid, verifiable form required for true algorithmic transfer. This fragility explains why reproducibility is so difficult in AI: we aren’t just looking for a good solution; we are aiming for a microscopic target in a high-dimensional landscape.

The Intellectual Honesty of a “Failed” Theory

A hallmark of the “craft” of engineering is the willingness to abandon a beautiful theory for an ugly fact. The researcher originally proposed that the condition number κ of the gradient covariance could prospectively predict success.

The “Post-hoc κ Analysis” revealed a more humble reality:

• A Diagnostic Signature, Not a Driver: While successful, discretized models consistently show κ≈1, this is a result of successful crystallization, not the cause of it.

• Predictive Failure: Attempts to use κ as a prospective predictor achieved only 58.3% accuracy — barely better than a coin flip.

• The Robustness of the Protocol: Despite the theory’s failure to predict, the recipe remained valid. Across 195 training runs, the specific protocol — 1000+ epochs, weight decay ≥1e−4, and the [24, 128] batch range — delivered a 68% success rate in inducing the target structure.

Active Construction vs. Passive Emergence

Traditional grokking research often adopts a passive stance: we train and watch for generalization to emerge. This work advocates for active construction. The researcher does not wait for Strassen multiplication to appear; they force its emergence through a two-phase protocol that begins with a rank-8 model and uses importance scores to prune it down to a rank-7 target.

By hardcoding this inductive bias, we are not “cheating.” We are performing a necessary engineering intervention to steer the model toward a specific discrete attractor. As the research concludes:

“Algorithmic structure does not passively emerge from optimization. It is actively constructed through precise manipulation of training dynamics.”

Zero-Shot Expansion: The Verification Payoff

The ultimate validation of this engineering approach is the stability of the transfer. Because the 2x2 Strassen structure was successfully “crystallized” into a rigid lattice, it can be applied to 64x64 matrices without any additional training. This is achieved through recursive block application, a mathematical property of the induced structure that allows the same logic to scale infinitely.

*Note: The slight growth in error at 64x64 is due to numerical accumulation inherent in recursive floating-point operations.

Toward a New Engineering Discipline

We are entering an era where we must move from “watching AI learn” to “engineering AI to crystallize.” This work demonstrates that while the underlying mathematical drivers (like the batch size effect) may remain partially obscured, we can still develop reliable “recipes” for inducing logical invariance.

However, the extreme fragility of this process serves as a warning. If a mathematically elegant algorithm like Strassen multiplication requires such precise trajectory engineering to emerge, what are the chances we are “accidentally” reaching correct logic in LLMs? We may find that many of our current systems are merely settling for generalizing local minima — sophisticated mimics of logic — rather than achieving the true algorithmic invariance required for reliable reasoning. The future of AI safety and interpretability may well depend on our ability to find more such recipes for logic.

Github: https://github.com/grisuno/strass_strassen

DOI: https://zenodo.org/records/18295001