RNNs have something which Transformers don’t

It also explains why context length in LLMs is limited even in theory

The Unique Advantage of Recurrent Networks

Traditional Recurrent networks (RNNs/LSTMs) possess a fundamental architectural advantage over modern alternatives like transformers, convolutions, and even state space models like Mamba. This same advantage explains one of the key challenges in infinitely scaling context lengths in transformer-based LLMs.

The Information Flow Difference

The critical distinction lies in how temporal information (across text tokens) or spatial information (across visual data) flows through the network:

• In transformers, convolutions, and even Mamba: Information flow across positions requires multiple layers. A single layer’s output contains only an aggregation of attended information, but transformations over this aggregated information happen primarily in subsequent layers.
• In LSTMs and RNNs: A single layer itself can process temporal information sequentially, with internal states carrying and transforming information from earlier to later positions within that same layer.

This architectural difference means RNNs/LSTMs can theoretically model sequential dependencies more efficiently in terms of layer count, though this advantage comes at a significant cost. (With LSTMs can do it even with just one layer potentially, transformers will need atleast 2)

The Training Paradox

Unfortunately, this same characteristic has been the primary limitation for RNN-based systems — their sequential nature severely limits parallelization, dramatically increasing training costs compared to transformer and convolutional architectures.

Implications for Context Length Scaling

When developing LLMs with increasingly large context lengths, this fundamental constraint affects all architectures:

Attention systems like transformers should theoretically support unlimited context scaling with just a few layers. However, their effective perception is constrained by the vector dimensions of the value layer. Due to this information bottleneck, attention systems effectively function as convolutions with very large but still limited effective receptive fields

The effective “receptive field” is typically smaller than the full context length of the transformer. This is why increasing the layer count improves performance on tasks requiring long context understanding — each additional layer extends the model’s ability to process dependencies across distant positions, similar to how deeper convolutional networks capture larger spatial contexts.

Even state space models like Mamba, despite their efficient sequence modeling design, encounter similar multi-layer requirements to fully process temporal information, though they offer better computational scaling properties than standard attention.

And that’s precisely why there are even theoretical limitations (apart from practical limitations from compute) for extremely high context sizes in transformers (i.e a transformer model with a given layer count might look like it can work with infinite tokens, but that’s not the case)