Mamba 3 Was Built Backwards — And That’s Why It’s Fast

Part 7 of the Mamba series. Previously: Mamba 3 restored state tracking via data-dependent RoPE. This article is about the other two contributions — and the single obsession that drove the entire architecture.

Photo by Jahanzeb Ahsan on Unsplash

Most language model architectures are designed training-first. You figure out how to train them efficiently, and inference is whatever it ends up being.

Mamba 3 was designed the opposite way. Inference-first. Every decision, starting from the mathematical foundations, was made with one question in mind: when this model is deployed and generating tokens one by one, what’s slowing it down?

This inversion matters more than it might sound. Training runs once per model. Inference runs billions of times per deployed model. The economics of a model — what it costs to run, what applications it enables — depend more on inference efficiency than anything else.

And inference efficiency is about one metric nobody outside hardware engineering talks about. A number called arithmetic intensity. If you don’t understand arithmetic intensity, modern model architectures don’t make sense. If you do, almost all of the recent design choices — from MLA to MoE to MIMO — suddenly do.

This article is about arithmetic intensity, and about the two Mamba 3 contributions built around it: trapezoidal discretization and MIMO.

The Human Version: Why a Fast Worker Can Still Be Slow

Imagine a worker in a warehouse. Their job is to pull items off shelves and pack them into boxes.

The worker themselves is fast. They can pack items instantly once the items are in front of them. But the warehouse is large, the shelves are distant, and most of the worker’s time is spent walking to the shelf rather than actually packing.

Now imagine you hire a faster worker. They pack twice as fast. Does total throughput double?

No. Because the bottleneck was never the packing. It was the walking. Upgrading the packing speed doesn’t help if most of the time is spent moving between shelves and the packing station.

This is exactly what happens with modern GPUs running sequence models. The GPU’s compute units are extraordinarily fast. They can do trillions of floating-point operations per second. But most of the time, they’re idle — waiting for data to arrive from memory.

The ratio of computation to memory traffic is called arithmetic intensity. If a model’s operations have low arithmetic intensity, the GPU’s expensive compute units sit idle while memory bandwidth becomes the bottleneck. You can buy a faster GPU, but it won’t help. The bottleneck isn’t compute. It’s the walking.

Why SSMs Have Low Arithmetic Intensity During Inference

During training, batched SSMs can achieve high arithmetic intensity. Many sequences process in parallel, compute units stay busy, life is good.

During inference — when the model is generating tokens one at a time — arithmetic intensity collapses.

Here’s why. At inference, each forward step does roughly:

1. Load the state (h_{t-1}) from memory.
2. Load the parameters (A, B, C, Δ) from memory.
3. Do a small amount of arithmetic to compute the new state and output.
4. Write the new state back to memory.

Steps 1, 2, and 4 are memory operations. Step 3 is compute. The ratio is terrible. The GPU spends most of its time moving data around, not computing anything.

This is the hidden inefficiency of SSMs that Mamba 3 was built to fix. Theoretically, SSMs are O(n). Practically, they run far below peak GPU throughput because the per-token work doesn’t have enough compute relative to the memory movement.

You can’t fix this by making the state bigger — that just moves more data, making the problem worse. You need to restructure the computation so that each trip to memory returns more work.

Enter MIMO.

MIMO: Multi-Input, Multi-Output

The name sounds complicated. The idea is simple.

A standard SSM processes one input channel and produces one output channel per state. One stream of information in, one stream out. SISO — single-input, single-output.

MIMO generalizes this: multiple input channels, multiple output channels, all sharing the same state.

Imagine the warehouse worker again. Instead of walking to the shelf, picking up one item, and walking back — what if they picked up four items per trip? The walking cost is the same. The packing throughput quadruples. Arithmetic intensity goes up.

That’s MIMO. The state is loaded once. The parameters are loaded once. But during that single trip to memory, the compute units process four input streams and produce four output streams simultaneously.

Concretely, in Mamba 3:

• SISO version: one trip to memory per state = one token of work
• MIMO version (rank 4): one trip to memory per state = four tokens of work

The compute cost goes up by 4x. But the memory cost stays the same. On a memory-bound workload — which decoding SSMs are — this is almost free. You’re using compute that was sitting idle anyway.

Why MIMO Doesn’t Just Help Speed

The surprising thing about MIMO isn’t that it speeds up inference. It’s that it also improves model quality.

At first glance, this seems wrong. If MIMO and SISO have the same parameter count, why would MIMO be better?

The answer is about how the parameters are used. In SISO, each parameter is tied to one specific input-output channel. In MIMO, parameters are shared across multiple input-output pairs. This sharing acts as an implicit regularization — it forces the model to find parameters that work for multiple contexts simultaneously.

The result, in the paper’s experiments: at the same parameter count, MIMO variants of Mamba 3 achieve noticeably better perplexity than SISO variants. At the 1.5B scale, the MIMO variant adds 1.2 percentage points of downstream accuracy on top of the base Mamba 3 — itself already 0.6 points ahead of the next-best linear model.

So MIMO is doing two things simultaneously: filling idle compute at inference time, and improving model quality through parameter sharing. Both from the same architectural change.

The Other Improvement: Trapezoidal Discretization

The second improvement is quieter but mathematically interesting. It’s about how to convert continuous-time dynamics into discrete steps.

Recall from earlier articles: SSMs are continuous-time systems at heart. To use them for language, you discretize. Mamba 1 and Mamba 2 both used a method called Euler discretization — the simplest way to discretize a continuous system.

Here’s what Euler assumes: between one token and the next, the input is constant. Imagine a car whose GPS updates once per second. Euler’s method assumes the car is going at the starting speed for the entire second.

This is obviously wrong. The car is accelerating, decelerating, turning. But it’s a first-order approximation — good enough for small timesteps, bad when steps are larger.

Mamba 3 uses trapezoidal discretization instead. Same car, same GPS updates — but now the method averages the speed at the beginning and the end of the interval. It’s a second-order approximation. Closer to the truth, at the cost of slightly more arithmetic.

In SSM terms: trapezoidal discretization makes the state update depend not just on the current input, but on a weighted combination of the current input and the previous input. It’s as if the model runs a small convolution inside its state update, capturing a bit more local context before committing to a new state.

The practical effect: better quality at the same parameter count. Empirically, trapezoidal discretization accounts for a meaningful chunk of Mamba 3’s quality gains over Mamba 2.

And it costs almost nothing. The extra arithmetic fits inside the same memory bandwidth. Arithmetic intensity goes up. The GPU’s idle compute units get one more thing to do with the data they’ve already loaded.

Putting It All Together

The three Mamba 3 improvements aren’t independent additions bolted onto Mamba 2. They’re three answers to a single question: given that inference is memory-bound, what’s the maximum quality we can extract per trip to memory?

• Trapezoidal discretization: Do more math per memory trip → better quality at same memory cost.
• Data-dependent RoPE: Restore state-tracking expressivity → new capabilities without extra memory cost.
• MIMO: Process multiple streams per memory trip → better throughput and better quality simultaneously.

Every improvement increases the work done per byte of memory traffic. That’s what “inference-first design” looks like in practice. It’s not about making the model smaller or simpler. It’s about making every memory access pay for more.

The result: Mamba 3 achieves comparable perplexity to Mamba 2 with half the state size. When matched on state size, it beats Mamba 2 significantly. At the 1.5B scale, the full Mamba 3 MIMO variant delivers a 1.8 percentage point accuracy improvement over Gated DeltaNet, the next-best linear model.

These numbers sound small. They aren’t. At frontier scale, half a point of downstream accuracy takes months of compute and billions of dollars of investment. 1.8 points from a single architectural generation is a big jump.

What Mamba 3 Still Can’t Do

No architecture is universal. Mamba 3 has real limitations worth naming.

Retrieval from unstructured text. This is the hardest remaining gap. When a model needs to answer a question that requires looking up specific information — a name, a date, a fact buried in the middle of a long document — Transformers have a structural advantage. The KV cache retains every token verbatim. Mamba 3’s compressed state, no matter how expressive, loses some detail.

On “needle-in-a-haystack” tasks — which are highly structured — Mamba 3 performs well. On realistic retrieval tasks with messy, unstructured context, Transformers still win.

Very long context reasoning. Mamba 3 handles long sequences efficiently, but reasoning across long contexts — where the model needs to link information at positions far apart — remains harder for a compressed state than for explicit retrieval.

Hybrid architectures dominate. The best current models don’t choose between Mamba and attention. They use both. The standard ratio that works empirically: roughly one attention layer for every four to five Mamba layers. This is now the default in frontier hybrid models. Mamba 3 isn’t a replacement for attention. It’s the linear sequence mixer that makes hybrid architectures efficient.

The One Thing Most Explainers Miss

Here’s what gets lost in the three-improvements framing.

Mamba 3 drops something that had been in every previous Mamba version: the short convolution layer. The Mamba 1 and Mamba 2 block had a small 1D convolution before the SSM, applied across tokens. It was considered essential — it helped the model handle short-range dependencies that the state-based mechanism struggled with.

Mamba 3 removes it. The paper hypothesizes — without fully proving — that trapezoidal discretization plus the data-dependent BC bias now perform an equivalent function. The short convolution was a workaround. Trapezoidal discretization makes it unnecessary.

This is a quiet sign of architectural maturity. When an architecture stops needing auxiliary components, it usually means the core mechanism has gotten strong enough to stand on its own. The 1D conv in Mamba 1/2 was a training wheel. Mamba 3 takes it off.

The One Idea to Hold

Mamba 3 is not a redesign. It’s a refinement — three surgical improvements aimed at a single metric most model papers don’t discuss.

The field has been racing to make models bigger, to train on more data, to use more sophisticated attention variants. Mamba 3 asks a different question: given that most of a deployed model’s life is spent generating tokens one at a time, how do we make that inference step do more work per memory access?

The three improvements are the answer. Together they push SSMs past a limit that Mamba 2 couldn’t cross — delivering the quality of a larger model with the inference cost of a smaller one.

But understanding the individual architectures isn’t the endpoint of this series. The deeper question is which architecture to use for what — and whether the Transformer vs Mamba framing is even the right way to think about it.

Next: Transformers and Mamba, side by side. Where each one wins, where neither does, and why the future of sequence modeling is almost certainly not one architecture but two — working together.