Subquadratic Raised $29M for Linear Attention. Here’s What I Built in Two Hours

The title is misleading. Let me set the record straight before going further.

Subquadratic Inc. launched their SubQ model yesterday with a claimed 12 million token context window and 1000x reduction in attention compute at extreme context lengths. Their announcement materials say they’ve achieved O(N) linear scaling. Their architecture documents describe the mechanism as content-based selection followed by exact attention over selected positions, with explicit critiques of state-space models like Mamba for their inability to retrieve from arbitrary positions.

Reading their materials carefully, what they describe sounds closer to Mamba than to standard sparse attention. To achieve true O(N) scaling at frontier scale, they need something that processes the sequence in linear time. Pure content-based selection where every query is compared against every key is itself quadratic. So whatever makes their selection sub-quadratic must involve some mechanism analogous to state-space compression, hierarchical access, or learned hashing. Their MRCR v2 score of 65.9 third-party verified versus Opus 4.6’s 78.3 is consistent with this. A twelve point gap on the hardest retrieval benchmark suggests the architecture has tradeoffs typical of approaches that compress the activation space gradient descent operates on.

What I built in two hours of experimental time is not what they built. The architecture I’ll describe is still O(N²) in the original sequence length. Specifically it’s O((N/16)²) for attention plus O(N) for a compression stack that runs before attention. The asymptotic scaling is quadratic with the constant factor divided by 256, not linear. At realistic context lengths this provides substantial speedup, 9.2x at 4096 tokens with the speedup growing larger at longer contexts, but it doesn’t compete with truly linear scaling at extreme context lengths.

So this isn’t a reverse engineering of SubQ. It’s an investigation of one specific point in the architectural space their announcement made visible. The point I investigated has different properties than what they appear to have built, and the empirical results suggest it’s worth knowing about for deployment contexts where the simpler architecture and quality preservation matter more than asymptotic linearity.

The Architecture

The setup is straightforward. Take a transformer. Insert a compression stack between the input embeddings and the first attention layer. The compression stack reduces the sequence to one sixteenth its original length through learned mixing. The attention layers and everything else operate on the compressed sequence. The output projection at the end is standard.

The compression stack has four layers. Each layer is a strided 1D convolution with kernel size 2 and stride 2, followed by layer normalization and a GELU activation. Each layer reduces the sequence length by half. Four layers in sequence reduce the sequence to one sixteenth. The convolutions are causal so the architecture works for autoregressive generation.

The convolution kernels are learned during training. Gradient descent shapes the mixing weights to preserve information that matters for the language modeling task. Each output position of a compression layer is a learned combination of its two input positions. The combination is whatever the optimization found useful, not a fixed pooling operation.

The depth matters. A single layer of strided convolution is essentially a learned linear pooling. Each output is a weighted combination of inputs, where the weights are trained but the operation is otherwise simple. Multiple layers with nonlinearities between them is a different kind of operation. The stack can compute features of features, not just weighted sums. The first layer learns local features of small windows. The second layer composes these into larger features. By the fourth layer, each compressed position represents a learned function of the sixteen original positions it covers, where the function is whatever gradient descent found useful for the downstream task.

This is the same property that makes deep CNNs more powerful than shallow ones for image processing. Shallow CNNs detect edges. Deep CNNs detect objects. Without depth you can only compute features of fixed complexity. The four-layer stack lets the compression learn rich feature compositions over each window of sixteen original positions.

After compression, standard attention operates on the compressed sequence. If the input was N tokens, attention computes over N/16 positions. The cost of attention is O((N/16)²) which is O(N²/256). Substantially cheaper than O(N²) on the original sequence but not asymptotically different.

Why This Should Work

The conceptual argument starts with what attention is doing during training. The N×N attention matrix is the activation surface gradient descent operates on. Activations are transient, parameters are permanent, and gradient descent shapes the parameters by pushing through the activations they produce. The N² activation space is what the optimization has to work with. The size of that space matters for how much structure gradient descent can find.

Approaches that reduce attention’s cost can be sorted by what they do to this activation space.

Sequential state compression like Mamba processes the sequence position by position with bounded state. Each step decides what to keep in state and what to drop. The decision is made online based on what’s available now, without knowing what queries will need later. When the guess is wrong, information is gone. The activation space at any position is the current state, which has bounded capacity. Linear scaling, but the optimization has access only to whatever survived sequential compression.

Selection-based approaches like SubQ’s apparent mechanism choose which positions to attend to per query. The selected subset is computed by some learned mechanism, attention runs over the selected positions, non-selected positions don’t participate in this query’s computation. Linear or sublinear scaling depending on how selection is implemented. The activation space is the selected subset per query, which can adapt to query content but is smaller than dense attention.

Non-sequential compression like the architecture I tested mixes all positions simultaneously through learned operations. Nothing is dropped based on online decisions. Every position contributes to some compressed representation. The activation space is the compressed sequence, which is smaller than the original but contains information from all positions in mixed form. Compressed quadratic scaling. The optimization works on a richer surface than sequential compression provides because nothing was discarded based on guesses about future need.

The conjecture going in was that non-sequential compression preserves enough of the optimization surface to maintain quality, while still providing substantial efficiency gains. The compression is lossy in that exact information at any single position is no longer cleanly accessible, but the information is still present in compressed form across the mixed representations. Downstream attention can pull together aspects of the compressed positions to construct the representations needed for the task.

This conjecture had partial empirical support from prior work. Earlier saliency-pool experiments matched baseline at 4x, 8x, and 16x compression on Tiny Shakespeare. The current experiment used a different compression mechanism (strided convolution rather than saliency-based pooling) and pushed to a different ratio. Whether the conjecture would hold for the specific architecture and ratio was empirically open.

Why 16x Instead of 12x

Subquadratic doesn’t disclose their compression ratio. They describe content-based selection and don’t reveal what fraction of positions get selected. The 12 in their 12 million token context isn’t a compression ratio; it’s the maximum context length their architecture supports.

I’d been planning to use 12x compression because Subquadratic likely ran a hyperparameter sweep before settling on whatever their final architecture is. Companies with their funding can sweep many configurations and pick the best. If they landed somewhere with 12 as a relevant number, that might be the optimal point for similar architectures.

The reason 12x didn’t end up being clean is structural. Strided convolution with stride 2 produces 2x compression per layer. Three layers of stride 2 give 8x compression. Four layers give 16x. Getting exactly 12x requires a non-integer compression at one layer, which is awkward to implement cleanly. The options were three layers of 2x for 8x total, four layers of 2x for 16x total, or some non-uniform mixture. Four layers of 2x was the cleanest implementation that pushed to substantial compression, so that’s what got built.

This is probably fine. The architectural question is whether learned hierarchical compression preserves quality, not whether the specific ratio is exactly 12 or 16. If 16x works, 12x would presumably also work, possibly slightly better due to less aggressive compression. If 16x fails, 12x might or might not succeed depending on whether the failure is at a sharp threshold or graceful degradation.

The comparison to Subquadratic’s claim is approximate either way. We don’t know their effective compression ratio. They claim 1000x compute reduction at 12 million tokens, which implies very high effective compression at that scale. At more moderate context lengths their effective compression is presumably lower. The 16x I tested is in the same neighborhood as what their architecture probably operates at moderate scales.

What the Experiments Actually Showed

I ran two experiments. The first was on Tiny Shakespeare. The second was on WikiText-103.

Tiny Shakespeare gave a 17% perplexity gap. The compressed model achieved validation perplexity of 137.31 versus baseline’s 117.05. That fell into the spec’s “investigate failure modes” zone, neither clean adoption nor clear rule-out. The architecture worked in the sense that it learned, but it didn’t match baseline.

The result was puzzling at first because prior saliency-pool experiments had matched baseline at 16x compression on Tiny Shakespeare. The strided-conv mechanism is different from saliency-pool but I’d expected the broad finding to replicate. The 17% gap suggested the specific mechanism might be losing something the saliency-pool variant preserved.

The diagnostic interpretation was that Tiny Shakespeare’s local-dependency-dominated structure was the wrong substrate for this architecture. The corpus is character-level text where most predictive signal comes from very local context. The previous few characters or words usually predict the next one. A compression that mixes 16 positions together might be specifically destroying this local signal because adjacent positions get blended into compressed representations that lose fine-grained local information.

If this interpretation was right, a corpus with stronger long-range dependencies should compress better. The long-range signal would be preserved through the global mixing while the local signal that’s degraded by mixing would matter less. WikiText-103 is the natural test because it has substantial long-range structure across articles, paragraphs, and arguments.

WikiText-103 gave the clean result. Multiple measures all pointed the same direction.

Validation perplexity: 421.64 baseline versus 424.16 compressed. A 0.6% gap on direct held-out next-token-distribution likelihood.

Coarsened-anchor accuracy: 15 out of 64 for both models. Both predict the token 16 positions ahead with identical hit rate across 64 anchored trials.

Top-5 distributions on held-out prompts: visually almost identical. Same top tokens at similar probabilities. One concrete case where the compressed model picked the right next token over baseline’s wrong one.

Per-prompt anchor hits across 8 prompts: roughly balanced, with no systematic dominance. Compressed wins one prompt, baseline wins one, the rest tie.

The cost side: 9.2x faster inference at context length 4096. 8x faster training (38 seconds versus 311 seconds). The training speedup is particularly interesting because it makes the architecture cheaper to develop, not just cheaper to deploy.

The 17% Tiny Shakespeare gap and 0.6% WikiText-103 gap, with the same architecture and the same compression ratio, told a clear story. The architecture’s performance depends on whether the corpus has compressible long-range structure. On corpora dominated by local patterns the compression destroys signal. On corpora with substantial long-range structure the compression preserves what matters.

This is an empirical finding worth stating clearly. Hierarchical learned compression at 16x preserves quality on corpora with long-range structure, while running 9.2x faster on long contexts. The gap on corpora without long-range structure characterizes what the architecture is good for and what it isn’t.

What’s Honest to Claim

The piece does not show that this architecture matches SubQ’s claims. SubQ claims linear scaling and a million-token context window. The architecture I tested is quadratic with a constant factor reduction and was tested at context lengths up to 4096. These are different scales of result.

The piece does not show that this architecture handles needle-in-a-haystack retrieval. Both models scored zero on the NIAH evaluation in this experiment. The training duration was 2000 steps, which is well short of the in-context learning regime that NIAH requires. A real NIAH comparison requires much longer training, which is a separate experimental track.

The piece does not show that the architecture extends naturally to long contexts beyond what was tested. The position encoding learned during training caps at 2048 tokens. Going beyond requires architectural extensions like RoPE or extensible position embeddings. The compression stack itself has no problem with longer contexts; the position encoding is the bottleneck.

What the piece does show is that learned hierarchical compression is one viable point in the architectural space SubQ’s launch made salient. The architecture is simple, preserves quality on appropriate corpora, provides substantial speedup, and is implementable on consumer hardware in two hours of experimental time given prior infrastructure.

The contribution is that this point exists and has the properties measured. Subquadratic occupies a different point with different properties. State-space models occupy yet another point. The architectural space is large enough that multiple viable approaches exist, and characterizing them empirically is more useful than picking a winner.

What This Means for the Field

Subquadratic raised twenty-nine million dollars at a five hundred million dollar valuation to commercialize their specific point in this space. The funding lets them operate at scale, push to extreme context lengths, hire many researchers, and build production infrastructure. This is genuine capability that independent researchers don’t have.

What the funding doesn’t buy is a moat in the architectural space itself. The space is large. Multiple viable points exist. Independent researchers with adequate infrastructure can investigate them. The two hours I spent testing one variant doesn’t replicate Subquadratic’s full system, but it does show that the architectural ideas are findable without their funding.

This matters because it suggests the value Subquadratic provides is in the productization rather than in the architectural innovation. The architecture is one specific path through territory where alternatives exist. The product is the working API at scale, the engineering for production deployment, the customer support and infrastructure. These are real and they’re what the funding pays for.

The architectural space being large is good news for the field. It means progress on long-context AI doesn’t depend on a single approach winning. Different approaches have different tradeoffs and serve different deployment contexts. Selection-based approaches like SubQ’s apparent mechanism handle retrieval-heavy tasks well. State-space models handle generation-heavy tasks well. Compression approaches like the one tested here might handle whole-document reasoning well, with the empirical question being which tasks the compression resolution is sufficient for.

For independent researchers, the implication is that this territory remains accessible. You don’t need a frontier lab to investigate architectural questions. Existing infrastructure plus careful experimental work plus tools like Claude Code make hours-scale investigations feasible. The findings won’t displace what frontier labs build, but they contribute to public understanding of what’s possible in ways that closed commercial systems can’t.

Conclusion

The clickbait title got you reading. The actual contribution is more modest than reverse engineering SubQ but more useful than nothing. Hierarchical learned compression at 16x preserves quality on WikiText-103 with substantial speedup. The architecture is simple, the empirical results are clear, and the limitations are honestly scoped.

What Subquadratic actually built remains undisclosed. Their announcement materials describe the mechanism at a level that gestures at content-based selection without revealing the implementation. Their MRCR v2 results suggest the architecture has tradeoffs typical of approaches that compress the optimization surface. The exact mechanism that achieves their claimed linear scaling is closed.

What I built is one open alternative in the same territory. Different mechanism, different tradeoffs, different scope of demonstrated capability. The piece reports what’s known and acknowledges what isn’t. The architecture is available for anyone who wants to implement it; the conceptual framework is available for anyone thinking about long-context architectures.

Two hours of experimental time on consumer hardware, building on years of prior architectural work, produced this result. The years of prior work matter. The specific two hours produced the specific architecture. Both are real. Neither is enough on its own to compete with what well-funded labs build at frontier scale, but both contribute to public understanding of what’s possible.

The architectural space has many viable points. Each one explored is one more piece of public knowledge about how long-context AI can work. The field benefits from the exploration regardless of which specific architectures end up in production systems. This piece reports one such exploration honestly.

Appendix: Compression Plus Mamba

After publishing the piece above, I ran a follow-up experiment that’s worth reporting because it changes the architectural picture.

The piece describes compression-plus-attention as still asymptotically quadratic. Compression reduces the sequence to one sixteenth its original length, but the attention that runs over the compressed sequence is quadratic in the compressed length. The architecture provides a 256x constant-factor reduction in attention compute, which is substantial, but the scaling is still O(N²) in the original sequence length.

The follow-up question is what happens if you replace attention with Mamba blocks after compression. Mamba is linear in sequence length. Compression-plus-Mamba would be linear in the original sequence length, since both components scale linearly. This would address the asymptotic scaling limitation of the compression-plus-attention approach.

The concern is that Mamba has known weaknesses on retrieval tasks. Its sequential state has bounded capacity, and information at distant positions may not survive state evolution to where it’s needed. Combining it with compression might compound these weaknesses if the two compressions stack destructively.

The empirical result is that they don’t stack destructively. They stack productively.

Compression-plus-Mamba achieves perplexity 422.75 on WikiText-103 versus compression-plus-attention’s 424.16 and baseline transformer’s 421.64. Both compressed architectures are within 0.6% of baseline. Compression-plus-Mamba slightly beats compression-plus-attention, though the difference is within noise. Coarsened-anchor accuracy is 14 out of 64 versus 15 out of 64 for the attention variants, also within noise. The architecture works at quality parity.

The likely reason this works: Mamba’s bounded-state limitation is mitigated when it operates on a compressed sequence rather than raw tokens. The state evolution has fewer steps to traverse, one sixteenth as many, so less information gets lost to sequential compression. Meanwhile, the hierarchical compression preserves information from all positions in mixed form, so what Mamba’s state needs to maintain has been concentrated by the compression rather than dispersed across the raw sequence. The two compressions handle different aspects of the long-context problem and combine cleanly.

The wall-clock speedup didn’t materialize in this experiment because of an implementation issue. The mamba-ssm package, which provides optimized CUDA kernels for selective scan, doesn’t build for the Blackwell architecture (compute capability 12.0) on the RTX 5070 Ti. The fallback was a pure-PyTorch implementation of selective scan. This is correct but uses a Python loop that dominates wall-clock time at small sequence lengths. Compression-plus-Mamba ran 9 to 30 times slower than compression-plus-attention at the contexts tested.

The slowdown is entirely an implementation artifact. The architectural O(N) property is preserved in the pure-PyTorch implementation; only the constant factor is bad because the Python loop overhead is large. With proper CUDA kernels, the constant would shrink by an order of magnitude or more, and the crossover where compression-plus-Mamba beats compression-plus-attention would land at practical context lengths. The architecture is sound; the empirical demonstration of the speedup waits for kernel support.

This changes how to think about the architectural picture. Compression-plus-attention provides a constant-factor improvement with quadratic asymptotic scaling. Compression-plus-Mamba provides linear scaling with quality preservation in principle, pending implementation support. Both are points in the architectural space. The first is what the piece above describes. The second extends the architecture toward what truly linear approaches achieve, while keeping the optimization advantages that come from the compression preserving information from all positions rather than discarding non-selected ones.

The path to demonstrating the speedup empirically is bounded. Either wait for the mamba-ssm authors to update their package for Blackwell, or write a Triton kernel for selective scan that works across GPU generations. The first might take weeks or months depending on their priorities. The second is a contained engineering project that someone with Triton experience could complete in hours. Once the kernel is available, the experiment can be re-run for clean timing numbers.

For now, what’s been demonstrated is that compression and Mamba combine without destroying quality. The two compressions stack productively rather than compounding each other’s losses. The architecture provides a path to genuinely linear scaling that preserves what gradient descent needs to optimize on. The implementation is currently constrained by hardware support, not by anything architectural.

The piece above stands as published. This appendix extends the empirical record with an additional finding that arrived after publication. The architectural space has been mapped a bit further. The work continues.