Stop Hacking AI Memory: How SubQ-1.1-Small is Making “Whole-Artifact” Reasoning a Reality

The most valuable AI workloads today share an awkward shape. They don’t just want a tidy little excerpt; they want the whole thing. An entire codebase, a sprawling knowledge base, or months of interconnected legal documents.

For years, the AI industry has been duct-taping this problem. We’ve built elaborate retrieval (RAG) pipelines, aggressive chunking strategies, and complex agentic orchestration just to slice information into tiny fragments and stitch it painfully back together at inference time. As the newly released SubQ-1.1-Small Technical Report rightly points out, much of the modern AI stack is just a massive workaround for context scarcity.

The culprit? Dense attention. It scales quadratically, meaning reasoning directly over large artifacts gets explosively expensive right as the context window grows.

Enter Subquadratic. Founded by researchers from Meta, Google, Oxford, Cambridge, and BYU, they are attacking this constraint at its absolute root. Their new model, SubQ-1.1-Small, runs on Subquadratic Sparse Attention (SSA). Their thesis is wonderfully simple: Efficiency is intelligence.

The Architecture: Taming the Quadratic Beast

In standard dense attention, every token acts as a query that compares itself against every single other token. Double the context? You quadruple the compute. But here’s the secret: in trained models, the overwhelming majority of those interactions are basically useless noise. You’re paying top-tier computational prices for comparisons that barely matter.

While the exact underlying mechanics of SSA remain closely guarded, the technical report lays out three requirements the architecture finally solved simultaneously:

• Dense-Attention-Level Quality: SSA uses content-dependent routing. The model looks at the tokens themselves — not a fixed positional grid — to decide where to retrieve information, no matter where it hides.
• True Subquadratic Scaling: SSA is inherently sparse. It selects only a tiny, relevant subset of positions to attend to. Selection, retrieval, and attention are all linear (O(n)) end-to-end.
• Full-Context Flexibility: The efficiency holds strong during both full-context training and autoregressive generation.

The report doesn’t pull punches when comparing SSA to prior attempts. Fixed-pattern sparse attention? Blind to context. State-space models (like the Mamba line)? They degrade in exact retrieval as sequences grow. DeepSeek’s learned-sparse approach? The report argues its indexer is actually quadratic, meaning beyond ~52,000 tokens, it gets crushingly expensive — up to 190x the cost of teacher-attention at 12M tokens. SSA, by contrast, keeps routing content-dependent while staying strictly linear.

Frankensteining a Long-Context Brain

Here’s one of the most fascinating takeaways from the report: SubQ wasn’t built from scratch.

The team took an open-weight “donor” model with a 262K-token context, surgically removed its dense attention, and implanted SSA. From there, they grew its long-context capabilities using YaRN to rescale positional representations.

They leveled it up iteratively: 262K → 512K → 1M → 2M.

Instead of just jumping straight to the finish line, they ran continued pretraining between each stage using naturally long-form data (books, massive documents, entire repos) to teach the model true long-range dependencies. Because SSA made multi-million-token training economically feasible, the team didn’t have to guess at the recipe. They ran over 100 long-context experiments climbing a “memory-scaling ladder” — starting on a single node and scaling up to Ring Attention, keeping iteration cycles blindingly fast.

The Results: Breaking the Benchmarks

Let’s talk numbers, because the headline efficiency is staggering.

At a 1-million-token context, SubQ-1.1-Small achieves a 64.5x reduction in attention FLOPs compared to dense attention. Measured against FlashAttention-2 on a single layer, it hits parity near 16K tokens and completely pulls away at 1M tokens — running in ~966 ms compared to a sluggish ~54,164 ms.

And the generalization is wild. Check out these independently verified stats:

BenchmarkScoreWhat It MeansRULER (13-task)99.12%Near-flawless multi-task retrieval at a 1M token scale.Single-Needle (2M)100%Perfect recall within its maximum pretraining window.Single-Needle (12M)98%Retains insane accuracy an order of magnitude beyond its training length.GPQA Diamond85.4%Graduate-level science (PhDs score ~65%). Strong general capability without over-specializing.LiveCodeBench89.7%Post-training-cutoff competitive programming. High resistance to simple memorization.

Note: The model achieved these massive retrieval numbers while attending to only a tiny fraction of token pairs (described as nearly a 1,000x reduction).

Why This Matters: Whole-Artifact Reasoning

The real payoff of a 12M-token context isn’t just “fitting more text into the prompt.” It’s a paradigm shift from messy retrieval pipelines to true whole-artifact reasoning.

Think about reviewing a dense legal contract, reconciling cross-references in financial filings, or doing a sweeping architectural refactor across a codebase. The hard part isn’t finding a specific passage; it’s understanding the relationships scattered across the entire artifact. Fragmentation destroys those relationships before the AI even gets a chance to look at them.

Subquadratic is measured here — they aren’t saying RAG is entirely obsolete. For massive, ever-changing external corpora, RAG is still king. But they are claiming that a lot of our current AI scaffolding only exists to compensate for tiny context windows. As context grows linearly and cheaply, that scaffolding simply becomes unnecessary.

What’s Next? (The Products)

Subquadratic currently offers SubQ through two products, both in early access:

• The API: Aimed at enterprise teams, it exposes the massive 12M-token context window with streaming and tool use over OpenAI-compatible endpoints. The pitch? Process a full repository or pipeline state in a single call at linear cost.
• SubQ Code: A long-context layer for coding agents. It plugs seamlessly into tools like Claude Code, Codex, and Cursor. With a one-line install, it maps entire codebases and auto-redirects expensive model turns, giving your AI agent the ability to trace dependencies and execute cross-file refactoring in one pass.

SubQ-1.1-Small is just the vanguard. Subquadratic plans to roll out a broader lineup spanning 2M to 12M tokens by year-end.

The broader bet they are making is huge: Whoever makes multi-million-token reasoning genuinely cheap and reliable doesn’t just ship a faster model — they ship a fundamentally more capable one.

Because when you no longer have to hack memory, efficiency really is intelligence.