Beyond DeepSeek’s Engram: What If Compressed Attention Didn’t Have to Be Lossy?

DeepSeek has shipped two ideas in the last year that I keep coming back to. The first is Engram, from their paper on Conditional Memory via Scalable Lookup. The second is the Compressed Sparse Attention and Heavily Compressed Attention mechanisms in DeepSeek-V4. Both are good. The second one has a problem the first one doesn’t, and I think I see how to fix it — with a smaller architectural change than I initially expected.

This is an idea paper. I have not run the experiments yet. They are in the queue behind the HRS work I am finishing, and I will get to them eventually. But the idea is too good to sit on without a public timestamp, so here it is. If anyone reading this wants to run the experiments before I do, please go ahead. I would rather see this tested than see it sit.

The Engram paper builds a learned memory module that hashes short N-grams — two and three token sequences — into a lookup table the model can read in constant time. The mechanism handles collisions with multi-head hashing, normalizes the vocabulary so case and whitespace variants collapse to the same canonical form, and gates the retrieved memory through context. The result is that frequent local patterns get a dedicated, lossless representation. Princess Diana stays Princess Diana. New York stays New York. The model does not have to recompute these short combinations from their constituent tokens every time. It looks them up.

This is the curated version of a more general idea. DeepSeek decided which combinations were worth making lossless and built the machinery to do it for those combinations. The N-gram windows are short on purpose, because handling arbitrary-length combinations through hash tables blows up fast.

The V4 paper goes a different direction. Compressed Sparse Attention compresses KV entries by four-to-one using softmax-gated pooling with a learned positional bias, followed by a top-k selection over the compressed blocks via a lightning indexer running in FP4. Heavily Compressed Attention applies an even more aggressive 128-to-1 compression, without top-k selection — every compressed entry stays available, just at very low resolution. Layers alternate between CSA and HCA across the model. Together they are the reason V4 can attend over a million tokens cheaply. But the compression is lossy. When you fold 128 tokens into one entry, the conventional understanding is that you cannot recover which 128 went in. The pooled vector carries a representation downstream layers can attend to, but the constituent identities and their order are gone.

This is treated as a fundamental property of compression. You pool, you lose. The whole subquadratic-attention literature accepts the trade-off as the cost of efficiency.

I do not think it is fundamental. I think it is an artifact of how the embeddings are pretrained, and that the fix may be smaller than people would expect.

The proposal has two stages, and the separation matters. Stage one is embedding pretraining, with a new regularizer added to the loss. Stage two is downstream model training using compressed attention with the embeddings produced by stage one. The central claim is that stage one alone may be sufficient — that if the embeddings come out of pretraining with the right additive geometry, V4’s existing pool architecture, unchanged, can produce compressed entries from which the original tokens and their order are in principle recoverable. If stage one is not sufficient, a supplementary mechanism can be added in stage two. I will explain both.

Let me put the embedding-training loss function up front, because everything that follows is easier to read with it in mind.

For a vocabulary of N tokens, each token t has an embedding e_t. Let e_t^addr denote the projection of that embedding onto a designated address subspace of roughly 30 to 50 dimensions out of the total embedding dimension. The standard language modeling loss L_LM operates on the full embedding as usual. A new term L_sidon is added:

L_sidon = sum over k from 2 to k_max of L_sidon_k

where for each k:

L_sidon_k = sum over sampled pairs of k-tuples (T_1, T_2) of max(0, gamma — distance(sum(T_1^addr), sum(T_2^addr)))

In words: for every pair of distinct k-tuples of tokens, the regularizer penalizes the case where their address-subspace sums fall closer together than margin gamma. The constraint is A + B cannot equal Q + R, extended to triplets, quadruplets, and so on up to the maximum k the downstream model will need to pool. The sums are computed over the address-subspace projection only, so the semantic dimensions of the embedding are left alone and only the address dimensions get pushed toward Sidon-style separation.

The total embedding-training loss is:

L_total = L_LM + lambda · L_sidon

with lambda a hyperparameter controlling how heavily the regularizer weighs against the language modeling objective.

The k-tuples are sampled rather than enumerated exhaustively, because exhaustive enumeration would be combinatorially intractable — a 50,000-token vocabulary has about 1.25 billion pairs and 21 trillion triplets. Sampling makes the regularizer tractable. The sampling distribution can be uniform over the vocabulary, or biased toward tokens that co-occur in real text, or biased toward k-tuples the downstream pool will actually see during model training. Which sampling strategy works best is a design choice the experiments will resolve.

That is the loss function. The rest of the paper is about why this works, what it costs, and how the downstream model uses the resulting embeddings.

Identity first. For any group of tokens that the downstream model pools together, can the pool sum uniquely encode which tokens were in the group, ignoring order? This is a question about multiset injectivity, and the answer is yes, cheaply.

The information content of a k-multiset drawn from a vocabulary of N tokens is roughly the base-two logarithm of N choose k. For a vocabulary of 50,000 tokens and a pool size of 128 — the HCA setting — that is about 1,400 bits. A 64-bit floating-point number carries about 53 bits of mantissa. The minimum dimensional cost of representing this multiset injectively is 1,400 divided by 53, or roughly 27 dimensions. Allowing margin for training noise and downstream decodability, call it 30 to 50 dimensions. Out of a 4,096-dimensional embedding, that is one percent.

This is a much smaller tax than people might expect because the cost scales as the logarithm of the number of possible k-tuples, not as the number of tuples themselves. For pair-sums alone — k equals two — the cost is roughly one dimension. For k equals four, which is what CSA uses, the cost is two or three. The full scale-up to k equals 128 over a 50,000-token vocabulary still fits in a fraction of a single attention head.

The existence of multiset-injective configurations is not in doubt. Combinatorial number theory has known about Sidon sets — sets of integers where all pairwise sums are distinct — since the 1930s, and their generalizations to B_h sets, where all h-wise sums are distinct, are well studied. Such constructions show the geometry we need is realizable. The question is not whether multiset-injective embeddings can exist. The question is whether SGD can find them under the loss function above.

This is the central empirical question for stage one. The structural argument says yes — the loss landscape rewards Sidon-style geometry, the dimensional budget is sufficient, and the existence of solutions is guaranteed. The experiments have to confirm that gradient descent actually finds the geometry rather than getting trapped in a local minimum that satisfies L_LM well but not L_sidon. My expectation is that with appropriate lambda and a reasonable sampling strategy, SGD finds it. The experiments will tell.

Now order. Pure summation is commutative — A plus B equals B plus A — so a sum-pool of identical tokens in different orders produces identical outputs regardless of how clever the embedding geometry is. No embedding geometry can rescue order information from a commutative pool. This is the natural objection to any geometric injectivity claim, and it is correct as stated for pure sums.

But V4’s pool is not a pure sum. It uses softmax-gated weights with a learned positional bias, which means each within-group position carries its own learned bias term that affects how heavily the token at that position contributes to the pool output. Token at position 0 of the group gets weighted by a softmax over (gate(token) + bias_0), token at position 1 by (gate(token) + bias_1), and so on through position k-1. The pool is non-commutative by construction. Two pools containing the same tokens in different orders pair different tokens with different biases, producing different softmax distributions and different pool outputs.

This raises the central architectural question for stage two: is V4’s existing pool, given Sidon-shaped embeddings from stage one, already sequence-injective?

I think it might be. The k learned biases provide position-sensitivity, the gate function provides token-sensitivity, and the Sidon-shaped address subspace provides multiset uniqueness in the underlying embeddings. Combined, these three pieces have the structural ingredients to make permutations of the same multiset produce distinguishable pool outputs. Whether the expressive capacity is sufficient is exactly what the experiments would determine — particularly at k equals 128, where the demand on position-sensitivity is strongest because there are 128 factorial orderings to distinguish.

If V4’s existing pool is sufficient, the proposal simplifies dramatically. Stage one adds the Sidon regularizer to embedding pretraining. Stage two uses V4’s pool unchanged. No new modules, no new pool inputs, no architectural surgery. The entire contribution lives in the loss function for embedding pretraining.

If V4’s existing pool is not sufficient at large k, there is a supplementary mechanism that closes the gap. Give the pool an additional input — an ordinal list of the token IDs in sequence order. For a six-token group containing “the cat sat on the mat,” the list would be the integers [243, 5421, 132, 8999, 243, 9100] — one ID per token, in the order they appear. The pool function then takes both the dense token embeddings (with their Sidon-shaped address subspace) and the ordinal list. The list provides explicit order information beyond what the positional biases can carry. The pool learns, through gradient descent on the downstream task, how to combine them.

I would only reach for the ordinal list if experiments at k equals 128 showed that V4’s existing positional biases cannot achieve sequence injectivity with Sidon-shaped embeddings. The leading hypothesis is that the simpler version — stage one only, V4’s pool unchanged — works. The ordinal list is a fallback, not the central mechanism.

It is worth being precise about what the pool is actually learning in either version, because the binding step it performs is nontrivial. The pool is learning a joint encoding of a bipartite structure: token identities on one side, position indices on the other, with the pool output a function of both. The identity side comes pre-shaped from stage one — the Sidon-style address subspace gives the pool clean separation to work with. The position side comes from the positional biases (and, in the fallback version, also from the ordinal list). The binding is what the pool has to figure out during stage two. Most of the real training difficulty in stage two sits in that binding, not in either input channel considered separately.

This decomposition is clean. The embeddings have one job: develop multiset injectivity in the address subspace during stage one. The pool has one job during stage two: bind the Sidon-shaped embeddings to position information to produce sequence-injective outputs. Two separate training stages, two separate jobs, each with its own loss function.

There is a piece of the stage-one pipeline I want to flag because it determines whether the proposal works end to end. The embedding training in stage one needs to know what values of k the downstream pool will use, because L_sidon sums over k from 2 up to k_max. V4 uses k equals 4 in CSA and k equals 128 in HCA, interleaved across layers. Stage one needs to cover both — which is what the sum-over-k formulation does. The address subspace gets shaped to handle both the four-token pools that CSA produces and the 128-token pools that HCA produces, simultaneously.

The exact sampling strategy for L_sidon during stage one — whether k-tuples are drawn uniformly across all k from 2 to 128, weighted toward the k values the downstream pool actually uses, or scheduled across training in some curriculum — is a design choice the experiments will resolve. I expect to start with weighted sampling that emphasizes k equals 4 and k equals 128 because those are the values V4 actually uses. The point of this paper is not to specify the recipe. It is to argue that the recipe exists.

There is a deeper objection worth addressing directly. Language modeling pretraining rewards embedding geometry where semantically similar tokens cluster together. The Sidon regularizer rewards embedding geometry where any two tokens — including semantically similar ones — produce distinguishable sums in the address subspace. Do these two pressures fight each other?

Only if they operate on the same dimensions. The model has thousands of dimensions to work with. Gradient descent has no reason to route semantic clustering through the same dimensions that the regularizer is using for injectivity. Given the choice, the system will configure itself to use semantic dimensions for semantics and address dimensions for addressing, because that is the configuration that minimizes L_total. The partition does not need to be hand-specified. It emerges from joint pressure on a system with enough dimensional headroom to satisfy both losses without conflict.

This is the same pattern that produced ESS in the OpenMythos paper: the downstream Coda developed selective sensitivity to project_up’s specific column space through training alone, with no architectural specification of which subspace it should attend to. It is the same pattern as the K/V asymmetry in HRS, where keys ended up more separated than activations because the training signal pushed them that way. It is the same pattern as expert specialization in PEER. Joint training pressure produces subspace specialization. The model finds the partition.

This proposal differs from DeepSeek’s Engram in a way worth being precise about. Engram makes specific N-grams losslessly retrievable by giving them dedicated entries in a hash-indexed memory table. The set of N-grams that get this treatment is fixed by the hashing scheme and the training data. Tokens or short sequences outside the table’s coverage get no benefit. The proposal here does not curate. Every multiset of tokens up to the pool size gets a unique address-subspace sum, and V4’s positional biases (or the ordinal list, in the fallback version) separate different orderings. Princess Diana gets a unique pool output. So does any other pair, and any 128-tuple, including ones the model has never seen during training. The lossless property is universal, bounded only by the dimensional budget of the address subspace.

It also differs from Engram in what it costs. Engram adds a separate memory module — hash tables, multi-head hashing logic, gating mechanisms — that the model has to maintain alongside its attention machinery. The proposal here, in its primary form, costs one percent of the existing embedding dimensions and one new term in the embedding pretraining loss. No new modules. No architectural changes to V4. The compression operations V4 already uses become lossless without major modification. The fallback version, if needed, adds the ordinal list as a second input to the pool — still a small change relative to redesigning attention from scratch.

The result, if it works, is a version of CSA and HCA that does not pay the lossiness tax. Compressed attention becomes a strictly better trade-off than it currently is, because the only thing it gives up — fidelity to constituent token identity and order — is no longer given up. Million-token contexts get cheaper without becoming less expressive.

There is one more thing worth saying about why I find this idea compelling beyond the immediate engineering payoff. The dominant frame in the subquadratic-attention literature treats compression as an algorithmic choice — which tokens to pool, how to weight them, when to apply sparse selection. The frame here is that compression quality is partly a property of the embedding space and partly a property of the pool function, and the lossiness or losslessness of any particular setup is downstream of how both are configured and trained. If this turns out to be right, it suggests that some of what we treat as algorithmic innovation in attention is actually a search for compensations for embeddings that were never optimized for the pools they get fed into. Pretrain the embeddings for the geometry the pool needs, and the algorithms may already be doing more than we have given them credit for.

I do not know yet if this works. The experiments will tell me. But the structural argument seems clean enough that I would rather publish it now and find out I was wrong than sit on it and find out somebody else got there first.

Notes and citations

DeepSeek’s Engram paper: Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models. Available at github.com/deepseek-ai/Engram.

DeepSeek-V4 technical report, describing CSA and HCA: github.com/deepseek-ai/DeepSeek-V4-Pro, with the technical report linked from the model card.

Sidon sets and B_h generalizations are covered in any standard reference on additive combinatorics. Halberstam and Roth’s Sequences is the classic.

The K/V asymmetry and ESS findings I reference are from my own work on Hierarchical Routed Sinkformers and on testing the OpenMythos hypothesis in looped transformers, both available on Medium under this handle.

This piece was developed in conversation with Claude and ChatGPT. The early framing came from a Gemini draft that overstated the contribution in ways that did not survive scrutiny. Claude pushed back on the dimensional accounting and the Sidon-construction claims until what remained was defensible. ChatGPT then caught that pure-sum pooling cannot preserve order regardless of embedding geometry, which forced the recognition that V4’s existing positional bias is doing important work, and that the proposal could potentially be even simpler than initially framed. ChatGPT also contributed the precise framing of the pool’s job as learning a joint encoding of a bipartite structure. ChatGPT separately proposed a larger retreat from the lossless framing toward a more cautious “factorized compressed attention” headline, which I declined; the structural argument for losslessness is intact, and the right test of it is experiment, not preemptive hedging.

The cross-validation between models did what cross-validation is supposed to do. Without ChatGPT’s order objection, the published version would have overclaimed sequence injectivity on the basis of an argument that only supported multiset injectivity. Without Claude’s pressure on the math, the proposal would have inherited Gemini’s combinatorial loss formulation, which is intractable at scale.

The idea is mine. The corrections came from the models pushing on it.