State Aggregation and Latent Space Collapse

When Finer Representations Destroy Stable Features

Stable prototypes at low dimensions. Refine further and the prototypes shatter into noise. Theorem T-IC-01 predicts when this happens.

I trained a variational autoencoder last year. Sixteen latent dimensions. It worked fine — blurry, sure, but stable. Then I got ambitious. Bumped it to 64 dims. More capacity, more expressivity, finer distinctions. The thing collapsed. Not gradually. Catastrophically. Mode collapse, dead neurons, reconstructions that looked like pixel vomit.

I thought it was a bug.

It wasn’t a bug. It was a theorem.

The folk wisdom is wrong

Everyone in ML knows the script. Higher-dimensional representations are better. More dims means more capacity. Finer partitions mean richer features. If your model is struggling, add more layers, more parameters, more latent codes. Scale is the answer.

Except when it isn’t.

My VAE didn’t fail because I needed better hyperparameters. It failed because I refined the latent space across a mixing boundary. The 16-dim version worked by accident — it happened to align with the data’s metastable structure. The 64-dim version cut that structure into pieces, and the pieces didn’t survive forward propagation.

This is Theorem T-IC-01 from Six Birds Theory. And it says, very precisely: refinement can help or hurt.

What ML calls a latent space is a lens

Let me put the translation layer on the table.

Six Birds TheoryMachine LearningLens f : Z → XEncoder / latent representationPrototype uₓEmbedding / cluster centroidCanonical lift UₓDecoderIdempotent endomap Eτ,fAutoencoder reconstruction operatorIdempotence defect δτ,fReconstruction error / mode collapseRetention error ετ,fPrototype stability under forward passFixed points Fix(E)Stable learned features

These are not analogies. They are mathematically identical.

A latent space is a coarse-graining lens: a map from high-dimensional data Z to a lower-dimensional representation X. An autoencoder is an induced empirical endomap: encode, evolve (implicitly, via the training dynamics or forward pass), decode. Mode collapse is an idempotence defect: apply the reconstruction operator twice and you get a different answer.

This is not a metaphor. This is the same operator.

The theorem (in plain English)

Theorem T-IC-01 says this. Take a Markov kernel P (any stochastic dynamics). Pick a time scale τ. Now pick two lenses: one coarse, one fine. The fine lens refines the coarse — it makes finer distinctions, splits blocks, adds detail.

Now ask: how many stable prototypes does each lens support?

The answer is: it depends.

Which is like saying “all dogs are mammals; Rallis is a mammal; therefore Rallis is a dog.” Formally invalid. But if you’ve met my cat Rallis, you’d consider it. He bites. He’s territorial. He operates on canine logic. The syllogism is broken but the conclusion survives contact with reality. Refinement is like that — the intuition (“finer is better”) has the shape of a law, but the theorem says no. Sometimes yes, sometimes no. It depends on alignment.

Here’s the mechanism. If the fine lens aligns with metastable regions — if it splits blocks along natural boundaries where mixing is slow — refinement reveals more stable objects. The retention error stays small. The idempotence defect stays small. You get more features, and they’re stable.

But if the fine lens cuts across mixing boundaries — if it splits a metastable cluster into pieces that leak into each other — refinement destroys stability. The prototypes don’t stay put under forward propagation. The retention error blows up. The idempotence defect blows up. You lose features.

So refinement is not monotone-good.

Finer distinctions are not always better. Adding capacity can make your model worse, not by overparameterization, but by fracturing the stable structure your coarse representation accidentally captured.

That’s the theorem.

Why this matters (and why I wish I’d known it sooner)

I spent a week debugging that VAE. Tuning learning rates, adding regularization, swapping architectures. I thought the problem was optimization.

It wasn’t. The problem was geometry.

The 64-dim latent space was too fine. It split the data manifold across mixing timescales. The decoder’s prototypes (the learned cluster centers) were supposed to be stable fixed points of the reconstruction operator, but they weren’t. They drifted. The retention error — the probability that a prototype, after being pushed forward through the dynamics and re-encoded, lands back where it started — was too high.

The 16-dim version worked because it was coarse enough to respect the data’s natural grain. It didn’t try to make distinctions the dynamics couldn’t support.

Wait. This explains a lot of ML failure modes. VAE collapse when you increase latent dims. K-means breaking when you add too many clusters. Why you can’t naively upsample latent codes without retraining. Why hierarchical models sometimes get worse as you add levels.

The folk wisdom says: more capacity is always better, just regularize harder.

The theorem says: capacity is only useful if it aligns with retention structure.

I learned this after wasting a week on hyperparameter sweeps. Would’ve helped to know the theorem first.

The Six Birds framing

This is all P5 Packaging in Six Birds Theory.

Six Birds Theory (Emergence Calculus)

P1 Rewrite — change the rule. P2 Gating — restrict what’s allowed. P3 Protocol Holonomy — hidden phase, route mismatch. P4 Sectors — invariants, staging, timescale separation. P5 Packaging — idempotent completion; fixed points are “objects.” P6 Accounting — audits/monotones; what can’t be faked by forgetting.

P5 says: a lens induces a packaging map — an (approximately) idempotent endomap whose fixed points are the “objects” your theory recognizes. In ML, those objects are your learned features, your cluster centers, your latent codes. They’re stable if and only if the lens respects the retention timescale.

The meta-theorem says these primitives aren’t arbitrary. They’re forced by composability and limited access. Once you have processes with interfaces, you get lenses. Once you have lenses, you get packaging. Once you have packaging, you get idempotence defects. Once you have defects, you get stability conditions. Once you have stability conditions, you get Theorem T-IC-01.

It’s turtles all the way down, but the turtles have a proof.

Back to the VAE

So I rebuilt the thing with 16 dims. It worked. Then I ran the mixing-timescale diagnostic: I tracked how far each latent code drifted after a forward pass through the decoder and a re-encode. The 16-dim codes drifted around 0.02 in TV distance. The 64-dim codes drifted around 0.31.

That’s the retention error. That’s why one worked and the other didn’t.

The collapse wasn’t a bug. It was the theorem telling me: your representation is too fine for the dynamics you’re trying to model. You’re making distinctions the data can’t support. Scale back. Align with the grain.

I did. It worked.

Punchline

Representation quality is not monotone in dimension.

It’s monotone in alignment.

If your latent space cuts across mixing boundaries, adding dims just fractures your features. If it aligns with metastable structure, refining can reveal new stable objects. The difference is retention: do your prototypes stay put, or do they drift?

That’s Theorem T-IC-01. That’s why my 64-dim VAE collapsed and my 16-dim VAE didn’t. That’s why “add more capacity” is not a law of nature.

And that’s why, when someone tells me latent spaces are just a useful ML trick with no deep structure, I point them to the idempotence defect formula and say: no.

This is a lens. This is an endomap. This is a theorem.

Anyways. That’s it.