CompreSSM: Compressing State-Space Models During Training with Hankel Singular Values

How Control Theory Powers Efficient In-Training Pruning in Deep AI Architectures

A Surge in Efficient AI: Model Compression During Training

Cutting the computational costs of machine learning models, especially in deep architectures, is a top priority for researchers and practitioners alike. Now, a new approach called CompreSSM — highlighted by MIT CSAIL News and accepted at ICLR 2026 — proposes compressing state-space models while they’re still being trained. The key idea: use control-theoretic metrics, specifically Hankel singular values (HSVs), to prune away unnecessary model subcomponents early — resulting in faster, leaner models with essentially no drop in performance.

Why is this groundbreaking? Most pruning strategies and model distillation techniques are employed after training: a model is trained for days (or weeks), then compressed. CompreSSM, by contrast, injects pruning into the training loop itself. The result: accelerated learning and lower hardware requirements — a major win for real-world machine learning applications.

The Mathematics of Hankel Singular Values

Why Hankel Singular Values Matter

If you’re familiar with control theory, you’ve likely seen Hankel singular values before. In linear dynamical systems — core to state-space models — they quantify how much each state contributes to the system’s input-output behavior. Mathematically, given a continuous-time state-space system:

dx/dt = Ax(t) + Bu(t) y(t) = Cx(t) + Du(t)

where x(t) is the latent state, u(t) the input, and y(t) the output, Hankel singular values are derived from the solutions to the controllability and observability Gramians (P, Q):

A P + P Aᵀ + B Bᵀ = 0, Aᵀ Q + Q A + Cᵀ C = 0

The eigenvalues (or more precisely, the square roots of the eigenvalues) of PQ are the Hankel singular values.

A large HSV means that manipulating that part of the model’s state makes a big impact in output — small HSVs mean the state is nearly ‘invisible’ to the model’s predictions. These low-HSV subsets are prime candidates for removal with almost no impact.

Pruning State-Space Models via HSVs

Traditionally, pruning focuses on weights (e.g., L1, L2 norm) or activation magnitudes. But HSVs encode a more global, dynamical-system view:

• Prune entire subspaces corresponding to small HSVs, effectively lowering the rank and size of the hidden state.
• This can be done iteratively as the model trains — small HSVs emerge, their components pruned, and the network continues to adapt.

From Post-Training to In-Training Pruning

Traditional Compression Methods

• Pruning after training: Cut small-magnitude weights or entire neurons at the end, often requiring retraining to regain accuracy.
• Distillation: Train a smaller ‘student’ network to mimic a large ‘teacher’ model, typically as a second phase after initial training.

While effective, these methods are computationally expensive: you pay the full cost to train the big model, then pay again to compress and fine-tune.

The CompreSSM Paradigm

CompreSSM’s approach is different:

• Compute HSVs during training at regular intervals.
• Prune unnecessary state subspaces on-the-fly.
• Continue training the reduced model, allowing adaptation and filling gaps.

This not only saves memory and flops but can also speed up overall training time. In early benchmarks (per MIT CSAIL), CompreSSM achieved up to 3x speedups in training state-space models for sequence tasks, with no perceptible loss in test accuracy compared to standard training plus post-hoc pruning.

Implementing CompreSSM in Practice

Let’s see how CompreSSM-style pruning might look in a modern PyTorch training loop for a state-space model (SSM). For pedagogical purposes, assume you have a model where system matrices (A, B, C, D) are parameterized and differentiable.

import torch

def prune_by_hsv(A, B, C, threshold=1e-2):
    # Compute controllability and observability Gramians via Lyapunov equation.
    # For small models, can use SciPy (torch not available):
    from scipy.linalg import solve_continuous_lyapunov as lyap
    A_np = A.detach().cpu().numpy()
    B_np = B.detach().cpu().numpy()
    C_np = C.detach().cpu().numpy()
    P = lyap(A_np, -B_np @ B_np.T)
    Q = lyap(A_np.T, -C_np.T @ C_np)
    # Hankel singular values
    hsvs = np.sqrt(np.linalg.eigvals(P @ Q))
    # Indices to keep
    keep = hsvs > threshold
    # Prune A, B, C to principal subspace
    return A[keep][:, keep], B[keep], C[:, keep]

# Example in a training loop:

for epoch in range(EPOCHS):
    loss = train_one_epoch(model, data)
    if epoch % PRUNE_INTERVAL == 0:
        A, B, C = model.get_ssm_matrices()
        A, B, C = prune_by_hsv(A, B, C, threshold=THETA)
        model.set_ssm_matrices(A, B, C)

Of course, modern CompreSSM implementations offer more scalable, batched, and GPU-friendly code — but the principle remains: let the model self-compress according to its HSV spectrum as it learns.

When To Use CompreSSM

CompreSSM is best suited for:

• State-space models: e.g., S4, LSSL, DSS, and other ODE-inspired models.
• Sequential tasks: speech, audio, time series, language modeling, control.
• Resource-constrained training: edge devices, limited-GPU settings, fast/hyperparameter-search explorations.

It’s less relevant for purely feedforward or convolutional architectures without a dynamical system core.

Key Takeaways

• Hankel singular values offer a mathematically principled way to identify dispensable components in state-space models.
• Traditional pruning and distillation are effective but only after full training; CompreSSM moves pruning inside the training loop for faster, smaller, and more adaptable models.
• With CompreSSM, you can enjoy the benefits of model compression — smaller footprint, faster inference and training — without sacrificing accuracy or doubling your engineering effort.

For full details, code, and the latest updates, check the MIT CSAIL news story and the ICLR 2026 proceedings.