EPM-JEPA: Can World Models Learn From Experience Without Retraining?

A look at Vedant Pandya’ s & Jatinkumar parmar’s new preprint on adaptive predictors for JEPA-family world models.

EPM-JEPA: Can World Models Learn From Experience Without Retraining?

JEPA-style world models have become a popular foundation for predicting how environments evolve, but they share a common weakness: their predictor weights stay fixed at test time, so the model can’t adjust when the dynamics it encounters start to drift away from what it saw during training. A new preprint, EPM-JEPA: Operator-Side Experience Modulation in JEPA-Family World Models (arXiv:2606.12979), takes this problem head-on by asking a simple question: when a model accumulates “experience” at test time, where should that experience go?

Two Ways to Inject Experience

The paper compares two architectural choices for folding a compressed memory of past experience back into the predictor:

• Operand-side injection (EI-JEPA): the experience representation is added directly as a residual to the predictor’s hidden state, nudging the data flowing through the network.
• Operator-side modulation (EPM-JEPA): the same experience representation instead generates low-rank weight deltas via LoRA, effectively reshaping the predictor itself rather than the data passing through it.

It’s the classic distinction between adjusting what a model sees versus adjusting how the model processes it.

What the Experiments Found

Using a pre-registered comparison on Moving MNIST under a gravity shift, the headline numbers were close: EPM-JEPA scored 0.7848 ± 0.0078 on the shift metric across three seeds versus 0.8238 for EI-JEPA, a 4.74% difference that the authors classify as a null result under their own pre-registration criteria — and they’re upfront that this counts as a valid, informative outcome rather than a failure.

The more interesting story shows up in the secondary analysis: EPM-JEPA improved 1.90% over a no-memory baseline consistently across seeds, while EI-JEPA actually underperformed that same baseline, suggesting that modulating the operator’s weights — not just its inputs — is where the real benefit lives.

The Real Contribution: A Mechanism Analysis

Rather than stopping at a leaderboard number, the paper digs into why the shift metric behaves the way it does over time. It identifies three independent dynamical processes underlying the trajectory — buffer cycling, EMA target drift, and an intrinsic LoRA “settling transient” of +0.021 — arguing the curve reflects these overlapping dynamics rather than a clean convergence to equilibrium.

That diagnosis sets up the next step: the findings motivate a physics-grounded successor, PEM-JEPA, designed to address this dynamical-peak limitation.

Why It Matters

For anyone working on world models, context engineering, or test-time adaptation, this preprint is a nice example of treating a “null result” as data rather than noise — and of separating architecture questions (where should memory live?) from dynamics questions (how does the system evolve once it’s there?).

📄 Read the full preprint: https://arxiv.org/abs/2606.12979

WorldModels #JEPA #MachineLearning #AIResearch #SelfSupervisedLearning #ContextEngineering #DeepLearning #LoRA #ArXivPreprint #MultilingualLLMs #AICollaboration