Latent Space Models in AI: JEPA, Latent World Models, and the Shift Beyond Pixel and Token Prediction

For most of the last decade, self-supervised learning has followed a simple recipe: predict missing pixels, words, frames, or tokens. BERT masked tokens, GPT predicted them autoregressively, MAE reconstructed image patches, and video diffusion models generated frames sequentially. The recipe scales beautifully on benchmarks but encodes an implicit assumption that has become harder to defend as systems leave the lab: that intelligence is well captured by the ability to reproduce surface signals. Yann LeCun’s 2022 position paper “A Path Towards Autonomous Machine Intelligence” (OpenReview, June 2022; revised and discussed extensively in 2023) argued the opposite: that pixel-perfect or token-perfect prediction can waste capacity on details no agent needs, especially when the downstream task is perception, planning, or control rather than generation, and fails on the part that actually matters, namely building an internal model of how the world evolves at the level of abstraction relevant to acting in it. Since then, that argument has split into several related but still emerging research programs. One branch learns abstract predictive representations, as in I-JEPA, V-JEPA, V-JEPA 2, and VL-JEPA. A second branch learns latent dynamics models for control, as in TD-MPC2, DreamerV3, MuZero, and LeWorldModel. A third branch explores reasoning or language modeling in continuous or concept-level spaces, as in Coconut, Large Concept Models, Dynamic LCM, and the Byte Latent Transformer. These systems are not the same thing, but they share a dissatisfaction with raw pixel or token prediction as the only substrate for intelligence. This article surveys that landscape with engineering detail: what these models actually do, how the loss functions look term by term, how they avoid the collapse failure modes that haunted earlier latent approaches, what their measured benchmarks look like, and why a growing number of research groups is exploring prediction in abstract embedding spaces as an alternative to, and in some settings a partial replacement for, reconstruction or token-level generation.

1. Why Latent Space: The Failure Modes of Pixel and Token Prediction

A fundamental limitation of pixel-level world modeling is that the world is irreducibly stochastic at the resolution of pixels but mostly deterministic at the level of objects, contacts, and intentions. A driving model that tries to predict the next frame must simultaneously commit to the exact motion of every leaf on every roadside tree, the exact glint on every metal surface, and the exact placement of every pedestrian’s shadow. No agent needs that information, and the loss function spends most of its capacity learning textures rather than dynamics. This is not merely theoretical: in PlaNet and the original World Models, ablations showed that the reconstruction term dominated the gradient signal by one to two orders of magnitude relative to the dynamics term, and removing the decoder produced policies of lower quality only because the encoder had no other supervision to organize itself. Generative video world models, from PlaNet through Sora, Genie, and Genie 2, partially mitigate this with stochastic latent variables and diffusion-based decoders, but the cost remains: the bulk of the parameters and the bulk of the compute serve a reconstruction objective that is strictly downstream of the representation a planner actually uses, and the model cannot abstract over visual nuisance variables it was never told to ignore.

The same critique applies, in a weaker but still important form, to token-level language modeling. A language model forced to commit to a single next word at every step cannot represent ambiguity, branching reasoning, or the kind of “I don’t know yet” intermediate state that humans hold during multi-step problem solving. Chain-of-thought prompting works around this by externalizing intermediate steps as tokens, which is effective but expensive and constrains reasoning to whatever can be linearized into natural language. A theorem-proving step that, internally, considers three possible lemmas in parallel must serialize itself into a single chosen lemma at each token position; the alternatives are lost the moment the next token is sampled. Token-level prediction also commits hard to a tokenizer chosen at training time, with all the morphological brittleness that BPE brings (numbers split arbitrarily, multilingual rare words exploding into byte-level fragments, code identifiers fragmenting in inconsistent ways).

Latent-space prediction fundamentally changes this contract. Instead of predicting the raw signal y from the raw signal x, the model predicts an embedding s_y(y) of the future or masked target from an embedding s_x(x) of the context, optionally conditioned on a latent variable z that absorbs aleatoric uncertainty and an action variable a that conditions the dynamics. The objective collapses to:

where P is a learned predictor, D is a distance in embedding space (typically L2 or smooth L1), Ω is a regularizer that prevents the encoders from collapsing to a constant, and z is sampled from a learned posterior or a fixed prior. The key shift is that the loss operates only on learned embeddings rather than directly on raw pixels or tokens. Whatever information is irrelevant for predicting the embedding is allowed to be discarded by s_y, which is precisely the abstraction that pixel-level models struggle to learn. The trade-off is that the loss is no longer tied to a likelihood objective, so naive training collapses; most of the engineering of modern latent-space models is about making the training stable in spite of that.

2. Theoretical Framing: Energy-Based Models and the JEPA Taxonomy

The cleanest way to understand the JEPA framework is as a special case of energy-based modeling. An energy function F(x, y) assigns a scalar to each context-target pair, and training shapes F to be low for compatible pairs and high for incompatible ones. Generative models are one way to obtain such an F: define F(x, y) = -log p(y | x) and minimize negative log-likelihood. The catch is that this requires a normalized density over y, which is tractable only when y is discrete (tokens) or when one accepts the cost of decoding at full resolution (pixels, waveforms). JEPAs sidestep explicit normalization by operating directly in embedding space:

and shaping F low only for observed pairs, with collapse prevented by architectural and regularization tricks rather than by an explicit partition function. LeCun’s position paper organizes the design space along three axes: deterministic vs stochastic (does the predictor consume a latent z?), invariant vs equivariant (does the predictor consume an action or coordinate variable a?), and single-scale vs hierarchical (does the model predict at one time scale or several). The full hierarchical JEPA, sometimes labeled H-JEPA, predicts at multiple temporal granularities with high-level latents capturing slow events and low-level latents capturing fast ones; it remains aspirational, but the taxonomy is useful for placing the published models. I-JEPA is deterministic, equivariant on patch coordinates, single-scale; V-JEPA is deterministic, equivariant on spatio-temporal coordinates, single-scale; LeWorldModel is stochastic in its latent prior and equivariant on actions; Coconut is stochastic and recurrent; LCM is stochastic via flow matching in concept space.

3. Two Meanings of “Latent”: Compressed-Reconstructable vs Abstract-Predictive

Before surveying the individual model families, it is important to disambiguate a terminological collision that causes persistent confusion. The phrase “latent space” is used by two meaningfully different research programs, and failing to distinguish them makes the literature look more unified than it actually is.

Latent diffusion models like Stable Diffusion (Rombach et al., 2022), Sora (OpenAI, 2024), and Genie / Genie 2 (DeepMind, 2024) operate in a latent space, but the latent is produced by an autoencoder trained with a reconstruction loss on the observation space, and the dynamics or sampling model is a denoising network trained to invert a noising process whose ultimate objective is still reconstruction. The latent compresses, but it is still optimized to preserve reconstructable information: it preserves enough information to decode back to pixels, which means it preserves textures, colors, and visual nuisance variables. This is exactly the right design for generation, where the goal is to produce visually plausible outputs, and latent diffusion is the dominant paradigm for image, video, and audio synthesis.

JEPAs, TD-MPC, and LeWorldModel operate in a fundamentally different kind of latent space: one with no decoder, no reconstruction objective, and no constraint that the latent retain any specific information about the observation beyond what is needed to predict future latents. The two programs share a phrase but have opposite design intents: latent diffusion compresses while preserving visual fidelity, JEPA compresses while discarding everything that is not predictive. Sora can generate a minute of photorealistic video; V-JEPA 2 cannot generate a single pixel. V-JEPA 2 is explicitly evaluated for planning and robotic control, while Sora-style models are designed for high-fidelity video generation rather than explicit planning. Both are useful, and they are not in competition for the same role. The interesting open question is whether they will eventually merge: a model that can both reason and plan in an abstract latent and decode that latent to pixels when a human needs to inspect what the agent is thinking. Genie 2 takes a step in this direction by conditioning its diffusion decoder on action tokens, and several recent papers have proposed dual-decoder architectures in which a JEPA-style abstract latent feeds both a planner and a generative head.

This distinction matters throughout the rest of this article. When we say “latent-space prediction,” we mean the abstract-predictive variety unless explicitly stated otherwise.

4. A Map of the Territory: Representation, World Modeling, and Reasoning

The models surveyed here share a broad motivation, moving the predictive objective away from raw observations and into a learned embedding space, but they address different problems, operate at different levels of abstraction, and are at different stages of maturity. It is important not to conflate them. The landscape divides naturally into three clusters:

Latent predictive representations (I-JEPA, V-JEPA, V-JEPA 2, VL-JEPA, audio-JEPA, DINOv2). These models learn encoder features by predicting masked or future embeddings. Their primary output is a reusable feature backbone, and they are evaluated by linear probing, fine-tuning transfer, and downstream task accuracy. They do not, by themselves, plan or reason; they produce useful representations that downstream modules can consume.

Latent world models for dynamics and control (LeWorldModel, TD-MPC, TD-MPC2, DreamerV3, MuZero, action-conditioned V-JEPA 2). These models learn a forward dynamics model in latent space and use it for planning or policy optimization. Their primary output is a trajectory of predicted latent states that a planner can score. The gap between a good representation and a good world model is non-trivial: the latent must capture not just static features but causal transitions, and planning in latent space requires the dynamics model to be accurate over multi-step rollouts where errors compound.

Latent reasoning and concept-level language models (Coconut, recurrent-depth latent reasoning, Quiet-STaR, LCM, Dynamic LCM). These models explore reasoning or sequence modeling in continuous embedding spaces rather than token-by-token in vocabulary space. Their primary output is either a final text answer (Coconut) or a predicted next-concept embedding decoded to text (LCM). They address a different bottleneck than the vision models, namely the rigidity and cost of discrete token-level reasoning, rather than the cost of pixel-level reconstruction.

A fourth group, the Byte Latent Transformer (BLT), is related but distinct: it modifies the unit of computation from fixed BPE tokens to entropy-adaptive byte patches, which changes what the model sees but does not abandon the autoregressive language modeling objective. BLT shares the spirit of moving the prediction substrate to a more abstract level, but it does not operate in a decoder-free predictive embedding space the way JEPAs do.

These clusters share a family resemblance, and researchers increasingly move between them, but a learning-a-good-visual-backbone result from I-JEPA does not automatically validate a claim about latent-space planning, and a Coconut result on logical puzzles does not automatically transfer to video world modeling. Keeping this map in mind guards against overgeneralization.

5. The Lineage of Self-Supervised Vision: From SimCLR to DINOv2 to JEPA

The JEPA family did not appear in a vacuum. It is the latest stage of a five-year arc of self-supervised vision research. Contrastive methods like SimCLR (Chen et al., 2020) and MoCo (He et al., 2020) trained encoders by pulling augmented views of the same image together and pushing different images apart, requiring large batches or memory queues to provide enough negatives. They worked, but the negatives were a nuisance: they conflated semantically similar images as negatives whenever they happened to land in the same batch, and they made the loss landscape sensitive to batch composition. BYOL (Grill et al., 2020) startled the field by removing negatives entirely: a target network updated by EMA of the online network’s weights provided the prediction targets, and the asymmetric architecture (a predictor head only on the online branch) was empirically sufficient to prevent collapse. SimSiam (Chen and He, 2021) showed that even the EMA was not strictly necessary, as long as a stop-gradient was placed on the target branch.

In parallel, Barlow Twins (Zbontar et al., 2021) introduced the idea of regularizing the cross-correlation matrix of two views toward the identity, encouraging invariance on the diagonal and decorrelation off the diagonal. VICReg (Bardes, Ponce, LeCun, 2022) generalized this with three explicit terms (variance, invariance, covariance), making the regularization recipe interpretable and tunable. DINO (Caron et al., 2021) and DINOv2 (Oquab et al., 2023) brought the self-distillation framing to its current pinnacle: a student-teacher pair with EMA targets, multi-crop augmentation, and a sharpening / centering operation on the teacher outputs that prevents collapse without explicit regularization terms. DINOv2 features became, for a time, the de facto frozen backbone for downstream vision tasks, outperforming CLIP on dense prediction.

JEPA is the conceptual next step: it generalizes the Siamese setup of DINO and BYOL by introducing a non-trivial predictor that operates between context and target embeddings, conditioned on coordinates or actions, so that the model is no longer just learning view-invariant features but learning to predict targets it has never seen. The shift matters because it turns a representation-learning method into a latent dynamics model; the same architecture that learns features for image classification can, with the addition of a temporal axis, learn the dynamics of a video, and with the addition of an action input, the dynamics of a controlled environment. The continuity from SimCLR through DINOv2 to V-JEPA 2 is the story of how the field gradually realized that prediction in embedding space is a more powerful organizing principle than view invariance, and JEPA can be read as a generalization of the joint-embedding family: instead of only matching augmented views, the predictor learns to map context embeddings to target embeddings that may be spatially or temporally missing.

6. The JEPA Family in Depth: I-JEPA, V-JEPA, V-JEPA 2, LeWorldModel

I-JEPA (Assran et al., arXiv 2301.08243, 2023) instantiated LeCun’s framework on static images. Given an image, several large rectangular target blocks (typically four blocks covering 15–20 percent of the image area each) are masked out, the visible context is encoded by a context ViT, and a small predictor network (a few transformer blocks, roughly 5 percent of the context encoder size) conditioned on the spatial coordinates of each target tries to recover the embedding of the masked block as produced by an exponential-moving-average target encoder. There is no decoder, no pixel reconstruction, and no contrastive negatives. On ImageNet linear probing, I-JEPA reported competitive linear-probe performance with a ViT-H/14 trained for far fewer epochs than MAE or DINOv2, and strong transfer to downstream dense prediction tasks including semantic segmentation and depth estimation. The crucial design decision was the use of large, semantically meaningful target blocks rather than scattered patches: predicting a single masked patch is mostly a low-frequency interpolation problem solvable by averaging neighbors, while predicting an entire object region forces the encoder to capture object-level structure. The mask sampling strategy is itself a hyperparameter that materially affects what the model learns.

V-JEPA (Bardes et al., arXiv 2404.08471, 2024) extended the recipe to video by masking spatio-temporal tubelets. The context encoder sees a heavily masked video clip (typically 90 percent of tubelets removed), and the predictor recovers the embeddings of the missing tubes. A model with roughly 600M parameters, trained on a curated mix of public video totaling around 2 million clips without any text supervision, produced features competitive with image-text contrastive models on action recognition (Kinetics-400, Something-Something v2) and outperformed them on tasks that require fine motion understanding such as Epic-Kitchens-100. The training cost was roughly 6x lower than equivalent reconstruction-based video models, because the predictor only had to match a low-dimensional embedding rather than reconstruct hundreds of pixels per masked region.

V-JEPA 2 (Assran et al., 2025) scaled the encoder to roughly 1.2 billion parameters and the data to over 1 million hours of video, and added two key changes. First, an action-conditioned variant in which the predictor consumes a discretized action token alongside the context embedding, trained on a mix of passive video and a small set of action-labeled robot trajectories. Second, a longer training schedule with curriculum mask scheduling that gradually increases the spatial extent of the masked regions. The action-conditioned model demonstrated promising transfer to manipulation tasks (pick-and-place, drawer opening, simple stacking) with minimal additional supervision where the latent dynamics learned from passive video served as a planner for a physical arm with a comparatively small amount of action-labeled robot data. This is one of the clearest demonstrations so far of the idea that passive video pretraining can support downstream planning and control through latent predictive modeling, though the range of tasks demonstrated remains narrow and the gap between “JEPA features are useful for a downstream planner” and “a JEPA-based agent that autonomously solves open-ended problems” has not been closed.

LeWorldModel (Maes et al., arXiv 2603.19312, 2026) is the most recent and arguably the cleanest simplification of the JEPA-as-world-model recipe. Earlier JEPA-based world models depended on six or more interacting hyperparameters governing encoder regularization, target normalization, predictor temperature, EMA decay, masking ratios, and auxiliary losses, and were notoriously brittle to retune across datasets. LeWorldModel reduces the recipe to two terms:

a next-embedding prediction loss between the predictor’s output and a stop-gradient target embedding, and a KL term that pushes the marginal distribution of latent embeddings toward an isotropic Gaussian prior. The Gaussian regularizer prevents collapse without requiring contrastive pairs, EMA targets, or covariance whitening. A roughly 15M-parameter model trains end-to-end from raw pixels on a single GPU in a matter of hours and, when used as a learned dynamics model inside a sampling-based planner, executes plans significantly faster (reported up to 48x in the authors’ experimental setup) than world models built on top of frozen foundation encoders, while detecting physically implausible rollouts (objects passing through walls, gravity violations) more reliably than reconstruction-based baselines. LeWorldModel is best understood as a promising demonstration that end-to-end JEPA world models can be made far simpler and more stable than earlier recipes, not as a definitive resolution of the field’s open training-stability questions. The paper’s contribution is methodological simplification; whether that simplification holds as models scale by two or three orders of magnitude remains to be tested. A parallel line of work, Bardes et al.’s “Learning and Leveraging World Models in Visual Representation Learning” (arXiv 2403.00504, 2024), generalizes JEPA by treating the gap between context and target as an explicit transformation conditioned on a latent action variable, blurring the line between self-supervised representation learning and model-based reinforcement learning.

7. Avoiding Representation Collapse: Five Strategies and Their Tradeoffs

Any joint-embedding objective that minimizes a distance between two embeddings has a degenerate solution: map every input to the same point. The literature has produced five distinct strategies for preventing this, and each modern latent-space model uses some combination of them.

The first is contrastive negatives, used by SimCLR, MoCo, and CLIP. The InfoNCE loss explicitly pushes apart embeddings of unrelated samples, and the gradient on the negatives prevents collapse. The cost is large batch size requirements (typically 4096 or more) and sensitivity to false negatives in noisy data. The second is asymmetric architectures with stop-gradients, used by BYOL, SimSiam, DINO, I-JEPA, and V-JEPA. A predictor head sits only on the online branch, and the gradient does not flow into the target branch; the EMA update of the target keeps it slightly behind the online network and prevents the trivial solution from being reached. The third is explicit covariance regularization, used by Barlow Twins, VICReg, and W-MSE: the cross-correlation or covariance matrix of the embeddings is shaped toward the identity, which forces the embedding dimensions to be both informative and decorrelated. VICReg in particular decomposes the loss as:

where s is an MSE invariance term between paired embeddings, v(Z) = mean over dimensions of max(0, gamma — sqrt(Var(Z) + epsilon)) is a hinge variance term penalizing dimensions whose standard deviation falls below gamma (typically 1), and c is a covariance term that decorrelates feature dimensions by penalizing the squared off-diagonal entries of Cov(Z). The variance term alone is enough to prevent collapse without negatives, and the covariance term ensures the embedding uses its dimensions efficiently. The fourth is centering and sharpening of teacher outputs, used by DINO and DINOv2: a running mean is subtracted from the teacher’s logits to prevent any single output dimension from dominating, and a sharpening temperature is applied to keep the teacher distribution from being uniform. The fifth, and most recent, is explicit prior matching in embedding space, used by LeWorldModel: a KL divergence pushes the marginal distribution of embeddings toward a fixed prior (typically Gaussian), which subsumes both the variance and decorrelation roles of VICReg in a single term with a single coefficient.

One clear engineering trend is a reduction in collapse-prevention complexity: later methods increasingly replace large sets of stabilization tricks with smaller, more interpretable regularization recipes. SimCLR needed a temperature plus a batch-size threshold; BYOL needed an EMA plus a predictor; VICReg needed three coefficients; LeWorldModel needs one. Whether this simplification continues to hold at larger scales is an open empirical question, but the direction is encouraging.

8. Action-Conditioned JEPAs and the Bridge to Control

The vanilla JEPA framework predicts the embedding of a target from the embedding of a context with no notion of agency. To use a JEPA as a world model for control, the predictor must be conditioned on an action variable a that represents the intervention applied between context and target. Action-conditioned JEPAs were prefigured by the “Learning and Leveraging World Models” paper (Bardes et al., 2024), which trained the predictor to apply a learned transformation parameterized by a discrete or continuous action to the context embedding, and recover the target embedding produced by the same encoder applied to the post-action observation. V-JEPA 2’s robot variant followed the same template at scale, and LeWorldModel made action conditioning a first-class part of the architecture from the start. Once the model is action-conditioned, it can be plugged into a planner: at test time, the agent samples or optimizes a sequence of actions, rolls them out through the latent dynamics model to obtain a trajectory of latent states, and scores each rollout with a learned or hand-specified reward function. Cross-entropy method (CEM), model-predictive path integral (MPPI), and gradient-based trajectory optimization all work in this setting.

The key advantage over reconstruction-based world models is that the planner never needs to decode anything to pixels. A 30-step rollout of CEM with 1000 candidate trajectories costs 30000 forward passes through the latent dynamics model, which for LeWorldModel’s 15M-parameter model can be orders of magnitude cheaper than rolling out a pixel-space generative model, because the planner never has to invoke a high-resolution decoder. This is what makes the 48x planning speedup possible. The disadvantage is that the planner cannot use any observation-space inductive bias (penalize visually implausible images, encourage smooth motion in pixel space), because it has no observation-space output. In practice, learned reward models in latent space replace those biases.

9. Latent World Models in RL: PlaNet, Dreamer, IRIS, MuZero, TD-MPC

Latent world models have a longer history in reinforcement learning than in representation learning. Ha and Schmidhuber’s “World Models” (2018) trained a VAE encoder, an MDN-RNN dynamics model, and a small controller in a learned latent space, and showed that policies trained entirely inside the dream could transfer to the real environment. Hafner’s PlaNet (2018), Dreamer, DreamerV2, and DreamerV3 (2023) refined this into a robust recipe: a recurrent state-space model (RSSM) with stochastic and deterministic components, trained with a reconstruction loss on observations, a reward prediction loss, and a KL term between prior and posterior. The RSSM splits the latent into a deterministic recurrent state h_t and a stochastic state z_t, with the forward dynamics:

DreamerV3 in particular is notable for achieving strong performance across environments such as Crafter, Atari, DMLab, Minecraft, and continuous control with a single set of hyperparameters, an unusual achievement in deep RL, and for popularizing the symlog transform symlog(x) = sign(x) log(1 + |x|) for stabilizing reward and value learning across orders of magnitude. It also showed that an RSSM trained from scratch could solve Minecraft Diamond from raw pixels, a milestone that earlier model-free methods had failed to reach.

The Dreamer family, however, still pays the pixel-prediction tax: its latent is shaped largely by the requirement to reconstruct observations through the decoder, and ablations consistently show that removing the decoder degrades performance because nothing else organizes the representation. IRIS (Micheli et al., 2023) took a different path, discretizing observations into tokens via a VQ-VAE and modeling the latent dynamics with an autoregressive transformer, achieving state-of-the-art sample efficiency on Atari 100k. MuZero (Schrittwieser et al., 2019) trained a latent dynamics model whose only constraint was that rollouts had to predict reward, value, and policy correctly; it never reconstructed observations, and its latents were shaped purely by the demands of planning. MuZero can be seen as a conceptual precursor to decoder-free latent approaches, even though it was framed as a model-based RL method rather than a representation-learning one.

TD-MPC (Hansen et al., 2022) and TD-MPC2 (2023) take the next step: the encoder is trained jointly with a latent dynamics model and a value function, with no reconstruction loss at all. The objective minimizes a weighted sum of latent consistency (the next-state embedding should match what the dynamics model predicts), reward prediction, and value prediction, all in latent space. Empirically, TD-MPC2 matches or exceeds Dreamer on a wide range of continuous control benchmarks (DMControl, MetaWorld, ManiSkill) with substantially smaller models, and its features generalize better to held-out tasks because the latent is not forced to encode reconstructable noise. LeWorldModel can be read as the next step in this trajectory: removing the reward and value heads as well, training the latent dynamics from passive observation alone, and recovering control performance through model-predictive planning at test time. It is worth noting the pattern: three lines of work, started independently in self-supervised image learning, model-based RL, and offline robot learning, are arriving at architecturally similar designs and similar arguments about what should and should not appear in the loss function. Whether this constitutes genuine convergence on a single paradigm or parallel exploration of a shared design region remains to be seen.

10. Latent Reasoning in Language: Coconut, Recurrent Depth, and Quiet-STaR

The same critique that motivates JEPAs in vision applies, with a different accent, to chain-of-thought reasoning in language models. Standard CoT forces the model to commit to a discrete token at every reasoning step, even when the underlying state is uncertain or branching. Hao et al.’s Coconut, “Training Large Language Models to Reason in a Continuous Latent Space” (arXiv 2412.06769, 2024), removes the commitment. After a problem statement is consumed, the model enters a “thinking mode” in which the last hidden state, instead of being projected to a token through the vocabulary head, is fed back as the next input embedding. The reasoning trajectory unfolds entirely in continuous space for a fixed or learned number of steps, after which the model exits thinking mode and emits a normal token sequence as the answer. Training uses a curriculum that starts from standard CoT data and progressively replaces text reasoning steps with continuous ones, supervising the final answer with cross-entropy and letting gradient flow through the latent steps shape the intermediate representations.

Coconut provides evidence that some multi-step reasoning tasks benefit from latent continuous reasoning trajectories rather than fully tokenized intermediate chains. On logical reasoning benchmarks like ProntoQA and ProsQA, where solving the problem requires search over a space of possible deductions, Coconut shows improvements over standard CoT on several structured reasoning benchmarks at comparable parameter counts, and uses substantially fewer tokens of compute at inference time. The interpretive payoff is that probing the continuous thoughts shows them encoding multiple alternative next steps simultaneously, in effect performing a soft breadth-first search rather than committing to a single deterministic path. Follow-up work by Geiping et al. on “latent reasoning by recurrent depth” (2025) explored a related direction in which the model iterates a transformer block on a latent state for a variable number of steps controlled by an internal halting mechanism, achieving CoT-like benefits without producing intermediate tokens at all. Quiet-STaR (Zelikman et al., 2024) sits in between: it generates short rationales in token space at every position but trains the model to internalize the reasoning so that, at inference time, the rationales become unnecessary.

All three lines treat tokens as a constraint to be relaxed rather than a substrate for reasoning, mirroring the JEPA argument that pixels are a substrate for perception, not for representation. The shared hypothesis is that committing to discrete symbols at every intermediate step may be suboptimal for tasks requiring branching search, and that gradient-based optimization through continuous representations can capture some of what discrete chain-of-thought linearizes away. This remains an active hypothesis, well-supported on specific reasoning benchmarks but not yet demonstrated at the scale or breadth that would justify a general claim about CoT being superseded.

11. Concept-Level Models: Large Concept Models and the Sentence Embedding Substrate

Meta’s Large Concept Model line, introduced in late 2024 and extended through 2025 with Dynamic LCM (arXiv 2512.24617), pushes the abstraction one level higher. Instead of operating on tokens or on continuous thoughts derived from token-level hidden states, an LCM operates directly on sentence embeddings produced by a frozen multilingual encoder (originally SONAR). A document is segmented into sentences, each sentence is encoded into a fixed-dimensional vector (1024 dimensions in the original LCM), and the model learns to predict the next sentence embedding conditioned on the previous ones. The original LCM tried two formulations: a deterministic MSE regression and a diffusion-based generative model in embedding space. The diffusion variant won by a large margin, because predicting a single point estimate of the next sentence ignored the inherent multi-modality of “what could be said next.” At inference time, predicted embeddings are decoded back to text by a separate decoder (the SONAR decoder), but reasoning, planning, and summarization happen at the level of sentence-shaped concepts.

The motivation is pragmatic: token-level granularity is too fine for long-document reasoning, where the model spends capacity on syntactic glue, while paragraph-level granularity is too coarse. Sentence-shaped concepts hit a sweet spot where each unit carries roughly one self-contained proposition, and a 200-sentence document becomes a 200-step sequence rather than the 8000-token sequence it would be in a token-level model. Because the underlying encoder is multilingual and modality-flexible, the same LCM can ingest a sentence from English text, a sentence from French text, or a transcription of speech, and reason about all of them in a shared embedding space. LCM pushes reasoning farther away from surface-form token prediction than any prior language model, though it does not fully eliminate the surface-level dependency, since the SONAR encoder and decoder are both trained on surface-form data, and the quality of the concept embeddings is bounded by their fidelity. Dynamic LCM further adds an adaptive policy that decides, at each step, how many latent reasoning iterations to run before committing to the next concept, blending the LCM substrate with Coconut-style variable-depth latent reasoning. Early experiments suggest that LCMs are particularly strong on summarization, document-level translation, and long-form question answering, the tasks where token-level autoregression wastes the most compute on local syntactic structure.

12. Beyond Tokens: Byte Latent Transformers and Patch-Level Language Models

A related but distinct line of work attacks the tokenizer rather than the prediction substrate. The Byte Latent Transformer (BLT, Pagnoni et al., Meta, 2024) abandons fixed BPE tokenization entirely and operates on raw bytes, but groups bytes dynamically into patches of variable length based on an entropy estimate of the next byte. Regions of low entropy (predictable substrings, common words) are merged into long patches, while regions of high entropy (rare words, code identifiers, numbers) get short patches. A small local encoder maps each patch to a latent embedding, a large global transformer operates on the patch embeddings, and a small local decoder maps the predicted patch embeddings back to bytes. The architecture spends compute proportional to information density, which is more efficient than fixed tokenization on average and dramatically more robust on multilingual and code-heavy data where BPE behaves badly.

BLT shares the spirit of moving the computational substrate to a more abstract level: the global transformer never sees bytes directly and operates on learned representations shaped by next-patch prediction. However, BLT remains an autoregressive language model at its core, predicting the next patch rather than reasoning in a decoder-free embedding space the way JEPAs do. It belongs in this survey because it illustrates the broader dissatisfaction with fixed-granularity tokenization that also motivates Coconut and LCM, but it should be understood as changing the unit of computation rather than changing the fundamental prediction paradigm. The same architectural principle, dynamic entropy-adaptive grouping, could be applied to other modalities (variable-length audio frames, variable-resolution image patches), and several follow-up papers have explored this direction.

13. Multimodal Latent Predictors: VL-JEPA, Audio-JEPA, and Cross-Modal Alignment

The most recent extension of the JEPA framework is VL-JEPA (arXiv 2512.10942, 2025), which applies the predictive joint-embedding objective to vision-language data. Rather than aligning image and text embeddings through a contrastive loss in the style of CLIP, VL-JEPA trains a shared encoder to produce embeddings for image-text pairs and a predictor that reconstructs masked-out portions of one modality from the embedding of the other. Specifically, given an image-caption pair, the model masks either a region of the image or a span of the caption and asks the predictor to recover the masked embedding from the unmasked context, which now spans modalities. The result is a model that learns dense, structured cross-modal representations without the limitations of contrastive learning, namely the need for very large batches, the susceptibility to false negatives in noisy web pairs, and the fundamentally pairwise nature of the alignment signal. VL-JEPA outperforms CLIP-style baselines on fine-grained vision-language tasks (referring expression comprehension on RefCOCO, region-level captioning, dense visual question answering on GQA) at comparable parameter budgets, suggesting that the predictive formulation generalizes cleanly across modalities.

A parallel audio-JEPA line (Baevski et al. and others) applies the same recipe to speech, masking spectrogram tubelets and predicting their embeddings from context, often producing features that match or beat wav2vec 2.0 on speech recognition with substantially less labeled data. The emerging pattern is that JEPAs are becoming a reusable primitive for self-supervised representation learning: any pair of related signals where one can serve as context and the other as target admits a JEPA formulation, and the same regularization tricks (stop-gradient targets, variance-covariance regularization, Gaussian priors) carry over with minimal modification. Whether this primitive will prove as universally applicable as the transformer block has for sequence modeling is too early to say, but the breadth of modalities where it has already been tried (images, video, audio, text, vision-language) is notable.

14. What You Gain and What You Lose

The preceding sections present an appealing set of results, and it is worth being equally explicit about the costs. Moving the prediction objective from observation space to a learned embedding space is not a free lunch; it involves real tradeoffs that practitioners should weigh before adopting these methods.

What you gain. Compute efficiency: decoder-free models are dramatically cheaper to train (I-JEPA reaches strong ImageNet features in a fraction of MAE’s compute) and to use at inference (LeWorldModel plans 48x faster than foundation-model-based world models). Abstraction: the latent can discard visual nuisance variables, yielding representations that are more robust to lighting changes, textures, and other factors irrelevant to the downstream task. Planning speed: rollouts in latent space are orders of magnitude faster than rollouts in pixel space. Generalization: latents shaped by predictive objectives rather than reconstruction tend to transfer better to held-out tasks (demonstrated by TD-MPC2 and V-JEPA).

What you lose. The likelihood training objective: reconstruction-based losses provide a well-understood probabilistic framework (ELBO, log-likelihood bounds) that JEPAs sacrifice entirely. Easy evaluation: you cannot decode a JEPA latent and look at it, which makes debugging, model comparison, and stakeholder communication harder. Interpretability: there is no human-readable “output” to inspect without training auxiliary probes, and those probes are limited to properties the researcher anticipates. Direct generation: if the downstream application requires producing pixels, audio, or text, a JEPA must be paired with a separate generative head. Mature infrastructure: autoregressive models and diffusion models have years of engineering investment in serving, quantization, and tooling; the JEPA ecosystem is immature by comparison.

These tradeoffs are not hypothetical. The evaluation difficulty, in particular, is a real impediment: teams accustomed to watching reconstruction quality as a training diagnostic lose that signal with decoder-free models, and the substitute diagnostics (linear probing curves, downstream task sweeps, latent distribution statistics) are more expensive and slower to interpret. Practitioners considering a latent-space model for production should plan for this evaluation gap upfront.

15. From Better Representations to Better Agents: A Gap That Remains Open

A model that learns a good abstract representation of its environment is not automatically a good agent. The gap between useful representation and useful agent deserves explicit attention, because the literature sometimes elides it.

A JEPA-trained encoder that produces features linearly separable for object categories does not guarantee that those features also capture the causal structure needed for multi-step planning. A latent dynamics model that is accurate over single-step transitions can accumulate errors over a 50-step rollout that render the plan useless. A value function trained in latent space may be well-calibrated on the training distribution but fail in novel states that the encoder maps to unfamiliar regions.

The LeWorldModel and V-JEPA 2 results show that the gap can be closed in specific, relatively constrained settings (simple manipulation, structured environments with limited object diversity). TD-MPC2 shows it can be closed in standard RL benchmarks. But no published JEPA-based system has demonstrated the kind of robust, open-ended autonomous behavior that would justify calling the gap closed in general. The difference between “features that help a downstream planner” and “an agent that autonomously solves novel problems” is the difference between a useful tool and a paradigm shift, and the community is still firmly in the tool stage.

This observation is not a criticism; it is a calibration. Every paradigm starts with useful tools before producing capable agents. But it matters for how we read the claims: when a paper reports that JEPA features improve policy performance by 30 percent on a specific benchmark, that is a representation result, not an agent result, and the two should not be confused.

16. Open Problems: Evaluation, Hierarchies, and the Scaling Question

The latent-space program is not without unresolved problems. Evaluation is the first. Pixel-level world models can be evaluated by reconstruction quality on held-out frames; latent world models cannot, because the embedding space they predict in is itself learned, and a model can achieve arbitrarily low loss by collapsing the embedding. The community has converged on a battery of indirect evaluations (linear probing on downstream tasks, control performance when used as a planner, detection of physically implausible rollouts via held-out negative examples, alignment of learned latents with known causal factors in synthetic environments), but none of these is as cheap or as interpretable as a reconstruction metric, and comparing methods across papers is harder as a result. A subtle but important consequence is that JEPA papers tend to report many small benchmarks rather than one headline number, and reviewers have to assess whether the chosen evaluations are representative.

The second open problem is hierarchy. LeCun’s original vision called for hierarchical JEPAs that predict at multiple time scales simultaneously, with high-level latents capturing slow events (a person crossing a street) and low-level latents capturing fast ones (the swing of a leg). To date, no published JEPA architecture has cleanly demonstrated multi-scale hierarchical prediction at scale; V-JEPA 2 and LeWorldModel both operate at a single time scale, and stitching them into multi-resolution predictors remains an active area of research. The challenge is that the higher levels of the hierarchy need to be supervised by something, and absent a reconstruction loss, the natural supervisory signal at the top of the hierarchy is unclear. Several recent papers have proposed using contrastive or predictive losses across slow and fast time scales jointly, but none has yet shown the clean hierarchical specialization that the framework promises.

The third is the scaling question. Decoder-free latent objectives have shown excellent compute efficiency at small and medium scale, but the field has not yet seen a definitive demonstration that they scale as cleanly as autoregressive token prediction does in language. The bet implicit in the JEPA program is that they will, and that the abstraction gain compounds rather than saturates, but the empirical evidence is still being assembled. V-JEPA 2 at 1.2B parameters is the largest published JEPA, and there is no equivalent of the GPT-3 to GPT-4 scaling sweep for the latent-prediction family. Until that sweep happens, claims about latent prediction being a fundamentally better paradigm are extrapolations from a limited operating regime.

A fourth problem, less discussed but increasingly visible, is interpretability. A reconstruction-based latent has a natural interpretive handle: decode it and look at the result. A JEPA latent has no such handle. Probing methods (training small linear classifiers to predict known properties from the embedding) work but are limited to properties the researcher anticipates. As JEPAs move from research benchmarks to deployed systems (robot policies, planning components), the inability to inspect their internal state in human-readable form will become a real obstacle, and the community has not yet produced a satisfying answer.

Finally, the relationship between latent reasoning models in language (Coconut, recurrent depth, LCM) and latent world models in vision (JEPA, TD-MPC, LeWM) is conceptually suggestive but technically thin: they share an argument about prediction substrates but do not yet share architectures or training infrastructure. A unified model that reasons in a latent space derived from passive video and grounded language is an obvious aspiration, and several groups are openly working toward it, but no published system has achieved it.

17. Comparison of Latent, Reconstruction, and Abstract-Prediction Model Families

18. Practical Recipe: When to Reach for a Latent-Space Model

For practitioners deciding whether the latent-space toolbox is the right fit, the decision rule that emerges from the literature is reasonably clear. Reach for a JEPA-family model when (a) the downstream task is perception, control, or planning, not synthesis; (b) the input modality is high-dimensional and rich in nuisance variables (video, multi-camera robot streams, audio); c) labeled data is scarce but unlabeled data is abundant; and (d) inference latency matters, because decoder-free models are dramatically cheaper at test time. Reach for a latent diffusion model when the downstream task is generation and visual or auditory fidelity is a hard requirement. Reach for Coconut-style latent reasoning when the language task requires multi-step search and the cost of producing long CoT traces is prohibitive. Reach for an LCM-style concept model when the task is long-document understanding, summarization, or cross-lingual reasoning where token-level autoregression is wasteful. Reach for BLT when the task is multilingual or code-heavy and BPE artifacts are visibly degrading quality. Conversely, latent-space models are often a poor fit when the downstream task requires high-fidelity generation, direct likelihood estimation, or tight coupling to human-interpretable outputs, where reconstruction-based or autoregressive objectives remain more appropriate.

Once you decide to build a JEPA, the core architectural choices are now reasonably well-established. Use a ViT or video ViT context encoder. Use an EMA target encoder with decay around 0.996–0.999, increasing over training. Use a small predictor (5–10 percent of the encoder size) with coordinate or action conditioning. Mask large semantic blocks rather than scattered patches. Apply some form of embedding distribution regularization: Gaussian prior (LeWM), variance-covariance (VICReg), or DINO-style centering and sharpening. Train with AdamW, cosine schedule, and a warmup phase long enough to let the EMA target stabilize. Do not add a reconstruction loss as a “safety net”; ablations consistently show it hurts the final representation quality even when it makes early training look smoother.

Three operational concerns deserve special attention once a latent-space model leaves the lab. First, calibration of latent predictions: when the predictor’s output is fed into a downstream planner or value function, miscalibrated latent confidences can produce overconfident plans that fail catastrophically; calibration probes (matching predicted latent distributions to held-out empirical distributions) should be part of the standard evaluation suite. Second, training-serving skew: special care is needed to avoid drift where the latent representations seen during training differ from those encountered at inference time, which can occur when the encoder is updated without retraining the predictor, when input preprocessing differs subtly between offline and online pipelines, or when the EMA target stops being refreshed in deployment. Third, slice-based monitoring: aggregate metrics hide failure modes, so evaluating performance across slices (environment types, object categories, action regimes, language families for LCM, modalities for VL-JEPA) is essential to catch regressions invisible to global benchmarks.

19. Outlook: Multiple Lines of Work, One Shared Intuition

What loosely ties several of these lines of work together is a shared methodological preference: place the predictive burden in a learned internal space rather than directly in observation space. That preference is uncomfortable because it removes the safety net of pixel- or token-level supervision, and most of the engineering effort of the past four years has gone into building tools (asymmetric architectures, EMA targets, variance-covariance regularization, Gaussian priors, curriculum-based latent reasoning) that make training in this regime stable. The payoff, when it works, is models that are smaller, faster to train, more robust to irrelevant variation, and more directly useful as the substrate for planning, control, and reasoning. The recent papers, LeWorldModel for control, Coconut for reasoning, LCM for long-form language, BLT for tokenizer-free modeling, VL-JEPA for multimodal understanding, push in a broadly similar direction from different starting points.

Whether this constitutes the beginning of a paradigm shift or a productive but bounded set of improvements within the existing paradigm is too early to say. The scaling evidence is still thin: the largest published JEPA is 1.2B parameters, and the largest latent reasoning model is far smaller than frontier autoregressive LLMs. The hierarchical JEPA vision remains aspirational. The gap between good representations and capable agents has been narrowed in specific settings but not closed in general. And the autoregressive token-prediction recipe that these models aim to improve upon has its own momentum: it has absorbed enormous engineering investment and continues to produce strong results at every new scale.

The honest assessment in mid-2026 is this: multiple independent research groups are exploring prediction in abstract embedding spaces as an alternative to, and in some settings a partial replacement for, reconstruction or token-level generation, especially in settings where planning, control, or long-horizon reasoning matter more than surface fidelity. The results so far are promising and consistent enough to justify sustained investment, but they do not yet constitute a demonstrated replacement for the dominant paradigm at scale. The intermediate position, increasingly visible in industry, is hybrid: a JEPA-style abstract backbone for perception and planning, paired with a latent-diffusion-style generative head for the rare moments when a human needs to inspect what the model is thinking, and a Coconut-style continuous reasoning loop for problems that require search. None of these components is mature enough to deploy without supervision, but together they sketch a credible alternative research direction to the “scale up next-token prediction” recipe that has dominated the last five years. For ML systems builders, the practical lesson is narrower but immediate: when you design a self-supervised pretext task for a new modality, ask whether the loss really needs to live at the level of the raw observations, or whether predicting an embedding of the target would let you cut the model in half and double the downstream transfer. In an increasing number of settings, the answer appears to be the latter.