Understanding JEPA: From Latent Prediction to Video World Models

Figure 1: A clean schematic of the JEPA template (context encoder, target encoder, predictor) with the stop-gradient symbol on the target side and the EMA arrow from context to target. Source [6]

Self-supervised learning in vision has been dominated by two broad families. On one hand, generative methods such as Masked Autoencoders [1] reconstruct missing pixels. On the other side, contrastive methods like DINO and SimCLR pull augmented views of the same image close in feature space and push different images apart. Both families produce strong representations, but they share a common limitation: They spend modelling capacity on details that may not matter for downstream tasks.

Pixel-level reconstruction forces the encoder to remember low-level texture, lighting, and noise. Contrastive learning depends heavily on hand-crafted augmentations and on negative sampling strategies. LeCun argues in his 2022 position paper [2] that neither captures how humans and animals build internal models of the world. The proposed alternative is the Joint-Embedding Predictive Architecture, or JEPA: predict the missing parts of an input directly in the embedding space, not in pixel space.

This post is a practitioner-oriented tour of the JEPA family. We start with the fundamentals and the training tricks that make it work, then we go through I-JEPA, V-JEPA, and V-JEPA 2 in order, look at the broader variant zoo and the recent theoretical work (LeJEPA), and finish with the open problems that I think matter most for video and 3D research.

1. The core concept

A JEPA has three components:

• Context encoder f_θ​ that maps the visible part of the input x into an embedding s_x.
• Target encoder f_ϕ​ that maps the held-out part of the input y into a target embedding s_y.
• Predictor g_ψ​ that takes the context embedding s_x​ and a positional token describing where y lives, and produces a predicted embedding s’_y.

The training objective is a regression loss in the embedding space:

Here, p_y​ is the position information of the target region, and sg(⋅) is the stop-gradient operator. The loss is computed at the token level, so each masked region contributes to the gradient.

The reason this is interesting is that the encoder is free to throw away information that is unpredictable or irrelevant. Pixel reconstruction punishes the model for not memorizing every detail. JEPA does not, if a piece of detail is not useful for the prediction task in the latent space, the encoder can drop it. This is also the energy-based interpretation [3]: the model assigns low energy when the predicted and target embeddings agree, and high energy otherwise.

Figure 2: simple data-flow diagram with three boxes (context encoder, predictor, target encoder), with stop-gradient and EMA annotations. Source: Figure 2 of the I-JEPA paper [6]

2. The collapse problem

The core training danger of JEPA is well known: if both the context encoder and the target encoder are trained jointly to make the loss small, the trivial solution is for both to map every input to the same constant. Loss is zero, representations are useless. This failure mode is called representation collapse.

The standard fix has three pieces:

• Stop-gradient on the target encoder. The target embeddings are treated as fixed regression targets within a batch.
• EMA update of the target encoder. Instead of training it directly, the target encoder weights are an exponential moving average of the context encoder weights:

where the momentum coefficient mm m is close to 1 (often increased on a schedule, for example, from 0.996 to 1.0).

• Asymmetric predictor. The predictor g_ψ​ is intentionally smaller and less expressive than the encoders. If the predictor were too powerful, it could memorize trivial mappings, and the encoders would not need to learn anything useful.

This combination is essentially the BYOL self-distillation trick [4]. One JEPA training step in PyTorch pseudocode:

def jepa_step(x, y, pos_y,
              context_enc, target_enc, predictor,
              opt, m=0.996):
    # ----- Student (context) branch -----
    s_x = context_enc(x)                       # (B, N_ctx, D)
    s_y_pred = predictor(s_x, pos_y)           # (B, N_tgt, D)

    # ----- Teacher (target) branch — no gradients -----
    with torch.no_grad():
        s_y = target_enc(y)                    # (B, N_tgt, D)

    # JEPA regression loss in embedding space
    loss = F.smooth_l1_loss(s_y_pred, s_y)

    # Update student via gradient descent
    opt.zero_grad()
    loss.backward()
    opt.step()

    # Update teacher via EMA from student
    with torch.no_grad():
        for p_t, p_s in zip(target_enc.parameters(),
                            context_enc.parameters()):
            p_t.data.mul_(m).add_(p_s.data, alpha=1 - m)

    return loss.item()

It is important to mention that this combination is not a guarantee. Recent analysis [5] shows that even with EMA and stop-gradient, JEPA models can still fall into dimension-wise collapse if the masking strategy or predictor capacity is poorly chosen. Some variants add VICReg-style variance and covariance regularizers as a safety net. We will return to this point at the end.

3. I-JEPA — the first vision instantiation

I-JEPA [6] is the first concrete vision system in this family, released by FAIR in 2023. The recipe:

• Both encoders are Vision Transformers operating on 16×16 patches.
• The predictor is a narrower ViT, roughly the same depth, but with a smaller hidden dimension.
• The masking strategy is multi-block: sample 4 target blocks, each covering 15–20% of the image area, plus one large context block covering 85–100% of the image with the target overlaps removed.
• The target embeddings are extracted from the full image by the target encoder, then patches inside each target region are pooled.

The key insight is that the target blocks are semantic-scale, not single patches. Predicting one isolated patch from its neighbours mostly tests local texture continuity. Predicting a 15% region forces the model to reason about object structure.

Figure 3: The multi-block masking visualization context block in one color, target blocks in different colors. Source: Figure 4 of the I-JEPA paper [6] (page 4)

PyTorch pseudocode for one I-JEPA forward pass:

def i_jepa_forward(image,
                   context_encoder, target_encoder, predictor):
    # Patchify the image into N tokens
    patches = patchify(image)                          # (B, N, D)

    # Sample one context block and several target blocks
    ctx_idx, tgt_idx_list = sample_multi_block_masks(N)

    # ----- Target branch (full image, no grad) -----
    with torch.no_grad():
        all_tokens = target_encoder(patches)           # (B, N, D)
        target_embeds = [all_tokens[:, idx, :]
                         for idx in tgt_idx_list]

    # ----- Context branch (visible patches only) -----
    ctx_tokens = context_encoder(patches[:, ctx_idx])  # (B, N_ctx, D)

    # ----- Predict each target block from context + position -----
    losses = []
    for tgt_idx, tgt_emb in zip(tgt_idx_list, target_embeds):
        pos_tokens = positional_tokens(tgt_idx)        # (B, N_tgt, D)
        pred = predictor(ctx_tokens, pos_tokens)       # (B, N_tgt, D)
        losses.append(F.smooth_l1_loss(pred, tgt_emb))

    return torch.stack(losses).mean()

On ImageNet linear probing, I-JEPA with ViT-H/14 reaches strong numbers without any data augmentation beyond cropping. Compared with MAE under the same compute budget, I-JEPA is roughly 2–3× more efficient on a wall-clock basis. The reason is intuitive: predicting in latent space is cheaper than reconstructing 16×16×3 pixels per patch.

4. V-JEPA — extension to video

V-JEPA [7] takes the same template to video. A clip is split into spatiotemporal patches (a 3D grid of spatial × spatial × temporal). The encoder processes this as one long sequence of tokens.

The masking strategy is the most important change. Earlier video models like VideoMAE used tube masking: the same spatial region is masked across all frames in the clip. This is a strong shortcut if the model can fill in one frame; the same fill works for all frames, because the underlying scene rarely changes that fast. V-JEPA uses 3D block masking: random rectangular blocks in the spatiotemporal volume, with limited temporal extent. This forces the model to reason about motion and scene dynamics, not only static appearance.

Figure 4: Side-by-side comparison: tube masking (the same square hole across T frames, as in VideoMAE) vs 3D block masking (V-JEPA style). Source: V-JEPA paper [7]

V-JEPA shows strong performance on motion-heavy benchmarks such as Something-Something v2 and on action-anticipation tasks such as Epic-Kitchens-100. The model is action-free at this stage. It learns what tends to happen in videos, but it does not condition on agent actions. This last point becomes the main motivation for V-JEPA 2.

5. V-JEPA 2 — a world model on robots

V-JEPA 2 [8], released by Meta in June 2025, is where the architecture starts to look like a real-world model. The recipe has two stages.

Stage 1: Action-free pretraining. V-JEPA 2 is trained on more than 1M hours of internet video plus 1M images. The objective is the standard JEPA loss, predicting latent representations of masked spatiotemporal regions. After this stage, V-JEPA 2 reaches state-of-the-art on Epic-Kitchens-100 action anticipation with an attentive probe, and, when aligned with an LLM at the 8B parameter scale, achieves 84.0 on PerceptionTest and 76.9 on TempCompass for video question answering.

Stage 2: Action-conditioned post-training (V-JEPA 2-AC). Freeze the encoder. Train a new predictor on less than 62 hours of unlabeled robot interaction videos from the Droid dataset. The predictor now takes the current latent state, a proprioceptive signal, and an action vector to predict the next latent state. Two losses are used:

• Teacher-forcing loss for next-step prediction.
• Two-step rollout loss that enforces consistency over multiple prediction steps, so that errors do not compound during planning.

The deployment result that makes this paper interesting: V-JEPA 2-AC is run zero-shot on Franka Emika Panda arms in two labs that do not appear in the Droid training data. Goals are given as image targets. At inference time, the system runs a model-predictive control loop with cross-entropy method (CEM) action sampling:

def vjepa2_ac_planning(current_obs, goal_image,
                       encoder, predictor,
                       action_horizon=4,
                       n_samples=512, n_iters=3, top_k=64):
    """
    CEM-style action sampling with the V-JEPA 2-AC world model.
    Latent rollouts replace expensive pixel-space prediction.
    """
    z_current = encoder(current_obs)         # current latent
    z_goal    = encoder(goal_image)          # goal latent

    # Initialize action distribution over the planning horizon
    mu  = torch.zeros(action_horizon, action_dim)
    std = torch.ones(action_horizon, action_dim)

    for it in range(n_iters):
        # 1) Sample candidate action sequences
        actions = sample_actions(mu, std, n_samples)   # (N, H, A)

        # 2) Roll out the world model in latent space
        z = z_current.unsqueeze(0).expand(n_samples, -1)
        for t in range(action_horizon):
            z = predictor(z, actions[:, t], proprio_t)  # next latent

        # 3) Score by distance to goal latent
        scores = -((z - z_goal) ** 2).sum(dim=-1)

        # 4) Refit the action distribution to the top-k elites
        elite_idx = scores.topk(k=top_k).indices
        mu  = actions[elite_idx].mean(dim=0)
        std = actions[elite_idx].std(dim=0)

    # Execute the first action, replan at the next step (MPC)
    return mu[0]

Figure 5: The Franka arm robotics setup, or the MPC planning loop schematic, showing latent-space rollouts. Source: V-JEPA 2 paper [8]

Reported success rates on pick-and-place with novel objects in novel environments are in the 65–80% range. The system is also reported to be roughly 30× faster at planning than a Cosmos-based [9] alternative. The reason is direct: planning happens in a 1024-dimensional latent space, not in pixel space. You do not have to decode anything to evaluate a candidate trajectory.

This is the part I find most relevant from a video research perspective. The community has spent the last two years scaling diffusion-based video models for generation. V-JEPA 2 makes an explicit case that for prediction and planning, generation is wasted compute. You do not need to render the future; you only need to predict its representation.

6. The variant zoo

The JEPA template has been ported to many modalities. A short tour of what exists:

• A-JEPA [10] applies the recipe to mel-spectrograms with a curriculum masking strategy. The interesting empirical finding is that random unstructured patch masking works better than block masking for audio, because correlations in time–frequency are more local than in natural images.
• MC-JEPA [11] trains a single encoder to predict both motion (optical flow) and content (image features) at the same time, using a multi-task objective.
• Point-JEPA [12] and 3D-JEPA [13] adapt the architecture to point clouds. A “patch” in this setting is a small group of points; the predictor must reason about 3D geometry rather than 2D appearance. This is a direction worth tracking for anyone working at the intersection of self-supervised learning and 3D reconstruction.
• Graph-JEPA uses subgraphs as patches, with hyperbolic projections of the targets.
• VL-JEPA [14] (December 2025) is more radical. Instead of autoregressively decoding text tokens like a standard VLM, it predicts the continuous embeddings of the target text. A lightweight decoder is invoked only when actual text output is needed. The reported result is a stronger performance with 50% fewer trainable parameters compared to a token-space VLM under the same vision encoder and training data.

The most important recent development, in my view, is LeJEPA [15] (Balestriero & LeCun, November 2025). The paper questions whether all the heuristics, EMA, stop-gradient, asymmetric predictor, and VICReg regularizers are even necessary. The authors prove that the optimal embedding distribution for downstream prediction risk is an isotropic Gaussian. They then introduce Sketched Isotropic Gaussian Regularization (SIGReg), which uses random one-dimensional projections of the embeddings and a univariate normality test (based on the Cramér-Wold principle) to enforce this distribution in linear time and memory.

The full LeJEPA objective is a JEPA prediction loss plus SIGReg, fitting in around 50 lines of code. No EMA, no stop-gradient, no asymmetric predictor. Reported empirical results are competitive on standard benchmarks. If this scales, the entire collapse-prevention machinery built up over the last three years can be replaced by a single regularizer with theoretical justification.

7. Open problems and critique

JEPA is a promising direction, but there are real concerns worth knowing before building on top of it.

(a) The JEPA loss is a poor proxy for representation quality. The training loss measures how well the predictor matches the target encoder, but the target encoder is itself moving via EMA. Loss can be low while representations are degenerate. Practitioners rely on downstream linear probing or attentive probing as the real signal of progress. This makes hyperparameter search expensive, because each candidate has to be evaluated end-to-end on a probe task.

(b) The training recipe is brittle. Mask scale, mask count, EMA schedule, predictor depth, and target normalization all matter, and they interact. The literature [16] reports multiple failure modes (entire collapse, dimension collapse, mean-learning deficiency) that are hard to detect from the loss curve alone. The LeJEPA work is, in part, a response to this brittleness.

(c) Scaling laws are unclear. Generative video models, both diffusion-based and autoregressive, have a well-understood scaling behaviour: more data, more parameters, more compute equals better generation in a fairly predictable way. The JEPA scaling story is less mature. V-JEPA 2 is encouraging at 1B+ parameters and 1M+ hours of video, but we do not yet have a clean curve showing how representation quality scales as we keep growing the model.

(d) Action-conditioning is the bottleneck for world models. V-JEPA 2 separates pre-training (action-free, web-scale) from post-training (action-conditioned, robot-scale). This is pragmatic, but it means the action representation only sees ~62 hours of robot data. The pretraining videos contain many human actions implicitly, but they are not labelled with action vectors. Closing this gap — learning action-aware representations directly from internet video is, in my opinion, the most important open problem in this space.

(e) Comparison with diffusion-based world models is not yet settled. V-JEPA 2 makes a strong efficiency argument against pixel-space world models like Cosmos. But diffusion-based generators carry information that latent predictors do not — they can render plausible futures, which matters for human-in-the-loop interfaces, for sim-to-real transfer with photorealistic rollouts, and for training of downstream policies that take pixels as input. The two approaches may turn out to be complementary rather than competitive. A hybrid architecture — predict latents fast for planning, generate pixels slow for visualization or as a teacher signal — is something I expect to see in the next year.

8. Closing thoughts

JEPA started as a bet that prediction in latent space is the right primitive for self-supervised learning. Three years in, the picture is concrete: I-JEPA showed the recipe works for images, V-JEPA showed it transfers to video, V-JEPA 2 showed it can drive a real robot zero-shot, and LeJEPA showed the theory may finally be catching up with the practice.

For practitioners working on video generation or 3D representation learning, JEPA is worth tracking for two reasons. First, predictive latent models are becoming a serious alternative to generative world models for planning workloads. Second, the masking and target design choices that make JEPA work are themselves general lessons for designing self-supervised objective lessons that often transfer back to generative training.

The next milestones I am watching: scaling LeJEPA-style theory to video, better action-conditioning recipes that do not need a separate post-training stage, and hybrid world models that combine latent prediction with diffusion generation.

References

[1] He, K., Chen, X., Xie, S., Li, Y., Dollár, P., & Girshick, R. (2021). Masked Autoencoders Are Scalable Vision Learners. arXiv:2111.06377.

[2] LeCun, Y. (2022). A Path Towards Autonomous Machine Intelligence. OpenReview.

[3] LeCun, Y., Chopra, S., Hadsell, R., Ranzato, M., & Huang, F. J. (2006). A Tutorial on Energy-Based Learning. In Predicting Structured Data, MIT Press.

[4] Grill, J.-B., et al. (2020). Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (BYOL). NeurIPS. arxiv.org/abs/2006.07733

[5] Sobal, V., et al. (2025). On the Stability of Joint-Embedding Predictive Architectures. TMLR. arxiv.org/abs/2211.10831

[6] Assran, M., et al. (2023). Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture (I-JEPA). arXiv:2301.08243.

[7] Bardes, A., et al. (2024). Revisiting Feature Prediction for Learning Visual Representations from Video (V-JEPA). arXiv:2404.08471.

[8] Assran, M., et al. (2025). V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning. arXiv:2506.09985.

[9] Agarwal, N., et al. (2025). Cosmos World Foundation Model Platform for Physical AI. NVIDIA Technical Report.

[10] Fei, Z., Fan, M., & Huang, J. (2023). A-JEPA: Joint-Embedding Predictive Architecture Can Listen. arXiv:2311.15830.

[11] Bardes, A., Ponce, J., & LeCun, Y. (2023). MC-JEPA: A Joint-Embedding Predictive Architecture for Self-Supervised Learning of Motion and Content Features. arXiv:2307.12698.

[12] Saito, A., et al. (2024). Point-JEPA: A Joint Embedding Predictive Architecture for Self-Supervised Learning on Point Cloud. arXiv:2404.16432.

[13] Hu, N., et al. (2024). 3D-JEPA: A Joint Embedding Predictive Architecture for 3D Self-Supervised Representation Learning. arXiv:2409.15803.

[14] Chen, D., et al. (2025). VL-JEPA: Joint Embedding Predictive Architecture for Vision-language. arXiv:2512.10942.

[15] Balestriero, R., & LeCun, Y. (2025). LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics. arXiv:2511.08544.

[16] Mo, S., et al. (2024). Connecting Joint-Embedding Predictive Architecture with Contrastive Self-Supervised Learning. https://arxiv.org/abs/2410.19560