After LLMs: why World Models are becoming the next AI architecture?

The AI landscape is entering a new architectural phase. After years of optimizing Large Language Models through scale and next-token prediction, a different direction is emerging: systems that learn internal models of the world.

Google’s Genie 3 and the recent wave of JEPA-based research point in the same direction — toward models that capture state, dynamics, and cause-and-effect, rather than surface-level text patterns.

Why World Models Matter Beyond LLMs

This converts theory into strategy.

The Rise of Non-Autoregressive Reasoning

For technical decision-makers, the signal here is the move away from pure autoregression. The recent “LeJEPA” (Balestriero and LeCun #) publication demonstrated, with mathematical rigor, that pretraining Joint Embedding Predictive Architectures (JEPA) (LeCun #) can be achieved without heuristic tricks, specifically by aligning embeddings with isotropic Gaussians. In the meantime, the newly released VL-JEPA (Vision-Language JEPA) (Chen et al. #) provides a concrete illustration of the benefits of applying JEPA to multimodal tasks.

VL-JEPA: Predicting Representations Instead of Tokens

VL-JEPA (Vision-Language Joint Embedding Predictive Architecture) (Chen et al. #) is a novel approach to integrating visual perception and textual understanding. Unlike standard Multimodal LLMs (MLLMs) that process inputs to autoregressively generate discrete tokens, VL-JEPA operates entirely within a continuous latent space.

This distinction is crucial for engineering leaders to understand:

• Higher-Level Abstraction: By predicting representations rather than pixels or tokens, the model captures semantic meaning (e.g., understanding that “the room is dark” and “the lamp is off” are state-equivalent) without being unduly influenced by surface-level variability.
• One-Shot Generation: It is non-autoregressive and can predict the entire target embedding sequence in a single forward pass.

Figure 1. Comparison of VL-JEPA Architecture with standard JEPA diagram (LeCun #) (Chen et al. #)

Architectural Breakdown

The architecture of VL-JEPA is a masterclass in component reusability and efficiency. It adapts the standard JEPA design (see Figure 1) for multimodal data by integrating specialized encoders:

1. Vision Encoder Enc(Xv): It utilizes a pretrained V-JEPA 2 model. Crucially, this encoder is frozen during training, with only the projection layers being fine-tuned. This leverages V-JEPA 2’s existing understanding of video and physical dynamics.
2. Text Encoder/Latent Z(XQ): Embedding layer that converts text tokens to embeddings.
3. The Predictor: This is where the reasoning occurs. It comprises the top 8 transformer layers of a Llama 3 (1B) model, repurposed to predict the target-text embedding from the visual input. It has removed causal attention and is not autoregressive.
4. Y-Encoder Enc(Y): Initialized with EmbeddingGemma-300M, it generates target embeddings from target text (Y) used during training.

The Mathematics of Alignment: InfoNCE Loss

To ensure that the predicted representations are both meaningful and distinct, VL-JEPA employs the InfoNCE loss function. This objective balances two competing forces:

1. Representation Alignment: Pulling embeddings of positive pairs (matching image-text) closer together.
2. Uniformity Regularization: Pushing embeddings of negative pairs (batch noise) apart to prevent representation collapse.

The loss can be formalized as:

Where SY,i is the target representation, ŜY,i is the prediction, and τ is the temperature parameter. This regularization enables the model to learn a structured world model without requiring pixel-perfect reconstruction.

What This Means for Enterprise AI Systems

From an engineering perspective, architectures like VL-JEPA are less about research novelty and more about how production systems can become faster, more stable, and less prone to hallucination.

In enterprise settings, this shift toward embedding-based prediction changes how systems are composed, scaled, and evaluated in practice.

Works Cited

1. Balestriero, Randal, and Yann LeCun. “Lejepa: Provable and scalable self-supervised learning without the heuristics.” arxiv, vol. 2511, no. 08544, 2025, https://arxiv.org/pdf/2511.08544.
2. Chen, Delong, et al. “Vl-jepa: Joint embedding predictive architecture for vision-language.” arxiv, vol. 2512, no. 10942, 2025, https://arxiv.org/pdf/2512.10942.
3. LeCun, Yann. “A path towards autonomous machine intelligence version 0.9. 2, 2022–06–27.” Open Review, vol. 62, no. 1, 2022, pp. 1–62, https://openreview.net/pdf?id=BZ5a1r-kVsf

Michał Kulczykowski, Senior Machine Learning Engineer at deepsense.ai