World Models: Why I Picked Latent Space Over Pixels

A practical case for structure-based learning in the physical world.

The debate between generative world models and Joint Embedding Architectures (JEPA) is far from settled. Both camps claim impressive results. Pixel-level generative models (like Sora or Genie) can synthesize stunning videos, leading many to argue that if a model can generate the world, it must understand it.

But there is growing evidence that when the goal is semantic understanding — actions, intent, motion, interaction, cause and effect — joint embedding approaches have a structural edge. They learn what matters and discard what doesn’t.

This is not a manifesto against generative models. It’s a practical argument: if your task is understanding rather than synthesis, betting on latent space is the better trade.

[embed]

The Illusion of Scale

A new architecture emerges every few months. LLMs get bigger. Scaling laws hold. GPT‑4, Claude, Gemini, Llama — each generation leapfrogs the last.

I’ve been building systems with LLMs for a while. For language, reasoning, and code generation, they are transformative. But alongside that, I’ve been working on robotics and real‑time computer vision projects. And that’s where the illusion breaks.

LLMs are not useful when a system needs to understand what is happening in images in real time.

They can plan. They can reason abstractly. But the moment a robot must understand what’s in front of it — what moved, what changed, or what action is unfolding — language models hit a wall.

The Pixel Trap

So, you reach for traditional vision pipelines: frame-by-frame analysis, optical flow, reconstruction. And then the real pain begins. You spend half your engineering time fighting phenomena that have nothing to do with the task:

• Shadows & Lighting changes
• Color shifts & White balance
• Motion blur
• Sensor noise

The pixel-level approach fights you at every step. This gets worse in domains like medical imaging (Ultrasound, fMRI) where the signal-to-noise ratio is naturally low. When the signal is weak, wasting capacity on pixel-level detail is not just inefficient — it actively blocks understanding.

JEPA: A Different Bet

Then I found Yann LeCun’s work on JEPA (Joint Embedding Predictive Architecture). It is built on a simple but radical idea:

Don’t predict pixels. Predict representations.

Instead of reconstructing images or video frames, JEPA models learn to predict missing information in latent space — an abstract representation that captures structure, not appearance.

The intuition is deeply human. We don’t predict the exact RGB values of the next thing we’ll see. We predict what will happen — objects, motion, interactions.

The Core Distinction

Here is the difference in one line:

• Pixel prediction asks: “What does this look like?”
• Latent prediction asks: “What is happening here?”

The Rule of Thumb:

• Use Pixels for: Image synthesis, texture generation, fine detail.
• Use Latent for: Noisy data, motion, robotics, classification, anomaly detection.

I Built a Test

Theory is cheap. So I built a real-time action recognition system using V‑JEPA 2.

The Constraints:

1. No fine‑tuning.
2. No classifier training.
3. 10 examples per action.

If the latent space contains real structure, this should work. If not, it should fail completely.

The System

The pipeline is shockingly simple: Camera → 16 frames → V‑JEPA → z vector → cosine similarity → label.

def encode(frames):
    inputs = processor(frames, return_tensors="pt").to("cuda")
    with torch.no_grad():
        out = model(**inputs)
    # Normalize the mean of the last hidden state
    return F.normalize(out.last_hidden_state.mean(dim=1), dim=-1).cpu().numpy().flatten()
def recognize(frames):
    z = encode(frames)
    best_sim, best_label = 0, None

    # Simple nearest neighbor lookup
    for entry in memory:
        sim = np.dot(z, entry['z'])
        if sim > best_sim:
            best_sim, best_label = sim, entry['label']

    return best_label, best_sim

There is no neural classifier. No training loop. Just a nearest neighbour search in latent space using a memory bank of ~10 examples per class (from the Something-Something V2 dataset).

The Results

It didn’t just work; it generalized.

• Speed: Real‑time (≈15 FPS)
• Accuracy: ~90% on test set
• Training: Zero

I trained it on hands pushing random objects. It correctly recognized toy cars approaching, real cars in dashcam footage, and people walking toward the camera.

The most surprising part? The old problems disappeared. Shadows, contrast shifts, and colour changes became irrelevant. The latent space encodes motion and interaction, not illumination.

Why This Matters: Synthesis vs. Understanding

Generative models like Sora or WorldDreamer are incredible, but they solve a different problem.

Decoders are expensive. They must model lighting, texture, noise, blur, and camera artifacts — whether you need them or not. If your goal is understanding, that is wasted compute.

JEPA changes the workflow:

• Traditional: Collect 1,000 labels -> Train Classifier -> Wait -> Deploy.
• JEPA: Encode 10 examples -> Label them -> Done.

You are not training a model. You are simply “naming” regions in a geometric space that already exists.

For medical imaging, this is massive. A new pathology appears? Encode a few confirmed cases, and the system immediately recognizes similar patterns. No retraining required.

World Models in Latent Space

Once everything lives in representation space, the rest becomes natural:

• Prediction: z_next = predictor(z_current, action)
• Planning: actions = find_path(z_current, z_goal)
• Anomaly Detection: if distance(z, memory) > threshold: flag()

No pixels required. World understanding becomes geometry.

Technical Deep Dive: The Implementation

For those interested in the engineering specifics, here is the exact setup I used.

• Model: facebook/vjepa2-vitl-fpc16-256-ssv2 (ViT-Large)
• Input: 256x256 resolution, 16 frames per clip.
• Embedding Strategy: I extracted three overlapping clips from different temporal positions and averaged their normalized embeddings. This improved robustness regarding when the action occurred.
• Vector Size: ~2048 dimensions (Concatenation of encoder spatial features and predictor temporal features).

Recognition Logic

I used a simple Cosine Similarity threshold.

• Object Threshold: 0.92
• Action Threshold: 0.90

If you prefer a classifier, a simple SVM (RBF kernel) trained on the frozen embeddings works instantly, but the raw cosine similarity was sufficient for the demo.

Bottom Line

If your goal is to understand the physical world, betting on pixels is a mistake.

I built a real-time action recognition system with 10 samples, 0 training, and 50 lines of code using Joint Embeddings.

LeCun’s bet is paying off. The decoder path gives you beautiful images. The encoder path gives you a world model. For robotics, medicine, and embodied AI, I’ll take structure.

Resources:

• Model: V‑JEPA 2 (Meta)