World Model Research On Your MacBook

World Model Research On Your MacBook

Local world-model research platform

pseudo-marble is a small, open research platform for testing one of the central ideas in current AI: that appearance, physics, and action are not separate capabilities but different views of a single underlying understanding of the world. The framing comes from Fei-Fei Li’s functional taxonomy of world models, which groups such systems by what they produce along the perception — action loop. Renderers output pixels, simulators output physical state like mass and friction, and planners output actions, with the planner described as the inverse of the renderer. Li argues these are really three projections of one representation, and that recent systems such as World Labs’ Marble, which generates both a visual model and a collision mesh from a single network, are converging toward it.

The catch is that producing two outputs from one model only shows the outputs can be packaged together. It does not show they come from a genuinely shared understanding rather than two separate predictors attached to a common trunk. pseudo-marble exists to test that specific distinction at a scale an individual can actually run, on an ordinary Apple-silicon laptop using free tools.

The model takes several views of an object and encodes them into one internal representation. From that single representation it predicts three things in parallel: how the object behaves when dropped, tilted, or pushed; its physical properties; and its appearance. The training data is synthetic but principled. Each material is drawn from a hidden continuous “essence” that shapes both look and behavior with added noise, so the system has to infer physics from appearance instead of memorizing labels, and it is tested on material types held out from training.

The key tool is a coherence measure. Nudging the internal representation and checking whether appearance and behavior shift together reveals whether the model formed one understanding or merely glued two guesses side by side. Early in-sandbox work uncovered an important subtlety: an untrained network already looks coherent simply because both predictions read the same representation, so the honest signal is how much training raises coherence above that baseline. The platform is built to give a clear answer either way, including a negative one.

Source code: https://github.com/sbayer2/Project-Free-Robot-Agent-World

The model

A single shared-latent network: multi-view images → encoder → z → three projections

• behavior head (21-dim drop/tilt/push outcomes — the real target)

• essence head (density/friction/restitution — aux supervision)

• render head (conv decoder → mean-view image — appearance)

One architecture, three mirrored backends from one config: mlx_net (canonical, Mac/Metal), numpy_net (forward-only), torch_net (CPU, trainable). Fed by continuous materials (hidden essence → appearance + physics, noisy) and drop/tilt/push probes, with held-out essence regions for generalization. The payoff measurement: coherence — does nudging z move appearance and behavior together?

Initial sandbox findings (CPU, no Mac)

These establish the apparatus works — not the science.

1. No usable Linux MLX (broken wheel, no GPU) → numpy/torch backends fill the gap.

2. Authored coupling is predictive-but-noisy (metallic-look vs log-density, 0.3 < r < 0.99) — learnable, not a lookup.

3. Full model trains end-to-end — overfit loss drops ~4.8×, so gradients flow through all heads incl. the decoder.

4. Scale ceiling: comfortable to ~1M params / 128px / 16 views (~0.85 s/step); memory never the bound (1.3 of 15 GB); compute-time bound past 256px.

5. Caught a metric bug — unit-sphere sampling made disjoint subspaces score negative; fixed to Gaussian directions.

6. ⭐ The big one: an untrained shared model already scores ~0.45 coherence (independent baseline ~0.008) — purely architectural. So the naive “shared beats independent” comparison would be a false positive. The honest signal is learned_coherence = trained_shared − untrained_shared, averaged over seeds (baseline wobbles 0.36 — 0.46). The harness caught this before any Mac time was spent.

What’s still unknown (the honest gaps)

• No training on real renders — every result so far is overfitting, proving wiring not learning.

• The coupling is authored — we’d learn the generator’s eigenvector, not reality’s (GSO is the parked escape).

• The headline number — learned_coherence on held-out essences — is unmeasured. That’s the Mac run.

in the figures (all real, CPU-measured, not local Mac runs at this time.)

1. Material coupling — 500 sampled materials, “looks metallic” vs. log-density, r = 0.80. Confirms the authored coupling is predictive but noisy, so the model must infer physics, not memorize it.

2. Training convergence — torch overfit curve, loss 0.461 → 0.106 over 150 steps. Gradients flow cleanly through encoder + all three heads.

3. CPU scale ceiling — step-time vs. parameter count, log scale. The shaded zone is the comfortable region; ~1M params / 128px sits at ~0.6s/step, then 256px jumps to ~1.6s.

4. The coherence control — 8 untrained shared models land in a ~0.36 — 0.55 band, while the independent baseline sits at ~0. This is the visual proof of the key finding: a shared latent looks coherent before any training, so the honest metric is learned = trained − untrained.