NVIDIA Just Gave Robots a Brain That Can See the Future

Cosmos 3 reasons about a scene, dreams up what happens next, and decides how to move. One model does all three.

Most robots live in the present tense. A robot arm can look at a coffee mug and know it is a coffee mug. A self-driving car can spot a pedestrian and label it correctly. That part is mostly solved. The hard part is the next half-second. What is the pedestrian about to do. Where does the mug go if I nudge it. What breaks if I get this wrong. Knowing what something is turns out to be the easy half. Knowing what it is about to do is the whole game.

Image Generated using Chatgpt

If you can’t read the article further, than please click here.

NVIDIA’s answer, announced this week at Computex, is Cosmos 3. And I will be honest about where I started: another NVIDIA launch, another leaderboard sweep, another press release with the word “frontier” in the title. I usually scroll past these. I almost did.

Then I skimmed the technical report. It is 138 pages, and the argument underneath it is genuinely good. That is what changed my mind. Let me walk you through it.

https://research.nvidia.com/labs/cosmos-lab/cosmos3/technical-report.pdf

The robot that needs four models to clear a table

The paper opens with a small scene that lands harder than it looks. A home robot is told to clear the dining table after dinner. Simple instruction. Now count the models it needs under today’s setup.

It needs a vision-language model to find the plates and write a plan. It needs an action model to turn that plan into arm movements. It needs a world model to imagine what happens if it picks up the wine glass at that angle, so it does not knock it over. Three different model families, three pipelines, glued together with hope.

The authors call this fragmented and computationally wasteful. I would call it the reason most of us never ship the robot. You do not become a roboticist. You become a plumber, wiring outputs into inputs and praying nothing leaks. Their question is the whole paper in one line: why not one model that does all of it.

Why I almost ignored it

Cosmos is not new. NVIDIA shipped a whole family of these world models through 2025, and the sprawl was the problem. There was Cosmos Predict for world generation. Cosmos Transfer for controlled generation. Cosmos Reason for scene understanding. Cosmos Policy for actions. Four models, four jobs.

So when “Cosmos 3” showed up, my first guess was: bigger version of the same pile. It is the opposite. It is the pile collapsed into one model. And the way they collapsed it is the part worth understanding, because it is not a marketing trick. It is an architecture decision with real teeth.

The core claim: understanding and generation were never two problems

Here is the bet the paper makes. To understand a scene properly, you have to reason about how it will evolve. You cannot really know what a falling glass means without some model of what falling does. So understanding secretly contains prediction.

To generate a plausible future, you need a compact, structured grasp of how the world works. You cannot dream up a realistic next frame without understanding the scene you are continuing. So generation secretly contains understanding.

The two are not neighbors. They are the same muscle, used in two directions. Splitting them into separate models was never a law of nature. It was a workaround for the era when our models were not good enough to hold both at once. Cosmos 3 is the bet that they are finally good enough.

The architecture: two towers, one sequence

This is the technical center of the thing, so let me slow down and make it crisp. Cosmos 3 is built on a Mixture-of-Transformers, or MoT. Forget the name for a second and picture the mechanics. Every input, whether it is text, an image, a video, audio, or a robot action, gets encoded and dropped into one long sequence of tokens. That sequence then flows through the network.

But each transformer layer is split into two pathways with two separate sets of weights. The first pathway is the Reasoner. It handles the understanding tokens, the autoregressive ones, predicting the next token the way a language model does. This is the brain. It interprets the scene: motion, objects, cause and effect.

Images from the paper

The second pathway is the Generator. It handles the diffusion tokens, the ones that get produced by iterative denoising. This is the imagination. It paints physically plausible video and writes out action sequences like joint angles and gripper positions.

Same sequence, two specialist towers. Both towers, by the way, are initialized from the weights of a pre-trained Qwen3-VL vision-language model, so the model inherits real language and vision skill on day one rather than learning to see from scratch.

The one detail that makes it click

Here is the piece I had to read twice, and it is the piece that justifies the headline. How do the two towers talk to each other. The answer is a careful, lopsided attention rule.

The Reasoner tokens use causal attention. Each one looks only backward, only at other Reasoner tokens, exactly like a normal language model. This protects the model’s language ability. Nothing the generator does can corrupt the reasoning stream.

The Generator tokens use full attention, and they get to look at everything: all the other generator tokens, and all the reasoner tokens. So when the model paints the future, it can see the full understanding the brain built up.

Architecture of COSMO 3, generated using prompts.

And the crucial asymmetry: the Reasoner is never updated based on the Generator. Information flows one way only. The brain reasons, then hands its understanding to the hands. The hands never reach back and rewrite the brain.

That is the whole idea, mechanically. Understanding first, conditioned generation second, in one forward pass, with a wall that keeps the reasoning clean. The brain thinks before the hands move, and that ordering is baked into the attention math.

About those parameter counts

If you read the launch coverage you saw two different sets of numbers, and it was annoying. Some outlets said Nano is 8B and Super is 32B. Others said 16B and 64B. I flagged it as a discrepancy. The paper resolves it, and the answer is satisfying: both were right.

Cosmos 3 results overview (from paper)

Each tower has its own parameters. Cosmos 3 Nano is a 16B model: an 8B reasoner tower plus an 8B generator tower, built on Qwen3-VL 8B. Cosmos 3 Super is a 64B model: 32B plus 32B, built on Qwen3-VL 32B. The dev blog was quoting the per-tower size. The paper quotes the total. There is also a 4B Edge variant (2B plus 2B) mentioned for a later release.

Nano is built to run on workstation-grade hardware like an RTX PRO 6000, for real-time robotics. Super targets datacenter Hopper and Blackwell GPUs, for large-scale synthetic data generation and the heaviest reasoning.

One model, many jobs

Because understanding and generation share a backbone, the same checkpoint shape-shifts depending on what you feed it. No architecture changes, just different input and output modalities.

Images taken from paper.

Feed it text, it generates a physically plausible image. Feed it text or an image, it generates video. Feed it a video and a question, it answers in text like a VLM. Feed it an action plus a starting frame, it shows you the consequence, which is a forward dynamics model. Feed it before-and-after video, it infers the action that caused the change, which is inverse dynamics. Feed it a scene and a task, it outputs the actions a robot should take, which is a policy.

One model that is a vision-language model, a video generator, a world simulator, a dynamics model, and a robot policy, selected by how you call it. The paper’s word for this is omni-functional, and for once the jargon is earned.

The part nobody screenshots: the data work

Big models are easy to admire and easy to overrate. The quiet signal of seriousness is in the data, and this is where Cosmos 3 earned my respect.

NVIDIA did not caption its training videos with an off-the-shelf model. They trained their own captioners, LoRA-tuned from Qwen3-VL-8B, specifically so they could control the output structure and avoid hallucinated or missing fields. Captions are stored as structured JSON with fixed semantic slots: subjects, layout, lighting, camera motion, state changes, audio cues.

For images they use a quadrant-scan trick, describing each corner and the center separately, because standard captioners tend to miss the second and third subject in a busy frame.And here is the admission I respect most, stated plainly in the paper. Web-scale video is great for variety but thin on actual physics. In real interactive settings, they write, the model often struggles with scene dynamics, physical interactions, and long-horizon behavior. So they built a large synthetic-data corpus specifically to patch those holes, targeting exactly the failure cases generic web data underrepresents.

That is a team telling you where their model is weak and what they did about it. You do not see that in a press release. You see it in a real report. They then open-sourced six of those synthetic datasets: robot manipulation, physics interactions, spatial reasoning, digital humans, driving, and warehouse safety. So you can post-train Cosmos 3, or your own model, on data that used to be a moat.

The results, with the asterisks intact

NVIDIA claims state of the art across most of its benchmark suite. Here are the specific numbers that matter, and I am keeping the caveats attached because they change the story.

On physical-world reasoning, the gap is the headline. On driving reasoning, Cosmos 3 Super scores 79.3 against Gemini 3.1 Pro’s 47.2. That is not a nudge, that is a different league, and it is the kind of margin you only get from a model built for physical scenes rather than general chat. On smart-infrastructure reasoning, Super edges Gemini 3.1 Pro too, 62.6 to 58.6.

But on general reasoning, the closed frontier model still wins: Gemini 3.1 Pro scores 77.5, Cosmos 3 Super 73.7. Cosmos beats the open Qwen3-VL-32B there, not the best closed model. Worth saying out loud.

On generation, the standout is that an open model is trading blows with closed ones. A post-trained Cosmos 3 Super variant scores 91.36 on text-to-image, just past Gemini 3 Pro Image at 90.85. On image-to-video it hits 82.8 against Veo 3.1’s 82.6. On robot policy, the post-trained Nano scores 39.7 against the specialist π0.5 baseline at 28.1.

Now the asterisk, because it is load-bearing. The numbers that beat closed models come from post-trained specialist variants, not the base checkpoint. NVIDIA fine-tuned a version specifically for text-to-image, another for image-to-video, another into a robot policy. The base omni-model is the strong generalist. The record-setting scores are its specialized children. That is a legitimate and common way to ship, but “open model beats Veo” is really “an open model, fine-tuned for this one task, edges Veo on this one benchmark.” Different sentence.

They also shipped a new evaluation harness, HUE, because the old video benchmarks are saturating. When every top model scores in the high nineties, the leaderboard stops discriminating. HUE breaks each generated clip into atomic yes/no fact checks across alignment, physics, geometry, and visual integrity. I like that they admitted the old rulers broke and built a sharper one.

The genuinely open part

This is not a weights-only release with the good stuff held back. NVIDIA released the model checkpoints, the training code, the post-training recipes, the synthetic datasets, and the evaluation benchmark, all under the Linux Foundation’s OpenMDW-1.1 license. The stated goal is reproducibility: other labs should be able to verify the numbers and build on top, instead of trusting a marketing chart.

I have learned to treat the word “open” with suspicion. It usually carries an asterisk you find later. This release has a smaller asterisk than almost anything at this scale, and that is the rarest part of the whole announcement.

The honest caveats

It is not magic, so here is where I would keep my guard up. NVIDIA is grading its own homework, state of the art is NVIDIA’s word, about NVIDIA’s model, on benchmarks NVIDIA partly designed. The saving grace is that the full release makes every claim checkable, which is the entire point. But the independent verification has not happened yet. Today’s numbers are claims, not verdicts.

The record-beating results are specialist variants, not the base model, as covered above. Hold that distinction when you read the hype. Generation is not fully turnkey today. The Cosmos 3 Reasoner is available as a production microservice now. The Generator microservice, the part that does the actual video and action generation, is still listed as coming. You can run generation from the open repo, but the polished one-command deployment for the most exciting half is pending.

And “physically plausible” is not “physically correct”, model that paints a convincing collision is not a physics engine that is right. For generating diverse synthetic training data, plausible-and-varied is often exactly what you want. For anything safety-critical, plausible is where the worry starts, not where it ends. The paper’s own admission about weak physical interaction modeling is the tell.

None of this is hidden, the report is upfront about the staged availability, the synthetic-data patching, and the new benchmark. Which is more than I can say for most of the coverage.

Why it matters

Strip it all back and Cosmos 3 makes one argument: understanding, prediction, and action were never three problems. We split them up because our models could not hold them together. So we became plumbers.

Putting them in one model, in the right order, brain before hands, with a wall that keeps the reasoning clean, is the bet that they belong together. Handing the whole thing over open, datasets and benchmark included, is a second bet: that the field moves faster in the open than behind a wall.

The payoff is not academic, a robot that has to grab the thing you pointed at. A car that has to guess what the cyclist does next. A warehouse system that needs to flag a hazard a fraction of a second before it happens. All of that runs on a model that can reason about a scene and imagine its next moment, cheaply enough to actually deploy.

That is the part worth keeping, the interesting moves in this field are not always a bigger model. Sometimes it is someone noticing that three problems were one problem the whole time, and being honest about what is still broken.

Disclosure: I used AI tools to help with spellcheck and editing (grammarly), and image generation (chatgpt image 2). The ideas, final writing, fact-checking, and publishing decisions are my own.