NVIDIA’s 3B Model Just Embarrassed 30B Vision Models

It detects anything you can describe, runs 10x faster than its rivals, and fits in a third of the size. The trick: it stopped spelling out boxes.

Modern vision-language models have a strange habit. When you ask one to draw a box around an object, it does not draw a box. It spells it. It says “1.” Then “3.” Then “0.” Then a comma. Then “6,” “4,” “7.” One digit at a time, like a kid reading a phone number off a screen.

Image Generated using ChatGPT.

If you cant read the paper further then please click here

To produce a single box, the model takes four to seven separate steps. For an image with 200 objects, that is hundreds of tiny sequential decisions, each one waiting on the last.I always assumed this was fine. A small tax. The price of using a language model for vision.

LocateAnything, a new model from NVIDIA and four universities, argues it was never fine. And the fix turns out to improve not just speed, but accuracy. That second part is what changed my mind, let me explain.

Why a language model is drawing boxes at all

First, some context, because the habit did not come from nowhere. For years, object detection belonged to specialized models. You trained them on a fixed list of categories. They were fast and accurate, but rigid. They only knew the classes you gave them, and they could not take an instruction in plain English.

Vision-language models changed the deal. Because they speak language, you can ask them anything. “Find the red-roofed houses in the corner.” “Point to the second person from the left.” “Detect every product on this shelf.” One model, open-ended questions, no fixed class list.

But there was a price. A language model only knows how to do one thing: predict the next token. So to make it output a box, researchers had to turn the box into text. They tried two ways. The first was textual digits: write the number 1024 as the characters “1,” “0,” “2,” “4.” The second was quantized tokens: chop the image into a thousand bins and emit a special token for each coordinate, like <130> then <647>.

Both share the same flaw. They take a flat, two-dimensional object and stretch it into a one-dimensional string. Then they make the model produce that string one piece at a time.

The problem, in plain terms

A box is one object. Four numbers that belong together. The top-left and bottom-right corners are tightly linked: if you know three of them, you can almost guess the fourth.

But when the model spells the box out token by token, it treats those four numbers like four unrelated words in a sentence. It predicts each one looking only at what came before. The coupling between corners, the thing that makes a box a box, gets thrown away.

That costs you twice: First, speed. Sequential generation means you cannot start the next number until the current one is done. Boxes pile up. Latency grows with every object in the scene. In a crowded image, the model spends most of its time waiting on itself.

Second, and less obvious, accuracy. By forcing a coupled object into a one-dimensional stream, you hand the model a worse learning target. It never gets to see the box as a single whole, so it never fully learns the geometry that ties the corners together.

Why not just predict tokens in parallel?

Here is the obvious objection. The language world already has a trick for this. It is called multi-token prediction. Instead of one token at a time, you predict several at once. Faster decoding, same model. People use it everywhere.

So why not bolt that onto a detector and call it a day? LocateAnything’s answer is the most interesting part of the paper. Generic multi-token prediction is structure-blind. It grabs a fixed-size chunk of the sequence and predicts all of it together, with no idea where one box ends and the next begins.

Architecture and Block-Based Output Representation (image from paper)

So it learns nonsense. It will happily predict a chunk that runs off the end of one box and into the middle of the next, splicing together coordinates that belong to different objects, sometimes different categories entirely. It is forced to model correlations that do not exist, between numbers that have nothing to do with each other.

The result is a model that fits a lot of unreliable patterns, hallucinates structure, and decodes slower for the trouble. Parallel, yes. Useful, no. The lesson lands hard: parallelism is only a win if you parallelize the right thing. Chunk by arbitrary size and you break the object. Chunk by the object and you keep it whole.

The fix: say the whole box at once

LocateAnything’s idea is almost embarrassingly simple. Stop spelling. Say the whole box in one breath. And make the chunk a box, never anything smaller or larger.

They call it Parallel Box Decoding. Instead of emitting four coordinates in four steps, the model emits all four in a single step. The box becomes an atomic unit: indivisible, predicted as one thing.

To keep this clean, every output is packed into a fixed slot of six positions: four for coordinates, two for structure (the open and close markers). If a slot is unused, it gets filled with a null token so the shape stays constant. That uniform shape is what lets the hardware predict the whole block in one parallel pass.

The model uses four kinds of these blocks:

• a Semantic block for the words (what the thing is),
• a Box block for the coordinates,
• a Negative block to say “this object is not here,”
• an End block to stop.

That Negative block is quietly important. We will come back to it.

The trick that makes it actually work

Here is the catch. If you just train a model to blurt out boxes in parallel, you damage the thing that made it good in the first place: its step-by-step reasoning. The language ability that let it understand the question in the first place.

So LocateAnything trains two ways at once: One stream learns the old way, token by token, left to right, preserving the model’s language and reasoning. The other stream learns the new way: whole boxes in parallel. Same answer, two formats. The model is taught both side by side.

Results image from original paper.

The clever bit is how it keeps the two streams from poisoning each other. It uses a custom attention mask with three rules, and the rules are worth understanding because they are where the geometry gets learned.

Inside a single box, all four coordinates can see each other. This is the key one. The corners attend back and forth, so the model can finally capture the relationship between them. The coupling that the old method threw away is now exactly what the model is learning.

Qualitative Results (from paper)

Across different boxes, attention runs one way only. A box can look at the boxes that came before it, but not the ones still to come. This lets the model remember what it has already placed, which is how it avoids predicting the same object twice or skipping one. And the old reasoning stream stays strictly causal and walled off, so the model never loses the language skills it started with.

Three rules. One outcome: it keeps its reasoning, and it gains the ability to fire off whole boxes in one shot. You get the speed without paying for it in intelligence.

Three speeds, on demand

Because the model can do both, it offers three modes.

• Slow mode spells everything out, the old way. Highest accuracy. Slowest.
• Fast mode predicts every box in parallel. Fastest. But it has two ways to stumble in hard scenes, and the paper is refreshingly specific about both. The first is a format mess. In a busy image with many categories, the model can get confused at the boundary between one class and the next, and emit a malformed block that mixes structural markers and coordinates into garbage. The second is spatial blur. When objects sit in a tight, regular grid, like rows of identical items on a shelf, the parallel guess can land on a coordinate halfway between two of them. Neither box is right. The overlap with the true object collapses.
• Hybrid mode is the clever default. It runs fast by default, but it watches every block as it comes out. When a block looks unreliable, it throws that one block away and quietly redoes it the slow, careful way. Two simple signals trigger the fallback: the model is not confident enough about its top coordinate, and its top few guesses are spread too far apart. One bad block gets fixed. The rest stay fast.

This is the part that makes it usable in the real world. You get most of the speed, and you catch the ugly failures before they ship.

Where the data comes from

None of this works without scale. LocateAnything is trained on a dataset the authors built themselves: 12 million images, 138 million language queries, 785 million boxes. Six domains: everyday detection, GUI screens, language grounding, OCR, document layout, and pointing.

Data engine for multi-targets grounding (from paper)

They built it with a pipeline of other models. One model writes the queries, another predicts points, a segmentation model turns points into boxes, and a final pass verifies everything. They also deliberately added 22 million negative samples: queries for objects that are not in the image. That teaches the model to say “not here” instead of inventing a box, which is the whole reason the Negative block exists. A detector that cannot say no is a detector that hallucinates.

The model itself

The architecture is lean. A Moon-ViT vision encoder reads the image at its native resolution, which matters more than it sounds: keeping the full resolution preserves the fine detail you need to place a tight box. A simple two-layer connector passes those features to a Qwen2.5 language decoder, which does the actual generating. That is the whole thing. About 3 billion parameters. Hold that number, because it is about to do some heavy lifting.

The results: faster and better, not faster or better

Now the payoff, the usual story with a speed trick is a trade. You go faster, you lose accuracy. LocateAnything mostly refuses that trade. At 3 billion parameters, in its default hybrid mode, it runs at 12.7 boxes per second. That is about 2.5 times faster than the closest comparable model, Rex-Omni, and roughly 10 times faster than a standard text-coordinate model like Qwen3-VL.

Image from original paper.

And it is more accurate, especially where it counts. On LVIS, a long-tailed detection benchmark, it improves mean F1 by 3.8 points over Rex-Omni. The gap is widest at high precision: at the strict overlap threshold of 0.95, it scores 31.1 against Rex-Omni’s 20.7. Tight, accurate boxes are exactly where the old spelling method was weakest, and exactly where saying the box as a whole helps most.

Results from paper

It is not just detection

The reason it is called LocateAnything is that the same model does six jobs.

On dense, tiny-object scenes like drone footage, it reaches 39.9 mean F1 on VisDrone, ahead of Rex-Omni’s 35.8. On document layout, it hits 76.8 on DocLayNet and 70.1 on M6Doc, well past Rex-Omni’s 55.6 on the latter. On referring expressions, where you describe an object in a full sentence rather than name it, it stays competitive with the best and tops the RefCOCOg test set among the models compared.

Pointing is where it really shows off. Here the model has to drop a single point inside the target rather than draw a box. It posts 83.9 on COCO, 87.6 on the dense benchmark, and 91.0 on RefCOCOg, beating point specialists and models several times its size. The box-aligned idea, it turns out, transfers cleanly to points.

The ablations that prove the point

Good papers earn their claims with control experiments, and this is where the thesis gets tested.

Take three models, train them identically on the same data, change only how they represent a box. Plain text coordinates score 49.1. Quantized tokens score 50.1. The box-aligned method scores 52.1. The structure helps even in slow mode, before you turn the speed on at all. This is not a faster way to be the same. It is a better way to learn.

Then the direct test of the earlier objection. Put box-aligned decoding next to the structure-blind parallel methods, the generic chunking approaches. Those land around 45 to 46 F1 while crawling at 5 to 6 boxes per second. Box-aligned decoding hits 49.6 at nearly 17 boxes per second. Faster and more accurate, against the very methods that were supposed to be the shortcut. Parallelizing the right unit wins on both axes.

One more, quietly reassuring. They take the whole approach and drop it onto a different backbone, Qwen3-VL-4B, trained only on COCO. The plain version scores 50.8 at 2.8 boxes per second. With parallel box decoding in hybrid mode, the same backbone reaches 52.0 at 9.4 boxes per second. The idea is not glued to one model. It travels.

The size punchline

Here is the number that should stop you. LocateAnything is a 3B model. On ScreenSpot-Pro, a hard benchmark for grounding elements on professional software screens, it scores 60.3 and takes the top spot. The models it beats include GUI-Owl at 32B and a 30B Qwen3-VL variant. On pointing it beats OVIS2.5 at 9B. On detection it beats Qwen3-VL at 8B. A 3B model, beating 8B, 9B, 30B, and 32B models. Not on a toy task. On the hard benchmarks.

Results from paper

The lesson is not “small is better.” It is that a better formulation beats raw scale. The bigger models are spending parameters to be good at a bad habit. Fix the habit, and a third of the size is enough.

The honest caveats

It is not magic, it is trained almost entirely with supervised fine-tuning. The authors say reinforcement learning is the obvious next step, the lever for pushing accuracy higher and the fallback rate lower, and it is not done yet.

It still loses to a specialized detector on its home turf. On document layout at high precision, the purpose-built DocLayout-YOLO is clearly ahead. And fast mode genuinely does drop accuracy in dense scenes. The hybrid fallback covers this, but it is a patch, not a cure.

The model is fastest exactly when scenes are simple, and slows down to stay correct when they are not. The speed number you see is the speed on easy images. None of this is hidden. The paper is upfront about all of it, which is more than most of the coverage was.

Why it matters

Strip everything away and LocateAnything makes one argument: the way we have been getting boxes out of language models was built on the wrong assumption. We treated a box like a sentence. It is not. It is one coupled object, and it should be predicted as one.

Fixing that bought them three things at once: more speed, more accuracy, and a smaller model. You almost never get all three. When you do, it usually means someone found a real mistake, not a clever trick.

My experiment: worked really well on ai generated image. (by author)

And the payoff is not academic. A small, fast, language-grounded model that can detect, point, read text, and ground instructions on a screen is exactly what an agent needs to act in the world. A robot that has to localize what you asked for. An assistant that has to find the right button to click. That future runs on models that are quick and cheap enough to deploy, not just accurate on a leaderboard.

That is the part worth keeping. The interesting advances in this field are not always bigger models. Sometimes they are someone noticing that everyone has been holding the tool the wrong way.

Disclosure: I used AI tools to help with spellcheck and editing (grammarly), and image generation (chatgpt image 2). The ideas, final writing, fact-checking, and publishing decisions are my own.