The Part of an AI Model Nobody Talks About — And Why Google Just Deleted It

A beginner-friendly look at how Gemma 4 12B understands images and audio without the component every other multimodal model relies on

When Google DeepMind released Gemma 4 12B on June 3, 2026, the headline spec sounded routine: a 12-billion-parameter model slotting neatly between the edge-friendly E4B and the larger 26B mixture-of-experts model. Labs fill gaps in their model lineups all the time — not usually a reason to write an article.

But buried in the architecture is a genuinely useful lesson for anyone trying to understand how modern AI models actually “see” or “hear.” Gemma 4 12B is encoder-free. It understands images and audio without using a vision encoder or an audio encoder — components that almost every other multimodal model on the market still depends on.

If you’ve ever wondered how an AI model looks at a photo and describes it, or listens to a voice memo and transcribes it, this is a good moment to find out — because Gemma 4 12B does it differently, and seeing both approaches side by side makes the whole idea click.

[embed]Introducing Gemma 4 12B: a unified, encoder-free multimodal model An overview of Gemma 4 12B, a model designed to bring high-performance multimodal intelligence directly to your laptop.blog.google

First, How Does an AI Model “See” at All?

Here’s a question worth sitting with: text is naturally suited to LLMs, but a photo isn’t. So how does a model trained mostly on text ever make sense of pixels?

Text is the easy case. An LLM’s token embedding layer slices text into tokens and converts them into embeddings the model already knows how to work with. Images and audio don’t get that luxury — they arrive as raw pixels or sound waves, nothing like the tokens an LLM was built to digest.

The standard fix, used by nearly every multimodal model today, is to hire a translator. A separate mini-transformer — the vision encoder or audio encoder — sits in front of the LLM with one job: chew through the raw input and turn it into something the LLM can use.

Think of it like a tour guide translating for a tourist who only speaks one language. The guide (encoder) listens to the foreign speech, translates it into the tourist’s language, and only then does the tourist (the LLM) start to understand what’s being said. There’s just one issue: the guide’s translation doesn’t come out in quite the right phrasing. So there’s a second step — a “connector,” typically a small linear layer — that cleans up the translation into a shape the LLM expects. Encoder, then connector, then finally the LLM gets to work.

This is exactly how Gemma 4’s other models handle multimodality. The E2B and E4B models carry a 150-million-parameter vision encoder; the 26B and 31B models use a 550-million-parameter version. The smaller models also pack a 305-million-parameter audio encoder. It’s a well-tested pattern, used by other major open multimodal models too.

But hiring a translator has a cost. The guide has to finish the whole translation before the tourist can respond — that’s latency. And training gets messier: when you fine-tune a model, you’re usually only fine-tuning the LLM, not the encoder bolted onto it, so the two drift apart over time instead of growing together.

So What Did Gemma 4 12B Remove — And How Does That Even Work?

Gemma 4 12B asks a blunt question: what if you skip the translator entirely, and just hand the tourist a phrasebook instead?

For images, that “phrasebook” is a lightweight embedding module with effectively one layer — replacing what used to be 15 to 27 transformer layers of translation. Raw 48×48-pixel patches get projected straight into the LLM’s hidden dimension through a single matrix multiplication. No attention, no semantic pre-processing — just pixels in, embeddings out, and the LLM figures out the rest itself.

That single projection step is doing more work than it looks like. Each patch carries 48 × 48 × 3 = 6,912 pixel values, and squeezing that into Gemma 4 12B’s 3,840-dimensional space takes roughly 26 million parameters on its own. Add the positional handling described below, and the whole module lands around 35 million parameters — compared to 550 million for the equivalent vision encoder elsewhere in the Gemma 4 family. That’s not a smaller, smarter translator. It’s no translator at all: the parameter count dropped because the job of “understanding” what’s in the image moved entirely into the LLM, learned during training rather than computed by a dedicated sub-network.

Here’s the part that makes this genuinely clever, though. If you remove the translator, you also lose something important: a sense of where things are. A real vision encoder uses attention to track where each patch sits relative to the others in the image. Strip that out, and you need another way to tell the model “this patch was in the top-left, that one was in the middle.”

Gemma 4 12B’s solution is almost charmingly low-tech: two lookup tables, one for x-coordinates and one for y. For every patch, the model looks up its x-position and y-position, adds those two values together, and tacks the result onto the patch’s embedding before it goes anywhere near the LLM. It’s a hand-drawn map standing in for what used to be a built-in GPS.

[embed]Gemma 4 12B: The Developer Guide Meet Gemma 4 12B: the first medium-sized, encoder-free multimodal model capable of natively ingesting audio and video…developers.googleblog.com

Audio gets an even simpler fix, for a tidy reason: sound is already a sequence over time, the same way text is a sequence of words — so it doesn’t need the positional workaround images do. The model slices raw audio into 40-millisecond windows (16,000 samples per second, 640 raw values per window) and projects those values straight into the LLM’s embedding space. No encoder, no intermediate features. Just slice, project, done.

[embed]Gemma 4 12B: The Developer Guide Meet Gemma 4 12B: the first medium-sized, encoder-free multimodal model capable of natively ingesting audio and video…developers.googleblog.com

Why This Is Worth Caring About

The obvious payoff is speed. A translator-style encoder has to finish its whole job before the LLM can even start. Remove it, and image or audio information reaches the model almost instantly, so the model can begin “thinking” sooner. Google leans hard on this in its own framing: near-26B-model benchmark performance, in well under half the memory footprint, running on an ordinary 16GB laptop.

[embed]Bringing Gemma 4 12B to your Laptop: Unlocking Local, Agentic Workflows with Google AI Edge Google DeepMind's Gemma 4 12B model brings agentic, multimodal AI capabilities to everyday laptops with 16GB of RAM…developers.googleblog.com

The less obvious payoff is simplicity. One model, one set of weights to train and fine-tune, no separate encoder lifecycle to babysit and scale alongside the LLM as it grows.

None of this comes for free, and it’s worth saying so plainly. When this architecture went public, one reader asked the exact right follow-up question: how much capability does the LLM actually give up by absorbing the encoder’s job itself, and how much extra training data does that absorption cost? Google’s benchmarks suggest the trade-off is a good one at the 12B scale, but “nearly matches the 26B model” is still doing some quiet rounding — and independent testing against models like Qwen or DeepSeek will tell us more than the vendor’s own numbers ever will.

The Takeaway

Here’s what’s worth carrying away from all this, even if you forget every parameter count by tomorrow: for years, the default assumption in AI was that understanding images and sound required a dedicated specialist sitting in front of the language model. Gemma 4 12B is a working bet that the specialist isn’t necessary at all — that a large enough language model can learn to “see” and “hear” on its own, if you just hand it the raw material and get out of the way.

Whether that bet pays off everywhere, or only at this particular size and scale, is still an open question. But it’s a genuinely different idea about where intelligence in a multimodal system should live — and that’s the kind of architectural shift worth understanding now, before it becomes the default everyone takes for granted.

About Accredian

Enjoyed this read? Take the next step. Curiosity brought you this far, let Accredian take you further. Partnering with top global institutes, Accredian brings you rigorous, relevant, and impactful programs. Designed for professionals serious about growing, upskilling, and leading with confidence. If this article sparked something in you, imagine what the right program could do. Discover what’s possible at Accredian.

[embed]Accredian | Programs from IITs, XLRI, SP Jain & IIMs Edit descriptionaccredian.com