A Visual Guide to Gemma 4 12B

Text is Native to the LLM

When you feed text into a language model, the model itself does all the work. Its token embedding layer splits the text into tokens and converts each one into a vector embedding. Those embeddings then flow into the decoder stack, where self-attention layers build up increasingly contextual, meaningful representations of each token relative to the others. Nothing outside the LLM is involved — text is, by definition, the LLM’s native language.

Other Modalities Usually Need Outside Help

Images and audio are a different story. In most multimodal LLMs, the model itself doesn’t touch raw pixels or raw waveforms directly. Instead, a separate encoder — typically a smaller Transformer model that also relies on attention — is used to pre-process the input into a set of embeddings before the LLM ever sees it.

The general pattern: a dedicated encoder processes non-text input using its own attention layers, separate from the LLM’s decoder.

The Connector: Bridging Two Different Embedding Spaces

There’s a catch, though. The embeddings produced by an image or audio encoder don’t automatically fit into the LLM’s world. They typically have different dimensionality, and they occupy a differently-shaped vector space than the LLM’s own token embeddings — which makes sense, since a Large Language Model is, first and foremost, trained on text.

To reconcile this mismatch, multimodal architectures insert a small connector module — usually just a linear projection layer — that maps the encoder’s output into the same dimensional space as the LLM’s token embeddings.

The connector’s job: take encoder output and reshape it so it looks, dimensionally, like a token embedding the LLM already knows how to work with.

Together, an encoder + connector pair is the standard recipe used by many open multimodal models today, including the earlier Gemma 4 variants and models like Qwen 3.5. It works well — but it isn’t free. Two costs stand out:

1. Latency. The non-text encoders have to fully process their inputs before the LLM can even begin working with them, adding a sequential processing step to every multimodal request.
2. Parameter overhead. These encoders aren’t trivial in size — they add a meaningful number of parameters to the overall model, and because they’re usually trained (or at least fine-tuned) somewhat separately from the LLM, they also add complexity whenever you want to fine-tune or scale the model further.

This naturally raises the question DeepMind seems to have asked themselves: what if you didn’t need the encoder at all? What if the connector alone were enough?

To understand how they answered that question, it’s worth first looking at how the existing Gemma 4 encoders work.

The Vision Encoder

Every Gemma 4 model with a vision encoder — E2B, E4B, 26B A4B, and 31B — uses one to process input images, though the encoder’s size scales with the model: E2B and E4B use a vision encoder with roughly 150 million parameters, while the larger 26B A4B and 31B models use a bigger vision encoder with about 550 million parameters.

Mechanically, an input image is broken into small patches of 16×16 pixels. After the vision encoder processes these patches, they’re pooled together into a 3×3 grid of patches.

Because pooling groups 3×3 blocks of the original 16×16 patches together, the resulting patch embeddings effectively represent 48×48-pixel regions of the original image. After this pooling step, a small linear projection layer — the connector — transforms these pooled patch embeddings so they match the shape of the LLM’s own token embeddings. The resulting vision tokens are then interleaved directly with the text token embeddings before being handed to the LLM.

The Audio Encoder

The smaller Gemma 4 models — E2B and E4B — also ship with an audio encoder to handle spoken or other audio input. Both models share the same audio encoder, which has about 305 million parameters.

Just like the vision encoder, once the audio encoder finishes processing an audio clip, its output embeddings are projected into the same dimensional space as the LLM’s token embeddings, and the resulting audio tokens are interleaved with the surrounding text tokens.

While 305 million parameters is relatively modest next to the size of the full LLM, these encoders still add real cost. They increase inference latency because the model has to wait for them to finish, and they complicate fine-tuning: when you fine-tune a Gemma 4 model, you typically only touch the LLM weights, not the encoder weights, which makes it harder to keep the encoders’ capabilities growing in lockstep with the rest of the model.

That tension — real capability, but real cost — is exactly what motivated Gemma 4 12B’s design.

Replacing the Vision Encoder

Instead of a full vision Transformer with 15 or 27 attention layers extracting and refining visual features, Gemma 4 12B uses a lightweight embedding module containing just a single layer.

Rather than starting from small 16×16 patches and pooling them up as the encoder-based models do, Gemma 4 12B works directly with 48×48 patches from the outset. This might sound like it should lose detail compared to a “real” vision encoder, but since the embedder has no attention layers, it was never going to produce the same kind of semantically rich embeddings a full encoder would anyway — so skipping the extra pooling step doesn’t cost much in practice.

The positional embedding problem. One wrinkle with an attention-free embedder is that it needs some way to tell the LLM where each patch sits within the original image. The encoder-based models solve this with 2D-positional RoPE inside the attention layers — but that mechanism doesn’t exist here, since there’s no attention. The LLM’s own positional encoding doesn’t help either, since it’s designed to treat inputs as a flat, one-dimensional sequence of tokens, not a 2D grid.

DeepMind’s solution is to inject spatial position information directly into each vision token’s embedding before it reaches the LLM, using two dedicated matrices — one for x-coordinates, one for y-coordinates.

Each of these matrices is sized 1,120 (the maximum number of patches an image can be split into) by 3,840 (matching the model’s hidden dimension). That maximum-patches figure of 1,120 mirrors the token “budget” options already available in the encoder-based Gemma 4 models, which let users choose how finely an image gets split — 70, 140, 280, 560, or 1,120 patches, with more patches giving a more fine-grained view of the image.

For any given patch, the model looks up its x-coordinate’s row in the x-matrix and its y-coordinate’s row in the y-matrix, then simply adds those two vectors together to produce that patch’s positional embedding. For example, a patch at position x=2, y=1 gets the embedding formed by summing row 2 of the x-matrix and row 1 of the y-matrix.

After adding this positional information to each vision token embedding, a final LayerNorm is applied for training stability, and the embeddings are then projected into the exact dimensionality Gemma 4 12B expects.

Where did 35 million parameters come from? Given that this whole module replaces a 550-million-parameter vision encoder, its own ~35 million parameters might seem surprisingly large for something described as just “an embedding step.” The answer is almost entirely about raw pixel projection, not learned attention or feed-forward capacity. Each 48×48 patch has 48 × 48 × 3 = 6,912 raw pixel values (three color channels), which need to be projected into Gemma 4 12B’s 3,840-dimensional space. That single projection — sized 6,912 × 3,840, plus a 3,840-dimensional bias term — already accounts for roughly 26 million parameters on its own. In other words, the embedder isn’t secretly hiding 35 million parameters’ worth of attention or reasoning capacity; it’s mostly just the unavoidable cost of mapping a lot of raw pixels into the model’s embedding space.

Because the embedder has no attention mechanism, it processes each patch entirely in isolation — there’s no cross-patch context built at this stage at all. All of that relational reasoning between patches is now the LLM’s job, handled through its own attention layers once the vision tokens arrive.

The payoff for this simplification is speed: since vision tokens no longer need to wait on a full Transformer encoder to finish processing, they reach the LLM much faster, letting the model start reasoning about — and generating output from — image input sooner.

Replacing the Audio Encoder

If replacing the vision encoder required some clever positional-embedding engineering, replacing the audio encoder turned out to be even more straightforward.

Instead of extracting processed audio features the way a real encoder would, Gemma 4 12B works directly with raw audio input — much like it works directly with raw pixels for images. Audio has one convenient property that vision doesn’t: since it’s inherently a sequential (effectively one-dimensional, or “2D” in the sense of amplitude-over-time) signal already, it doesn’t need any extra positional embedding scheme the way image patches do. It can be treated similarly to how the model already treats ordinary text sequences.

Concretely, the raw audio input is split into 40-millisecond windows. Since audio is sampled at 16,000 samples per second (16 kHz), each 40ms window contains 640 raw amplitude values — numbers representing how high or low the sound wave is at each instant.

These 640 raw values per window are then passed straight through a linear projection layer, mapping them into Gemma 4 12B’s expected embedding dimensionality.

That’s genuinely the entire pipeline: split the raw waveform into fixed windows, project each window’s raw samples into the model’s embedding space, and hand the result to the LLM. There’s no dedicated tokenization step and no stack of Transformer decoder layers processing the audio beforehand, unlike in Gemma 4 E4B, which still needs to tokenize audio and run it through several stacked decoder layers before the LLM ever touches it.