Features &
Technical Capabilities
The implementation detail: attention layout, positional encoding, the encoder-free projections, MoE routing, the speculative-decoding drafter, and how quantisation-aware training gets a 2B model into 800 MB. Written as a spec sheet, not a pitch.
For what it does day to day, see capabilities & performance
The base transformer
Conventional in outline, with a handful of deliberate choices that show up later as memory savings.
Decoder-only
A standard decoder-only Transformer stack. No encoder-decoder split, no recurrence - unlike RecurrentGemma, which used a Griffin architecture.
Pre-norm and post-norm
RMSNorm applied both before and after each block, rather than one or the other. Combined with QKNorm on the attention queries and keys, this is what keeps training stable at scale.
262,144-entry vocabulary
SentencePiece with split digits, preserved whitespace and byte-level fallback - so arbitrary bytes always encode, and numbers tokenise digit-by-digit rather than in arbitrary chunks.
1234 as four separate
digit tokens rather than one or two merged chunks gives the model a consistent positional view of numbers.
It's a small vocabulary decision that shows up directly in arithmetic reliability - and Gemma 4 scores 89.2
on AIME. Preserved whitespace matters for the same reason in code, where indentation is semantic.
How 256K is made affordable
Long context is mostly a memory problem, not a modelling one. Three separate tricks attack it.
Interleaved local and global layers
Most layers attend only within a local window; one in every five or six attends globally across the full sequence. Local layers cost far less memory per token, so the expensive global attention is paid for only where it's needed.
E2B uses a denser 4:1 ratio - proportionally more global layers - which helps a very small model retain coherence.
pp-RoPE on global layers
Global layers use pp-RoPE with p=0.25; local layers use standard RoPE. The base frequencies differ too - 1M for global layers and 10k for local ones, so each layer type is tuned for the range it actually attends over.
Keys reused as values
In the global attention layers, the key tensors are reused as values rather than storing both. That single change cuts the global KV cache footprint by 37.5% - the specific reason a 256K context is practical on hardware you can afford.
KV cache sharing across layers
The edge models share KV cache between layers at fixed ratios - 20/35 on E2B and 18/42 on E4B. Fewer distinct caches means less memory held during generation, which is what lets E2B run in 676 MB on a phone GPU.
How images and audio get in
This is where the family genuinely diverges. Three different approaches sit inside one model generation.
| Model | Vision path | Audio path |
|---|---|---|
| 31B · 26B A4B | 550M ViT encoder | No audio input |
| 12B Unified | 35M projection - no encoder | Raw projection - no encoder |
| E4B · E2B | 150M ViT encoder, patch size 16 | 305M audio encoder |
The 12B's encoder-free vision
Instead of a vision transformer, the 12B takes 48×48×3 RGB patches and pushes them through a single 35M-parameter projection with 2D positional embeddings. That replaces a 550M encoder with something roughly one-fifteenth the size.
This is the main reason the 12B approaches 26B quality on less than half the memory - a large fraction of the saving is simply not carrying a vision encoder.
The 12B's encoder-free audio
Audio bypasses encoding entirely. Raw 40 ms chunks at 16 kHz become 640-dimensional vectors projected directly into the same embedding space as text tokens. There is no intermediate audio representation.
Which is why the 12B posts the family's best transcription result - 0.063 average word error rate on FLEURS - despite not being the largest model.
The 26B A4B configuration
The Multi-Token Prediction drafter
Not a separate draft model - a small head bolted onto the main one, which is what makes it practical to deploy.
Architecture
The drafter is a 4-layer Transformer that cross-attends to the main model's KV cache, so it shares state rather than duplicating it. Model dimension is 256 on E2B and E4B, and 1024 on the 26B A4B and 31B.
Because it reads the target model's own keys and values, the drafts it proposes are well aligned with what the full model would produce - which is what keeps acceptance rates high.
The vocabulary projection trick
A naive drafter would need a d×262,144 output projection - enormous relative to a 4-layer head. The efficient variant clusters the vocabulary and projects to d×4,096 instead, roughly a 64× reduction in that layer.
Without this the drafter's own cost would eat much of the speedup it exists to deliver.
Quantisation-aware training
Google trains the quantised models rather than compressing them afterwards, and the per-component detail is unusually granular.
| Component | Precision | Result |
|---|---|---|
| Mobile weights | int2 / int4 weights, int8 activations |
Mixed per layer - text-only footprint down to ~0.8 GB on E2B |
| Q4_0 checkpoints | Blockwise 4-bit | The published GGUF format - 2.9 GB for E2B, 17.5 GB for 31B |
| Vision encoder 150M | W8A8 |
2× smaller forward pass, 44% lower on-device latency |
| Audio encoder 305M | W{2,4,8}/A8, varying by layer |
390 MB → 87 MB on disk, a 78% reduction |
-qat-q4_0-gguf repo exists for your model, use it.
Infrastructure and data
Infrastructure
Trained on TPUv5p and TPUv6e, with optimiser state sharded using a ZeRO-3 style implementation across 16 data shards. Multi-pod coordination uses the Pathways approach with single-controller JAX, the GSPMD partitioner and the MegaScale XLA compiler.
Pre-training data
Web documents, code, images and audio across a wide range of domains. Filtered for benchmark decontamination and for removal of personal information and other sensitive data.
The January 2025 cutoff is why AIME 2026 is a meaningful test - those problems postdate training, so contamination isn't a plausible explanation for the 89.2.
Safety approach
Relevant if you're deploying this, because open weights mean the safety work is yours to extend.
Policy areas
Mitigations target child sexual abuse material and exploitation, dangerous content, sexually explicit content, hate speech and harassment.
Data filtering
Training data is filtered to remove personal information and other sensitive data, alongside benchmark decontamination.
Evaluation and governance
Automated and human evaluations run without safety filters in place, so the underlying model behaviour is measured rather than the filtered output. Governed under Google's Frontier Safety Framework.
Complete technical specification
Everything published, in one table. Blank cells mean the figure isn't in the technical report - not that the value is zero.
| Specification | 31B | 26B A4B | 12B | E4B | E2B |
|---|---|---|---|---|---|
| Total parameters | 31B | 26B | 11.95B | 4.5B | 2.3B |
| Active per token | 31B | 3.8B | 11.95B | 4.5B | 2.3B |
| Architecture | Dense | MoE | Unified | Dense | Dense |
| Context window | 256K | 256K | 256K | 128K | 128K |
| Local:global attention | 5:1 | 5:1 | 5:1 | 5:1 | 4:1 |
| Global positional | pp-RoPE, p = 0.25, base 1M | ||||
| Local positional | Standard RoPE, base 10k | ||||
| KV cache sharing | - | - | - | 18/42 | 20/35 |
| Global KV reduction | 37.5% - keys reused as values | ||||
| Vocabulary | 262,144 · SentencePiece, split digits, byte fallback | ||||
| Normalisation | RMSNorm pre + post, QKNorm | ||||
| Vision encoder | 550M ViT | 550M ViT | 35M proj. | 150M ViT | 150M ViT |
| Audio encoder | - | - | none, raw | 305M | 305M |
| Audio chunk | - | - | 40 ms @ 16 kHz | 40 ms | 40 ms |
| MTP drafter dim | 1024 | 1024 | - | 256 | 256 |
| MTP drafter layers | 4 | 4 | - | 4 | 4 |
| Memory · BF16 | 69.9 GB | 57.7 GB | 26.7 GB | 17.9 GB | 11.4 GB |
| Memory · Q4_0 | 17.5 GB | 14.4 GB | 6.7 GB | 4.5 GB | 2.9 GB |
| Mobile build | - | - | - | 2.5 GB | 1.1 GB |
| Licence | Apache 2.0 | ||||
| Data cutoff | January 2025 | ||||
Technical questions
What is pp-RoPE and why p = 0.25?
It's a variant of rotary positional embedding applied only to the global attention layers, with the local layers keeping standard RoPE. The base frequencies differ as well - 1M for global, 10k for local - so each layer type is tuned to the range it actually attends across. Google reports the configuration and the resulting KV cache reduction; the report doesn't derive why 0.25 specifically is optimal.
How does reusing keys as values actually save memory?
Normally each attention layer caches separate key and value tensors for every token. In Gemma 4's global layers the key tensor serves as the value too, so only one is stored instead of two. That's the 37.5% reduction in global KV cache - and since the KV cache, not the weights, is what grows with context length, it's the change that makes 256K affordable.
Is the 12B genuinely encoder-free, or is the projection just a small encoder?
Fair question, and the answer differs by modality. For vision there is still a learned projection - 35M parameters taking 48×48×3 patches with 2D positional embeddings - so it's better described as radically smaller rather than absent. That's still a fifteen-fold reduction from the 550M ViT.
For audio it's more literal: raw 40 ms chunks at 16 kHz become 640-dimensional vectors projected straight into the text embedding space, with no encoder stage at all.
Why does the MoE need all 26B in memory if only 3.8B activate?
Because routing is per token and unpredictable - any expert may be needed for the next token, so all of them must be resident. What you save is memory bandwidth, since only the selected experts are read for each token. Single-stream decoding is bandwidth-bound, which is why the saving translates into a 4.4× speedup despite the full model occupying 57.7 GB at BF16.
What's the 262k → 4,096 clustering in the MTP drafter?
A four-layer drafter would otherwise need an output projection of dimension × 262,144 to cover the full vocabulary - a layer far larger than the drafter itself. The efficient variant clusters the vocabulary so the projection targets 4,096 entries instead, roughly a 64× reduction. Without it, the drafter's own compute would consume much of the speedup it exists to provide.
How many layers do the models have?
The technical report gives the attention ratios, cache sharing ratios and drafter
dimensions, but doesn't publish per-model layer counts or hidden dimensions in the material available
here. The config files in each Hugging Face repository carry the exact values - check
config.json rather than trusting a secondary source.
Does fine-tuning break the safety training?
It can. Safety behaviour lives in the weights, and further training on unfiltered data will erode it - this is a well-documented property of open models generally, not a Gemma-specific flaw. If you fine-tune and deploy to end users, re-evaluate safety behaviour afterwards and add independent input and output filtering rather than relying on the base model's training.
Specs are one view.
See how these design choices show up as behaviour, throughput and benchmark scores.