Gemma 4 VRAM
Will it fit? Weights are the easy half - the KV cache scales with context and concurrency, and on the 31B at full length it grows larger than the weights themselves. Set your parameters and see the breakdown.
Overhead is a flat 2 GiB allowance for activations and CUDA context. Real usage varies with runtime and batch scheduling - treat the total as a planning figure, not a guarantee.
Most layers barely cache anything
Gemma 4's interleaved attention means context memory behaves very differently from a conventional transformer - and it's the reason a 60-layer model can hold 256K at all.
Local layers cap at 1,024 tokens
Sliding-window layers only ever cache their window - 1,024 tokens - regardless of how long your input is. Fifty such layers on the 31B contribute just 0.78 GiB, and that figure never grows.
Global layers carry all the cost
Only the full-attention layers scale with context. At 256K they account for 20.0 of the 20.78 GiB - about 96% of the cache from a sixth of the layers.
global_layers × kv_heads × head_dim × 2 × bytes × context plus
local_layers × kv_heads × head_dim × 2 × bytes × min(context, 1024). Computed that way, the
31B lands on 20.78 GiB and the 26B A4B on 5.20 GiB at 262,144 tokens - matching the published figures to
within 0.005 GiB, which is why those two models are marked verified above.
The MoE needs 4× less context memory
Already the faster model to serve. It turns out to be dramatically cheaper on long context too - for reasons that have nothing to do with the mixture of experts.
| Attention configuration | 31B dense | 26B A4B MoE | Effect |
|---|---|---|---|
| Total layers | 60 | 30 | Half as many caches |
| Global (full-attention) layers | 10 | 5 | Half as many that scale with context |
| Global KV heads | 4 | 2 | Half the width again |
| Global head dimension | 512 | 512 | Same |
| Local layers | 50 | 25 | Both capped at a 1,024 window |
| Local KV heads | 16 | 8 | - |
| KV cache at 256K | 20.78 GiB | 5.20 GiB | 4× smaller |
| KV cache at 128K | 10.78 GiB | 2.70 GiB | 4× smaller |
| KV cache at 32K | 3.28 GiB | 0.82 GiB | 4× smaller |
Quantising stops helping
A complaint that circulates about the 31B - that its context memory is unreasonable - is essentially correct, and this is why.
| Gemma 4 31B configuration | Weights | KV @ 256K | Total | KV share |
|---|---|---|---|---|
| BF16 | 57.8 GiB | 20.8 GiB | 78.6 GiB | 26% |
| FP8 | 28.9 GiB | 20.8 GiB | 49.7 GiB | 42% |
| 4-bit | 20.5 GiB | 20.8 GiB | 41.3 GiB | 50% - larger than the weights |
The fix is context, not precision. Dropping from 256K to 32K takes the cache from 20.8 GiB to 3.3 GiB - a far bigger saving than any quantisation step.
KV cache by context length
Single request. Multiply by concurrency for a server.
| Context | 31B | 26B A4B | 12B est. |
|---|---|---|---|
| 4K | 1.09 GiB | 0.27 GiB | 0.56 GiB |
| 8K | 1.41 GiB | 0.35 GiB | 0.81 GiB |
| 16K | 2.03 GiB | 0.51 GiB | 1.31 GiB |
| 32K | 3.28 GiB | 0.82 GiB | 2.31 GiB |
| 64K | 5.78 GiB | 1.45 GiB | 4.31 GiB |
| 128K | 10.78 GiB | 2.70 GiB | 8.31 GiB |
| 256K | 20.78 GiB | 5.20 GiB | 16.31 GiB |
31B and 26B A4B match published measurements exactly. The 12B column is computed from its config file and hasn't been checked against a measured figure - its global layers use 8 KV heads at 256 dimensions, which is why it sits closer to the 31B than its parameter count suggests.
Two ways to read the weight figures
| Model | Raw weights BF16 | Google's stated requirement | Difference |
|---|---|---|---|
| 31B | 57.8 GiB (62 GB) | 69.9 GB | Includes working memory and encoders |
| 26B A4B | 48.4 GiB (52 GB) | 57.7 GB | Same |
| 12B | 22.3 GiB (24 GB) | 26.7 GB | Same |
Both are correct for different definitions. Google's published table is labelled a memory requirement for inference and runs above raw parameter arithmetic; the calculator above uses raw weights plus an explicit overhead line so you can see where the difference goes.
How to use less
Roughly in order of how much they save.
Lower the context
The largest lever by far at long context. The 31B drops from 20.8 GiB to 3.3 GiB going from 256K to 32K - more than any quantisation step can save.
Use the 26B A4B
A quarter of the KV cache of the 31B at the same context, plus roughly 4.4× the throughput, for a two-to-three point accuracy difference.
Quantise the weights
Effective when weights dominate - BF16 to 4-bit takes the 31B from 57.8 to 20.5 GiB. Ineffective once the cache is the larger half.
Cap concurrency
Each simultaneous request holds its own cache. Eight concurrent 32K conversations on the 31B is 26 GiB of cache alone.
Drop a model size
The 12B at 4-bit needs 6.7 GiB of weights against the 31B's 20.5 - and on many real workloads the quality gap is smaller than the benchmark spread implies.
Tune the runtime
vLLM pre-allocates cache from --max-model-len, so set it to what you need.
--gpu-memory-utilization controls how much of the card it will claim.
Common questions
Why doesn't 4-bit quantisation solve my memory problem?
Because it only compresses weights. The KV cache stays at fp16 and scales with context and concurrency regardless of how the weights are stored.
On the 31B at full context, 4-bit weights are 20.5 GiB and the cache is 20.8 GiB - the cache is the larger half. Cutting context is the effective move at that point.
Can I run the 31B on a 24 GB card?
At 4-bit the weights fit, at about 20.5 GiB. What doesn't fit is much context - you'd have roughly 2 GiB left, which is around 8K tokens of cache plus activations. Workable for short prompts, not for long documents. The 12B or the 26B A4B is a better use of a 24 GB card.
Why does the 26B MoE need so much less cache than the 31B?
Nothing to do with the mixture of experts - it's the attention configuration. The 26B has 30 layers to the 31B's 60, five global layers to ten, and two global KV heads to four. Each of those roughly halves the context-scaling portion, and they compound to about 4×.
Does long context slow things down as well as using memory?
Yes. Prefill work grows with input length, so time to first token rises with longer prompts. On a server, a large cache also reduces how many requests fit concurrently, which shows up as queueing. FlashAttention 4 on Hopper GPUs helps the prefill side substantially.
How accurate are these numbers?
The 31B and 26B A4B KV figures reproduce published measurements to within 0.005 GiB, so the formula is sound for those two. The 12B is computed from its config file without a measured cross-check. The flat 2 GiB overhead allowance is a planning figure - actual activation memory varies with runtime, batch scheduling and whether you're using speculative decoding.
What about E4B and E2B?
Their attention configs weren't available in the sources I could reach, so they're not in the calculator rather than being estimated. In practice their memory is dominated by weights - 4.5 GB and 2.9 GB at 4-bit, or 2.5 GB and 1.1 GB as mobile builds - and their 128K ceiling keeps the cache modest. See the datasheet for the published figures.
Know what fits?
Pick a runtime, or see the full specification for every model in the family.