Gemma 4:
Limitations
What Gemma 4 doesn't do well, where it fails outright, and what to do about each. Drawn from Google's own model card and from independent measurement - with a mitigation for every item, because a list of complaints isn't useful on its own.
Sources: Gemma 4 model card, technical report, and independent testing
What Google documents
Worth reading before the measured findings, because these are the limits the people who built it chose to write down.
Training data gaps and bias
"Biases or gaps in the training data can lead to limitations in the model's responses." The model can reflect socio-cultural biases present in its training material.
Factual accuracy
Models "may generate incorrect or outdated factual statements." Fluency is not accuracy, and Gemma 4 is fluent enough that wrong answers read convincingly.
Nuance, sarcasm, figurative language
The model "might struggle to grasp subtle nuances, sarcasm, or figurative language" - relevant if you're classifying sentiment or moderating real human writing.
Common sense
It "might lack the ability to apply common sense reasoning in certain situations," which tends to surface on problems that are trivial for people and unusual in text.
Open-ended and complex tasks
"Open-ended or highly complex tasks might be challenging." Performance also varies with how much context you provide - more isn't automatically better.
Safety is delegated to you
Google states developers should "implement appropriate content safety safeguards based on their specific product policies." The model ships without one.
Where it underperforms
These come from independent testing rather than the model card, and they're specific enough to plan around.
Agentic and multi-step tool use
Independent aggregate testing ranks Gemma 4 31B #129 of 134 on agentic composites, scoring 25.5/100 - despite excellent MMLU Pro and GPQA results. Strong single-turn reasoning does not transfer to reliable multi-step execution.
Mitigation: keep chains short and verify each step rather than trusting a long autonomous run. Ensure you're on post-July-2026 weights, which improved tool calling substantially. For genuinely agentic workloads, a model built for it will serve you better.
Frontier-difficulty reasoning
HLE is deliberately brutal, and the 31B scores 19.5 - dropping to 8.7 for the 26B A4B and 5.2 for the 12B. Search grounding lifts the 31B to 26.5.
Mitigation: enable search grounding where available, and route genuinely hard problems to a frontier model. This is the widest gap to closed models and it is not closing at this parameter count.
Real-repository software engineering
LiveCodeBench v6 at 80.0 measures self-contained competitive-programming problems. On SWE-Rebench - patching actual repositories - independent testing puts the 31B at 41.6%. Both are real; they measure very different jobs.
Mitigation: scope coding work to single files and clear objectives. Read every diff. Don't infer repository-scale capability from competitive-programming scores.
Multi-hop reasoning over long context
E2B scores 4.1 on GraphWalks and 21.9 on BBH. Its 128K context works for retrieval but collapses when a task requires chaining across the document.
Mitigation: treat edge models as excellent at short, well-scoped tasks. If a job needs reasoning across a long document, use the 12B or larger.
Full tables and evaluation caveats are on the benchmarks page.
Modality limits
The one that catches people is that capability does not increase with size here.
Text output only
Every model in the family outputs text. None generate images, audio or speech. Images and audio are inputs, on the models that support them.
Mitigation: pair with a dedicated generation model. Agent Skills is designed to coordinate exactly this handoff.
Vision quality depends on a setting
Headline vision scores use 1120 vision tokens; the current default is 280. InfographicVQA falls from 92.0 to 82.8 at the default - a large drop on document-heavy work.
Mitigation: raise the token budget for documents and screenshots; leave it low for general image questions.
No native PDF or citations
Unlike some hosted APIs, there's no built-in PDF ingestion or citation mechanism. You handle document parsing and source attribution yourself.
Mitigation: extract text or render pages to images in your own pipeline before sending them.
Knowledge limits
Training data ends January 2025
Anything after that cutoff simply isn't in the weights - including, ironically, Gemma 4's own release. Ask it about recent events and it will either decline or confabulate, depending on how the question is framed.
Mitigation: retrieval or tool calling for anything time-sensitive. Search grounding is available through the Gemini API. Never rely on parametric knowledge for current facts.
Confident wrong answers
Google's model card states plainly that the models "may generate incorrect or outdated factual statements." Gemma 4 writes well enough that a fabricated citation or invented API method reads exactly like a correct one.
Mitigation: ground answers in retrieved sources for factual work, and verify anything checkable - package names, function signatures, quotations, figures. Structured output helps by forcing the model to fill defined fields rather than narrate freely.
Operational limits
Things that don't show up in any benchmark but shape what running this actually feels like.
| Limitation | Impact | Mitigation |
|---|---|---|
| Thinking mode costs tokens | Every published accuracy figure assumes it's on. Reasoning traces are billed and generated output the user never sees | Lower it for classification, routing and extraction; keep it high only for maths, code and multi-step work |
| Long context costs memory | KV cache grows with context; a 256K window can push layers onto the CPU on modest hardware | Set context to what the task needs, not the maximum. pp-RoPE already cuts cache by up to 37.5% |
| No prompt caching in some runtimes | Ollama's Anthropic-compatible endpoint doesn't support it, so long sessions reprocess the whole prefix every turn and get progressively slower | Clear context between tasks; prefer the MoE, which prefills faster; use a runtime with caching where it matters |
| Quantisation below 4-bit degrades badly | Q2 and Q3 builds load and run but lose significant reasoning and coding quality | Use the official QAT 4-bit checkpoints. A smaller model at 4-bit beats a larger one at 3-bit |
| Defaults are wrong out of the box | Ollama defaults to 4,096 context regardless of model capability, and GPU layers can silently fall back to CPU | Set num_ctx and num_gpu explicitly - see the
install guide |
| Dense decode is slow at low concurrency | The 31B manages about 40 tokens/sec single-stream on an H100 | Use MTP checkpoints for roughly 3.1×, or the 26B MoE, which is 4.4× faster |
Safety limits
The consequence of open weights that most deployment guides skip.
There is no server-side moderation between the model and your users
With a hosted API, the provider filters inputs and outputs. With open weights, whatever safety behaviour is trained into the weights is all you get.
Further training on unfiltered data weakens the safety behaviour in the weights. This is a well-documented property of open models generally, not a Gemma-specific flaw.
Google evaluates the underlying model with safety filters off, so published safety results describe raw model behaviour - not what a filtered production system would produce.
Independent input and output classification, sized to your product's risk. ShieldGemma 2 exists for this - though note it's still on the older Gemma Terms licence, not Apache 2.0.
Ecosystem limitations
Not the model's fault, but they constrain what you can build.
Variants are still on the old licence
Only Gemma 4 is Apache 2.0. TranslateGemma, MedGemma, EmbeddingGemma, ShieldGemma, PaliGemma and CodeGemma remain under the Gemma Terms, whose restrictions propagate to anyone you redistribute to.
No successor to the smallest models
Gemma 4's smallest is E2B at 2.3B. If you need something genuinely tiny, Gemma 3 270M is still the only Gemma answer - and it carries the older licence.
Not every size is hosted
Google's documented Gemini API list covers the 31B and 26B A4B. The 12B - the only large model with audio - isn't in it, so audio over a hosted Google API isn't straightforward.
No MTP checkpoint for the 12B
Multi-Token Prediction shipped for E2B, E4B, 26B A4B and 31B in April 2026, before the 12B existed. The 12B misses out on that speedup.
Weights changed mid-generation
The July 2026 refresh altered the chat template, tool calling and vision defaults without a version bump. Results from before and after aren't strictly comparable, and cached weights may be stale.
Benchmark figures are mostly self-reported
The headline tables come from Google's own technical report. That's normal and generally reliable, but the independent checks - Arena Elo and third-party aggregates - tell a less flattering story on agentic work.
If you only do six things
The mitigations with the best return, roughly in order.
Ground factual work in retrieval
The January 2025 cutoff plus confident hallucination is the failure mode most likely to reach your users. Retrieval or tool calling for anything time-sensitive or checkable.
Add your own safety filtering
Open weights come with no server-side moderation, and fine-tuning erodes what's in the weights. Independent input and output classification, sized to your risk.
Keep agentic chains short and verified
This is the weakest measured dimension. Short chains with a check at each step, rather than long autonomous runs you inspect at the end.
Pick the model by modality, not by size
Audio and video mean the 12B or an edge model. The 31B is not the most capable choice for every task, and assuming it is will quietly cost you a capability.
Fix the defaults before judging quality
Context stuck at 4,096 and layers silently on CPU both make the model look far worse than it is. Check these first when something seems disappointing.
Evaluate on your own task
Twenty or thirty representative examples from your real workload, run against two model sizes, tells you more than any leaderboard - including whether a smaller model is good enough.
Common questions
Is Gemma 4 safe to use in production?
For many applications, yes - with your own safeguards around it. The limitation isn't that the model is unusually risky; it's that open weights shift responsibility to you. There's no provider filtering inputs or outputs, and Google's model card says explicitly that developers should implement safety measures matched to their product.
Treat "we deployed the weights" as the start of the safety work rather than the end of it.
What's the single biggest limitation?
Agentic reliability. Gemma 4 31B ranks #129 of 134 on independent agentic composites despite strong knowledge and reasoning scores. If your product depends on the model executing multi-step tool-using workflows autonomously, this is the one that will actually hurt.
Does it hallucinate more than other models?
There's no evidence it's unusually prone to it, and Google documents the risk in the standard terms. What compounds it here is the January 2025 cutoff - over eighteen months of staleness means more questions fall outside what it knows, and a fluent model filling that gap sounds exactly like a model that knows the answer.
Will these limitations be fixed?
Some already improved: the July 2026 refresh materially helped tool calling and vision defaults without a version change. Others are structural - a 31B model won't reach frontier reasoning by patching, and the knowledge cutoff only moves with retraining. Ecosystem gaps like the variant licences could change at any time, so re-check rather than assuming.
Should these limitations put me off?
Not on their own. Every model has a page like this; most just don't publish one. The relevant question is whether these particular limits intersect your particular task.
For summarising, extraction, translation, classification, scoped coding and on-device work, Gemma 4 is genuinely strong and free to own. For autonomous agents and frontier reasoning, look elsewhere or route those cases out.
How do I test for these myself?
Build a small evaluation set from your own workload - twenty to thirty examples covering the cases you actually care about, including the awkward ones. Run it against two model sizes and read the outputs rather than only scoring them. That surfaces task-specific limitations no public benchmark will.
Know the limits, then decide.
See the full benchmark picture, or check what it's genuinely strong at.