How to Install
Gemma 4
Installing the runtime, enabling your GPU, and fixing the errors that stop people on their first attempt. Downloading the weights is the easy half - this is the other one.
Looking for the weights themselves? See downloads
# 1. Install the runtime curl -fsSL https://ollama.com/install.sh | sh # 2. Check you're on 0.22.0 or newer ollama --version # 3. Pull and run ollama pull gemma4:12b ollama run gemma4:12b # 4. Confirm the GPU is doing the work ollama ps
≥ 0.22.0required≥ 5.5.0required≥ 3.10for vLLMFour ways to install, in order of effort
They're not competing - most people should start at the top and only move down when they hit a specific need.
One installer, one command. Handles quantisation, serving and an OpenAI-compatible API for you. Start here unless you have a reason not to.
A desktop application with a model browser and chat window. Nothing to configure and nothing to type.
Python library access when you're writing code against the model rather than chatting with it. Watch the version requirement.
High-throughput serving for real traffic, with continuous batching and tensor parallelism across multiple GPUs.
Installing Ollama
Use 0.22.0 or newer. The 0.21.x builds have known correctness problems with tool calling, and you'll waste an afternoon before you work out why.
# Homebrew brew install ollama # Or download the .dmg from ollama.com - that build includes the # MLX runner, which is meaningfully faster on Apple silicon. ollama --version # expect 0.22.0 or newer ollama pull gemma4:12b ollama run gemma4:12b # Apple silicon uses unified memory, so there's no separate VRAM to # configure - but the model competes with everything else you have open.
# Download and run the .exe installer from ollama.com # The service starts automatically and listens on localhost:11434 # In PowerShell: ollama --version ollama pull gemma4:12b ollama run gemma4:12b # NVIDIA GPU not being used? Update your driver first - this is the # single most common Windows problem. Then confirm with: nvidia-smi # Set environment variables permanently: setx OLLAMA_KEEP_ALIVE "-1"
# Official install script curl -fsSL https://ollama.com/install.sh | sh ollama --version ollama pull gemma4:12b # NVIDIA: confirm the driver is visible before blaming Ollama nvidia-smi # AMD: ROCm 6.x must be on PATH rocminfo # Persist environment variables with systemd sudo systemctl edit ollama # [Service] # Environment="OLLAMA_KEEP_ALIVE=-1" # Environment="OLLAMA_CONTEXT_LENGTH=32768" sudo systemctl restart ollama
# Already had Ollama installed? Upgrade before troubleshooting anything. # macOS brew upgrade ollama # Linux - re-run the install script, it upgrades in place curl -fsSL https://ollama.com/install.sh | sh # Windows - download and run the current .exe over the top # Then re-pull the model. Weights published before mid-July 2026 # have chat-template and tool-calling bugs that were fixed since. ollama pull gemma4:12b
Two settings almost everyone misses
Both fail silently. The model runs, produces output, and quietly performs far worse than it should - which is why people conclude Gemma 4 is disappointing when it's actually misconfigured.
Regardless of what the model supports. Gemma 4 handles 128K–256K, so the default throws away almost all of it - and long documents get silently truncated rather than erroring.
If Ollama doesn't detect your GPU cleanly it falls back to CPU without complaint. You get correct answers at a fraction of the speed.
Fixing both permanently
# Option A - a Modelfile, so the settings travel with the model # Save as: Modelfile FROM gemma4:12b PARAMETER num_ctx 32768 PARAMETER num_gpu 99 # Build your configured variant, then use that name from now on ollama create gemma4-32k -f Modelfile ollama run gemma4-32k # Option B - environment variable for context (applies to everything) export OLLAMA_CONTEXT_LENGTH=32768 # Option C - per session, inside the ollama prompt # /set parameter num_ctx 32768
num_gpu 99 simply means "offload as many
layers as will fit" - it isn't a literal layer count.
Confirm the GPU is actually being used
Worth thirty seconds. A model running on CPU still answers correctly, so nothing tells you it's happening except the speed.
Ask Ollama directly
# With a model loaded, in another terminal: ollama ps # The GPU column should name your device. # Empty or "100% CPU" means it isn't being used.
If it shows CPU, add PARAMETER num_gpu 99 via a Modelfile and check
your driver situation below.
Check the driver layer
# NVIDIA - should list your card and driver version nvidia-smi # AMD - ROCm 6.x must be installed and on PATH rocminfo # Apple silicon - nothing to check, Metal is always available
If nvidia-smi isn't found or errors, the problem is your driver, not
Gemma. Fix that first - nothing downstream will work.
LM Studio
If the command line isn't your thing, this is the whole installation.
Download and install
Get the installer for macOS, Windows or Linux from lmstudio.ai and run it. It's an ordinary desktop application.
Search for the model
Open the model browser, search gemma 4, and pick a size that fits your machine. LM Studio shows an estimated memory requirement next to each option and warns you if it won't fit.
Raise the context and check GPU offload
In the model's load settings, increase the context length from the default and confirm the GPU offload slider is at maximum. Same two settings as Ollama, just with sliders instead of parameters.
Python and Transformers
One version requirement causes the majority of failed Gemma 4 installs.
Gemma 4 requires transformers 5.5.0 or newer
Older versions don't know the architecture exists. If you see this error, you've found the reason:
# The error The checkpoint you are trying to load has model type `gemma4` but Transformers does not recognize this architecture. # The fix pip install -U "transformers>=5.5.0"
# A clean environment avoids most of the pain python -m venv .venv source .venv/bin/activate # Windows: .venv\Scripts\activate pip install -U "transformers>=5.5.0" torch accelerate # Verify before you write any code python -c "import transformers; print(transformers.__version__)" # Gated repos need a token - accept the licence on the model page first pip install -U "huggingface_hub[cli]" hf auth login
transformers>=4.56.0,<5,
but Gemma 4 needs 5.5.0+. Installing both gives you
vllm 0.19.0 requires transformers<5, but you have transformers 5.5.0. Either upgrade
transformers anyway and accept the pip warning - it generally works - or use a vLLM nightly wheel or
build from source, where the constraint has been relaxed. Check whether a patched release is out before
doing either.
vLLM for real traffic
Worth the setup cost once you're serving more than yourself. Python 3.10 or newer.
# Fresh environment, Python 3.10+ python -m venv .venv && source .venv/bin/activate # Use a nightly wheel or build from source until the transformers # pin is relaxed in a stable release pip install -U vllm pip install -U "transformers>=5.5.0" # Serve - flags that matter for Gemma 4 vllm serve google/gemma-4-26B-A4B-it \ --max-model-len 131072 \ --gpu-memory-utilization 0.90 \ --enable-auto-tool-choice \ --tool-call-parser gemma \ --tensor-parallel-size 2 # Then it speaks the OpenAI protocol on :8000 curl http://localhost:8000/v1/models
Tool calling needs two flags
--enable-auto-tool-choice and a matching --tool-call-parser. Without both,
the model can't call tools and any agentic workload silently does nothing.
Set max-model-len deliberately
vLLM pre-allocates KV cache based on this. Asking for the full 256K will consume memory you'd rather give to concurrent requests.
gpu-memory-utilization
0.90 is a reasonable default. Push it higher only if nothing else shares the card - and lower it if you hit out-of-memory errors under load.
What your machine needs
Match to memory, not disk. Add roughly 20–30% on top of the weights for the KV cache at moderate context lengths.
| Model | 4-bit | Full precision | Realistic hardware |
|---|---|---|---|
| E2B | 2.9 GB | 11.4 GB | Any modern laptop; phones via the mobile build |
| E4B | 4.5 GB | 17.9 GB | 8 GB VRAM, or an 8 GB MacBook |
| 12B | 6.7 GB | 26.7 GB | RTX 3060 12 GB and up, or 16 GB unified memory |
| 26B A4B | 14.4 GB | 57.7 GB | 24 GB card at 4-bit; ~48–52 GB uncompressed |
| 31B | 17.5 GB | 69.9 GB | 24 GB card at 4-bit; 64 GB+ or multi-GPU uncompressed |
CPU-only inference works for the smaller models and is genuinely slow for the larger ones - expect single-digit tokens per second on a 12B without a GPU.
Troubleshooting
Ordered roughly by how often each one actually happens.
| Symptom | Cause | Fix |
|---|---|---|
| "Transformers does not recognize this architecture" | transformers older than 5.5.0 | pip install -U "transformers>=5.5.0" |
| Model forgets earlier conversation | Context stuck at the 4,096 default | Set num_ctx to 32768 or use OLLAMA_CONTEXT_LENGTH |
| Very slow generation | Running on CPU, or layers spilling to it | Check ollama ps; add num_gpu 99; verify the driver |
| Out of memory | Model plus KV cache exceeds VRAM | Lower the context, use a 4-bit QAT build, or drop a model size |
| Tool calls never execute | Parser flags missing, or pre-July weights | vLLM: both tool flags. llama.cpp: --jinja. Then re-pull the model |
Raw <unused24> tokens in output |
Chat template not applied | Add --jinja on llama.cpp; update your runtime |
| Slow first response, then fine | Model unloaded between requests | OLLAMA_KEEP_ALIVE=-1 keeps it resident |
| pip reports a vllm/transformers conflict | vLLM 0.19.0 pins transformers < 5 | Use a vLLM nightly or build from source; or accept the warning |
| 403 or "gated repo" on download | Licence not accepted, or not logged in | Accept terms on the model page, then hf auth login |
| Nothing on localhost:11434 | Service not running | ollama serve, then curl http://localhost:11434 |
Install questions
Do I need a GPU at all?
No, but you'll want one for anything above E4B. CPU inference works and is genuinely slow - single-digit tokens per second on a 12B. The edge models are usable on CPU, which is rather the point of them. Apple silicon is a good middle ground, since unified memory means the GPU can address far more than a typical consumer graphics card.
Which install method should I actually use?
Ollama, unless you have a reason not to. It handles quantisation, model management and serving, and gives you an OpenAI-compatible endpoint for free. Move to vLLM when you're serving real concurrent traffic, to Transformers when you're writing code against the model, and to LM Studio if you'd rather not use a terminal.
Why is my model so much slower than the benchmarks?
In order of likelihood: it's on CPU rather than GPU, the context is set high enough to push layers off the card, or you're comparing against datacentre hardware.
Check ollama ps first. If the GPU column is empty, everything else is secondary.
Do I need to install CUDA myself?
For Ollama and LM Studio, no - they bundle what they need; you only need a current NVIDIA driver. For vLLM and PyTorch you'll want a CUDA-enabled build, which pip installs by default on Linux and Windows. AMD needs ROCm 6.x installed and on PATH.
Can I install on Windows without WSL?
Yes. Ollama and LM Studio both ship native Windows installers, and Ollama runs as a background service on localhost:11434. WSL is only worth the trouble if you specifically need vLLM, which targets Linux.
How do I completely uninstall it?
Remove models with ollama rm gemma4:12b - they're multi-gigabyte and
worth clearing deliberately. Then uninstall the application normally for your platform. Hugging Face
downloads live in ~/.cache/huggingface and won't be removed by uninstalling anything, so
delete that separately if you're reclaiming space.
Does installing Gemma 4 send anything to Google?
Inference is entirely local - your prompts never leave the machine, and you can confirm it by disconnecting the network after the download. The download itself contacts whichever registry you pulled from, and the runtime you installed may have its own update checks or telemetry, so check that application's settings if it matters to you.
Installed and running?
Try it in the browser playground, or check what your model size is actually capable of.