How to Self-Host Gemma 4 & EmbeddingGemma 2
Self-hosting means running the models on servers you control and letting other people or programs use them. It gives you privacy and control, and it is usually not the cheapest option. This guide covers whether to do it, what hardware to use, how to serve both models, and how to keep the service safe.
For a team, a product or an internal tool · just for yourself? See run locally ·
Is self-hosting worth it?
A rented graphics card costs the same whether it is busy or idle. Hosted Gemma 4 costs cents per million tokens. For cost alone, hosted usually wins.
"Blended" assumes five input tokens for every output token, using the list prices on the pricing page. Card prices are RunPod's on-demand rates. The sum leaves out staff time, storage and spare capacity, and says nothing about whether one card can actually produce that many tokens. It is arithmetic, not a benchmark.
The good reasons to self-host
Data must stay in-house
Regulation, contracts or caution mean prompts cannot go to a third party.
You have fine-tuned it
Your own version of the model has to run somewhere you control.
Steady, heavy volume
Hardware you own or have reserved, kept busy around the clock.
Control
A fixed model version, your own limits, and no dependence on a provider's availability.
Choose a model and the hardware for it
The model has to fit in the graphics card's memory, with room left for every conversation in progress.
| Model | Memory, full precision | Memory, 4-bit | A workable card | vLLM-ready QAT build |
|---|---|---|---|---|
| Gemma 4 E2B | 11.4 GB | 2.9 GB | One 24 GB card | Yes |
| Gemma 4 E4B | 17.9 GB | 4.5 GB | One 24 GB card | Yes |
| Gemma 4 12B | 26.7 GB | 6.7 GB | One 48 GB card at full precision; 24 GB with the QAT build | Yes |
| Gemma 4 26B A4B | 57.7 GB | 14.4 GB | One 80 GB card | No (not published) |
| Gemma 4 31B | 69.9 GB | 17.5 GB | One 80 GB card, or two | Yes |
| Card to rent | Memory | Per hour | Always on, a month |
|---|---|---|---|
| RTX 4090 | 24 GB | $0.74 | $540 |
| L40S | 48 GB | $1.09 | $796 |
| A100 | 80 GB | $1.59 | $1,161 |
| H100 PCIe | 80 GB | $2.89 | $2,110 |
| H200 | 141 GB | $4.59 | $3,351 |
- The 26B A4B is the efficient pick. It uses only about 3.8B of its parameters for each token, so it does less work per answer than the 31B.
- Memory is not only the model. Each conversation in progress needs its own share. Longer context and more users both add to it.
- Start smaller than you think. The 12B fits modest cards and scores close to the 26B A4B on agent tasks.
- Loading takes minutes. Keep one copy running rather than starting from nothing when traffic arrives.
Memory figures from Google's model card; they cover the model only. "A workable card" follows the hardware list in the vLLM Gemma 4 recipes (one 24 GB+ card for E2B and E4B, one 80 GB card for the 26B A4B, one 80 GB card or two for the 31B); the 12B line is this site's reading of the memory figures. QAT builds from Google's collection; rental prices from RunPod, October 7, 2026. More on memory in the VRAM guide; scores on the benchmarks page.
Two ways to serve Gemma 4
Ollama, for a small team
One container, one command, and a server your colleagues' tools can call. The simplest route for a handful of users and internal tools.
vLLM, for real traffic
Built to serve many requests at once on Linux servers with NVIDIA, AMD or Google hardware. It speaks the same OpenAI-style format most tools expect.
# Ollama in Docker, serving on port 11434 docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama docker exec -it ollama ollama run gemma4:12b-it-qat # add --gpus=all to the first command to use an NVIDIA card
uv venv source .venv/bin/activate uv pip install -U vllm --pre \ --extra-index-url https://wheels.vllm.ai/nightly/cu129 \ --extra-index-url https://download.pytorch.org/whl/cu129 \ --index-strategy unsafe-best-match
# simplest launch, one graphics card
vllm serve google/gemma-4-26B-A4B-it \
--max-model-len 32768 \
--gpu-memory-utilization 0.90# text, images, thinking and tools
vllm serve google/gemma-4-26B-A4B-it \
--max-model-len 16384 \
--gpu-memory-utilization 0.90 \
--enable-auto-tool-choice \
--reasoning-parser gemma4 \
--tool-call-parser gemma4 \
--chat-template examples/tool_chat_template_gemma4.jinja \
--limit-mm-per-prompt.image 4 \
--async-scheduling# the 31B across two cards
vllm serve google/gemma-4-31B-it \
--tensor-parallel-size 2 \
--max-model-len 32768 \
--gpu-memory-utilization 0.90# vLLM's Gemma 4 image, in the usual vLLM Docker form
docker run --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-p 8000:8000 \
--ipc=host \
vllm/vllm-openai:gemma4 \
--model google/gemma-4-26B-A4B-it \
--max-model-len 32768 \
--gpu-memory-utilization 0.90vLLM commands are from the vLLM Gemma 4 recipes and the 26B A4B recipe, which gives 0.25.0 as the minimum vLLM version. The Docker example combines the recipes' vllm/vllm-openai:gemma4 image with the standard vLLM Docker form; this site has not run it, so check the recipe for the current image tag.
The recipes list audio for E2B and E4B, with uv pip install "vllm[audio]". Ollama details are on the Ollama page; other options in the runtime selector.
The vLLM settings that matter
| Setting | What it does |
|---|---|
--max-model-len | The longest conversation, in tokens, the server will accept. Smaller values leave more memory for more users at once. |
--gpu-memory-utilization | The share of the card vLLM may use. 0.90 is the value in the recipes. |
--enable-auto-tool-choice, --tool-call-parser gemma4 | Together these switch on tool calling. |
--reasoning-parser gemma4 | Returns the model's thinking in a separate field from the answer. |
--chat-template …gemma4.jinja | The chat template the recipes use with tools. |
--limit-mm-per-prompt | How many images or audio clips a request may contain. Set to zero for text-only use. |
--tensor-parallel-size | Splits one model across several cards. Needed when it does not fit on one. |
--kv-cache-dtype fp8 | Stores the conversation cache more compactly; the recipes put the saving at about 50%. |
--mm-processor-kwargs | Sets the default image detail, for example {"max_soft_tokens": 560}. |
From the vLLM Gemma 4 recipes; the plain-language descriptions are this site's. What thinking and tools do is under capabilities and features.
Calling your server
vLLM listens on port 8000 and accepts the OpenAI format, so existing tools need only a new address.
# pip install openai
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed-locally")
response = client.chat.completions.create(
model="google/gemma-4-26B-A4B-it",
messages=[{"role": "user", "content": "Summarise our refund policy in three points."}],
max_tokens=4096,
extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)
message = response.choices[0].message
if hasattr(message, "reasoning") and message.reasoning:
print(message.reasoning)
print(message.content)
- Thinking is set per request with
enable_thinking, and comes back in a separatereasoningfield when the reasoning parser is on. - Image detail can be 70, 140, 280, 560 or 1120 tokens an image; 280 is the default.
- Use Google's sampling settings unless you have a reason not to: temperature 1.0, top-p 0.95, top-k 64.
- Ollama uses a different port:
http://localhost:11434/v1.
Client pattern from the vLLM recipes. More examples are on the API page and in the prompt guide; coding tools on the coding page.
Self-hosting EmbeddingGemma 2
EmbeddingGemma 2 turns text, code, images, audio and video into vectors for search. It is far lighter to host than Gemma 4: 740M parameters, no hosted version from Google yet, and small enough to run without a graphics card.
# Ollama serves embeddings from the same server as chat
ollama pull embeddinggemma-2 # full model, 1.3 GB
ollama pull embeddinggemma-2:270m # text and code only, 378 MB
curl http://localhost:11434/api/embed -d '{
"model": "embeddinggemma-2",
"input": "What causes the northern lights?"
}'# pip install -U "sentence-transformers[image,audio,video]" transformers from sentence_transformers import SentenceTransformer model = SentenceTransformer("google/embeddinggemma-2") # index: run once over your documents, in batches doc_embs = model.encode(documents, prompt_name="Document") # search: run for every question query_emb = model.encode(question, prompt_name="SearchQuery") scores = model.similarity(query_emb, doc_embs)
Ollama names from Ollama's library; Python usage from Google's developer guide.
- Ollama is the quickest server, but it lists text and image input only.
- For audio and video, use Python. The sentence-transformers library covers every input type; put your own small web service around it.
- Other servers are named, not documented. Google's guide lists vLLM, SGLang, MLX, LM Studio and LiteRT, without commands. Check each for a working build.
- Plan storage. About 1.5 GB per million full-length vectors, or 250 MB at the shortest length.
- Do not use float16. Use bfloat16 or float32.
- Keep prompts consistent. Index and search with the matching task prompts, or results suffer.
Storage and float16 guidance from Google's guide and the model card. Task prompts are explained in the prompt guide; setup basics in how to use and the install guide. The model has no safety tuning; see safety for what that leaves to you.
Securing and watching the service
A model server has no login of its own. Anyone who can reach it can use it.
Keep it private
Bind the server to localhost or a private network. It should have no public address of its own.
Put a gateway in front
Sign-in, per-user limits and a cap on request size all belong in a layer you control.
Check content
Open models come with no provider moderation. Checks on what goes in and comes out are yours to add.
Watch it
Track waiting time, queue length, card memory and errors, and alert on unusual traffic. vLLM publishes metrics for this.
This section is this site's guidance. The safety tools Google offers are listed on the ethics and safety page; what the models get wrong is under limitations.
Launch checklist
Tick items off as you go. Nothing is saved or sent.
Setup
Operations
The checklist is this site's guidance, not a standard.
Is self-hosting Gemma 4 cheaper than an API?
Usually not. An always-on H100 at $2.89 an hour is about $2,110 a month, which buys roughly 16 billion tokens of the 31B at the cheapest hosted rate. Self-host for privacy and control.
Which model should I serve?
The 26B A4B for most workloads, the 31B when quality matters most, and the 12B when hardware is limited. See the comparison tool.
What do I need to run vLLM?
A Linux server with a supported graphics card and vLLM 0.25.0 or newer, according to the vLLM recipes. The recipes also list AMD, Intel and Google TPU options.
Can I serve it with Ollama instead?
Yes, for a small team. It is simpler to run; vLLM is built for many requests at once.
Is there a small build for vLLM?
Google publishes vLLM-ready QAT builds for E2B, E4B, 12B and 31B, but not the 26B A4B. See the QAT page.
Can I host both models on one server?
Yes. EmbeddingGemma 2 is small enough to sit beside Gemma 4, or on a separate machine without a graphics card.
Is it legal to offer Gemma 4 as a service?
The Apache 2.0 licence allows commercial use. Google also publishes a prohibited-use policy. For your own situation, take legal advice.
Not sure you need a server?
Compare the hosted prices first, or try the model on your own machine.