Gemma 4 Technical Report: How an Open Model Fights Bigger Ones With "Small Params + Thinking + Lean Memory"
Gemma 4's throughline is one sentence: use three engineering tracks to push "capability" and "cost" in the favorable direction at the same time.
Track one lifts intelligence: thinking mode (reason first, then answer). Track two cuts cost: local/global attention ratio, p-RoPE, KV sharing, K=V, QAT quantization, MTP speculative decoding. Track three expands modality: native image/text/audio, plus a "no separate encoder" unified architecture for the 12B. Every model size, table, and Arena rank below is evidence for this one sentence.
Thinking Mode
The main engine behind big math/code/STEM gains; almost every default eval has it on.
Long Context + Quantization + Speculative Decoding
KV can't blow up as context grows; 31B goes from ~64GB down to ~19GB; decoding speeds up too.
Native Multimodal + Unified 12B
Most sizes use a frozen encoder; the 12B ingests image patches and audio slices directly — two fewer big weight blobs.
- Family: E2B (effective 2.3B), E4B (effective 4.5B), 12B, 26B-A4B MoE (~4B active), 31B dense. Apache 2.0.
- Arena Text (2026-06-19): 31B Elo 1451 — reportedly #1 open dense model; the higher ranks nearby are mostly hundred-B-to-trillion-scale MoEs.
- Vs. Gemma 3 27B: STEM/code gaps are huge, but in the tables 4th-gen defaults to thinking while 3rd-gen 27B is mostly non-thinking — this shouldn't be read as a pure architecture win.
- On-device narrative: E2B closes in on 27B-class capability on several fronts (per official framing); audio encoder disk footprint goes from 390MB to 87MB.
Which Question in the Open-Source Race Is This Answering
By 2026 the open leaderboards were often dominated by giant MoEs. Gemma 4 picked a different lane: dense small-to-mid size + thinking + deployment-friendly.
If you only remember "another 31B just shipped" from this report, you'll miss what it's actually contesting. At the time, the top of Arena was crowded with open models at 300B, 700B, 1T, even 1.6T total params — mostly MoEs. Gemma rewrote the story as:
So the report's "core evidence" isn't a single SOTA point — it's a combination: the dense 31B holding its own against bigger MoEs (human preference); small models still fitting on-device (memory table + audio encoder size); long context without brute-forcing KV (RULER / LOFT).
Pick a Size First, See Who It Was Designed to Be
Click the five model tabs below. Same family, very different jobs.
E2B: On-Device Entry Point, Effective ~2.3B
Uses per-layer embeddings (same lineage as Gemma 3n) — total params are larger, but the "effective" count is 2.3B. Ships with its own 150M vision + 305M audio encoder. The report compares it to Gemma 3 27B, arguing "roughly an order of magnitude fewer params, close on many fronts."
E4B: On-Device Workhorse, Effective ~4.5B
Same playbook — small encoders + aggressive quantization. On vision evals, the report writes that it's not below Gemma 3 27B on any listed item (e.g. MMMU Pro 52.6 vs 49.7, MATH-Vision 59.5 vs 46.0). Long context is already ahead of 27B on several fronts too.