Google、次世代マルチモーダルモデル「Gemma 4」発表

\authfootnotetext

1See Contributions and Acknowledgments section for full author list. Please send correspondence to gemma4report@gmail.com.

Abstract

We introduce Gemma 4, a new generation of open-weight, natively multimodal language models in the Gemma model family. Designed to advance compute efficiency and reasoning, the Gemma 4 model suite features dense and Mixture-of-Experts architectures, ranging from 2.3B to 31B parameters. Alongside improved vision and audio encoders for all model sizes, we propose a unified, encoder-free architecture for our 12B model, which ingests raw audio and image patches. Furthermore, we integrate a thinking mode, enabling Gemma models to generate reasoning traces prior to responding. We improve inference speed, memory, and compute efficiency, as well as long-context abilities through critical design choices. Gemma 4 establishes a leap in performance across STEM, multimodal, and long-context benchmarks, and rivals larger, frontier open models in human-rated tasks.

1 Introduction

The rapid evolution of large language models has driven the need for open-weight models with strong multimodal understanding, reasoning, and computational efficiency. Building upon the foundations of its predecessors Gemma Team, [2024a, b, 2025a], we introduce Gemma 4, the most capable and efficient generation in the Gemma model family to date. Gemma 4 offers natively multimodal architectures, capable of seamlessly processing text, images, and audio while achieving frontier-level performance on highly complex reasoning tasks.

The Gemma 4 family is built to serve a variety of on-device hardware. The model suite includes both dense architectures (2.3B, 4.5B, 12B, and 31B parameters) and a Mixture-of-Experts Jacobs et al., [1991, MoE] variant with 3.8B activated and 26B total parameters.

We introduce several architectural and methodological innovations:

• •

Thinking mode for advanced reasoning: We introduce a thinking mode [OpenAI, 2024] to Gemma 4 models. By outputting a reasoning trace before the response, models demonstrate improved capabilities in reasoning-heavy domains such as mathematics and coding.

• •

Long-context efficiency: Extended contexts lead to a memory explosion in the KV cache. We conserve a 5:1 ratio of local sliding window to global self-attention (4:1 for the 2.3B model) and use pp-RoPE [Barbero et al., 2025] as positional encoding. Combined with KV cache sharing [Shazeer, 2019] and the reuse of keys as values in global layers [Kayyam et al., 2026], these optimizations reduce the global KV cache footprint by up to 37.5%.

• •

Compute efficiency: We release an autoregressive multi-token prediction (MTP) drafter head [Li et al., 2024] designed for speculative decoding [Leviathan et al., 2023] to improve the decoding speed of our models.

• •

Memory efficiency: We provide quantized versions of our models trained with quantization-aware training [Jacob et al., 2018, QAT] to reduce their parameter memory footprint and latency with minimal impact on quality.

• •

Encoder-free architecture: Gemma 4 models have frozen vision and audio encoders. We introduce a unified encoder-free architecture for the 12B model, which projects raw 40ms audio chunks and image patches into the LLM embedding space, alleviating the need for separate encoders and reducing memory fragmentation.

In this technical report, we outline the different model architectures across model sizes as well as the pre-training and post-training recipe of Gemma 4. Through comprehensive benchmarks and human evaluations such as Arena Chiang et al., [2024], we demonstrate that Gemma 4 operates at a level comparable to larger, frontier open-source models across text, image, and audio modalities. We release the Gemma 4 models under an Apache 2.0 license, empowering developers and researchers everywhere to build upon, customize, and extend these capabilities.

2 Model Architecture

Gemma 4 models follow a decoder-only Transformer architecture Vaswani et al., [2017]. Our models have pre-norm and post-norm with RMSNorm Zhang and Sennrich, [2019], and QKNorm Henry et al., [2020].

Dense and MoE: The Gemma 4 family of models comprises dense architectures, with effective 2.3B (E2B), effective 4.5B (E4B), 12B and 31B parameters, as well as an MoE model with 3.8B activated parameters for 26B total parameters (26B-A4B).

E2B and E4B use per-layer embeddings as in Gemma 3n Gemma Team, [2025b], making them 2.3B and 4.5B effective out of 5B and 8B total parameters respectively.

Long-context efficiency: Our local to global attention ratio patterns follow Gemma Team [2025a], that is, 4-to-1 local attention blocks for E2B and 5-to-1 for the rest. We improve memory efficiency by re-using keys as values in the global attention layers (except in E2B and E4B), i.e. , values=keys\text{values}=\text{keys}. We encode position with pp-RoPE with p=0.25p=0.25 on global attention layers and with RoPE on local attention layers, effectively reducing the global KV cache by 37.5%. The RoPE frequencies are set to 1M and 10k on global and local attention layers, respectively. Finally, we share the KV cache with ratios of 20/35 and 18/42 for the E2B and E4B model.

2.1 Vision modality

E2B and E4B Gemma models come with a 150M vision encoder, while larger models use a 550M encoder (except for the unified 12B). Both are Vision Transformers Dosovitskiy et al., [2021, ViT] with a patch size of 16, whose architectural differences are detailed in Table 10 in Appendix.

Our vision encoders support variable aspect ratios (see Figure 2 and Algorithm 1) and incorporate both axial 2D-RoPE Heo et al., [2024] with non-causal attention and 2D absolute positional embeddings. We restrict the maximum number of tokens, NmaxN_{\text{max}} to the values 70,140,280,56070,140,280,560 and 11201120 (see Algorithm 1 for implementation details).

2.2 Audio modality

E2B and E4B Gemma models use a 305M audio encoder that processes audio in 40ms chunks with Mel filterbank inputs. The encoder architecture is based on the Universal Speech Model Zhang et al., [2023, USM], consisting of two downsampling convolution layers followed by twelve Conformer layers Gulati et al., [2020]. While the architecture remains similar to that of Gemma 3n, we reduce the number of parameters by 55% (from 680M to 305M). We do not use vector quantization; the LLM ingests the continuous representations produced by the audio encoder. As with the vision encoder, we keep weights frozen during pre-training.

2.3 Encoder-free architecture

Gemma 4 12B is trained from scratch based on a new, unified, and encoder-free model paradigm, replacing the separate vision and audio encoders with lightweight projection modules. For the vision modality, Gemma 4 12B takes in 48×\times48×\times3 RGB patches, but replaces the 550M vision encoder by a single large matmul (35M parameters). Spatial awareness is maintained by adding 2D coordinate-based positional embeddings directly to the patch representations before a final LayerNorm layer Ba et al., [2016].

For audio, the 305M USM-based conformer encoder is entirely discarded. Raw audio is segmented into 40ms chunks at 16kHz, resulting in 640-dimensional vectors per chunk. These are projected directly into the LLM embedding space. Since audio is a temporal sequence, it does not require additional positional encoding.

2.4 Pre-training

We follow a similar pre-training as Gemma 3.

Training data. Our pre-training dataset is a large-scale, diverse collection of data from a wide range of domains and modalities, including web documents, code, images, and audio (for E2B, E4B and 12B), with a cutoff date of January 2025.

Tokenizer.

We use the same tokenizer as Gemini Team [2025] that is, a SentencePiece tokenizer Kudo and Richardson, [2018] with split digits, preserved whitespace, and byte-level encodings.

The vocabulary has 262k entries.

Filtering. We filter data to decontaminate benchmarks, and to reduce the risk of unwanted or unsafe utterances and the risk of recitation.

2.5 Quantization-Aware Training

We provide quantized models and encoders in different formats along with the raw checkpoints.

Based on the most popular open source quantization inference engines (e.g. llama.cpp) as well as efficient hardware support, we focus on two sets of weight representations:

• •

mobile quantization: per-channel low bitwidth weight (mix of int2 and int4) and activation quantization (int8).

• •

Q4_0 quantization: blockwise quantization, often referred to as Q4_0.

In Table 3, we report the memory filled by raw and quantized models with and without a KV cache for a sequence of 32k tokens.

Furthermore, to enable stable inference in fp16, we introduce a scalar scale at each block in order to bound the activation ranges to fit fp16.

We also apply QAT to the image and audio encoders. On the 150M image encoder, quantizing activations and weights to 8-bit precision (W8A8) yields a 2×\times reduction in total forward-pass memory footprint (from 400 MB to 200 MB, including on-device compilation overhead) and a 44% reduction in on-device latency relative to Gemma 3n on newer hardware.

On the audio encoder, we further reduce activation precision to 8 bits and weight precision to {2,4,8}\{2,4,8\} bits, varying by layer cluster. Overall, we achieve a 78% reduction in on-disk footprint, from 390 MB in Gemma 3n to 87 MB in this version.

2.6 Multi-Token Prediction Drafter

We train a small autoregressive MTP drafter head with our models, used for speculative decoding. In our MTP procedure, the model’s last layer activations from the previous step and token embeddings are fed into the MTP head. The MTP head generates future tokens sequentially using a separate embedder and a 4-layer Transformer block that cross-attends to the KVs of the main model (Figure 1), thus eliminating the need for MTP prefill and supporting any draft length. The Transformer block has model dimension 256 for E2B and E4B, 1024 for 26B-A4B and 31B, three local, and one global attention layers.

Efficient MTP Decoding.

For the E2B and E4B drafters, we reduce the decoding overhead by replacing the projection operation to the entire vocabulary by a top-k operation on clusters of tokens. As a result, final matrix multiplication is reduced from d×262,000d\times 262,000 to d×4096d\times 4096 while preserving a similar acceptance rate.

2.7 Compute Infrastructure

We train our models with TPUv5p and TPUv6e as outlined in Table 2.

Each model configuration is optimized to minimize training step time.

For our larger models, we leverage Slice-Granularity Elasticity Gemini Team, [2025], which allows continuous training with fewer “slices” of TPU chips when there is a localized failure. This reconfiguration reduces the delay caused by interruptions from many minutes to a few seconds.

The optimizer state is sharded using an implementation of ZeRO-3 Ren et al., [2021].

For multi-pod training, we perform a data replica reduction over the data center network, using the Pathways approach of Barham et al. [2022].

We use the single controller programming paradigm of JAX Roberts et al., [2023] and Pathways, along with the GSPMD partitioner Xu et al., [2021] and the MegaScale XLA compiler XLA, [2019].

3 Instruction Tuning

Pre-trained models are turned into instruction-tuned models with a similar post-training approach as in Gemma 3. A significant difference is the addition of a thinking mode, where the model can output a reasoning trace before answering.

Data filtering. We carefully optimize the data used in post-training to maximize model performance. We filter examples that show certain personal information, unsafe or toxic model outputs, mistaken self-identification data, and duplicated examples. Including subsets of data that encourage better in-context attribution, hedging, and refusals to minimize hallucinations also improves performance on factuality metrics, without degrading model performance on other metrics.

PT versus IT formatting.

All models share the same tokenizer, with some control tokens dedicated to IT formatting.

A key difference is that PT models output an <eos> token at the end of generation, while IT models output <turn|> at the end of the generation. An example is given for IT in Table 11.

Fine-tuning either model type thus requires adding their respective end tokens. We detail how to activate thinking and how models handle function calling in Table 11.

4 Evaluation of final models

In this section, we evaluate the IT models over a series of automated benchmarks and human evaluations across a variety of domains, as well as static benchmarks such as MMLU Pro.

4.1 Human evaluation

We report the performance of our 31B and 26B-A4B models on Arena Chiang et al., [2024] in blind side-by-side evaluations by human raters against other state-of-the-art models.

We report Elo scores in Table 4.

Gemma 4 31B is the top open model in the dense category, and both Gemma 4 31B and 26B-A4B show performance equal to much larger open models.

CoVoST (CorpusBLEU ↑\uparrow )

Params

Size

ja →\to en

de →\to en

fr →\to en

es →\to en

it →\to en

ru →\to en

zh →\to en

AVG

Gemma 4 E2B

305M

87 MB

21.4

39.2

39.2

43.2

40.8

46.4