Zyphra が Zamba2-VL を公開：ハイブリッド Mamba2–Transformer 型ビジョン言語モデルが初トークン生成時間を約 10 倍短縮

Zyphra has released Zamba2-VL, a family of open vision-language models. The release covers three sizes: 1.2B, 2.7B, and 7B parameters. Each model is built on the Zamba2 hybrid SSM–Transformer backbone.

Vision-language models (VLMs) read images and text together. They answer questions about charts, documents, and photos. Most open VLMs use a dense Transformer as the language model. Zamba2-VL replaces that with a hybrid state-space design. The goal is competitive accuracy at lower latency.

What is Zamba2-VL

Zamba2-VL follows the now-standard LLaVA-style VLM template. A pre-trained vision encoder turns image patches into features. A lightweight MLP adapter projects those features into the language model’s space. The language model then reads an interleaved sequence of vision and text tokens. The models support single and multi-image understanding and grounding.

Zyphra pairs each Zamba2 backbone with the Vision Transformer from Qwen2.5-VL. That encoder was chosen for two specific properties. It uses 2D rotary position embeddings and native dynamic-resolution processing. A two-layer MLP adapter connects the encoder to the backbone.

imagehttps://www.zyphra.com/our-work/zamba2-vl

The Architecture

The Zamba2’s backbone is where the design diverges from typical VLMs. It is a hybrid of Mamba2 state-space layers and shared transformer blocks. The Mamba2 layers run in linear time with a fixed-size state. A small number of shared attention layers are interleaved between them. Each shared block carries a unique LoRA adapter at each layer.

The Mamba2 layers carry the bulk of computation cheaply. The shared attention layers preserve in-context retrieval that pure-SSM models give up. The hybrid trades full-attention expressivity against state-space efficiency.

Zamba2-VL uses the Mistral v0.1 tokenizer. It was trained on 100B tokens of vision-text and pure-text data. That data was sourced from open web datasets.

imagehttps://www.zyphra.com/our-work/zamba2-vl

Model Quality and Benchmarks

The research team evaluated Zamba2-VL across 14 benchmarks. These span chart, diagram, and document understanding. They also cover general perception, reasoning, and visual counting. All scores come from Zyphra’s evaluation harness, which is based on VLMEvalKit. The report compares against the Molmo2, Qwen3-VL, and InternVL3.5 families.

EvalZamba2-VL-2.7BInternVL3.5-2BQwen3-VL-2BMolmo2-4BQwen3-VL-4B

DocVQA (test)90.989.493.387.895.3

ChartQA (test)79.681.678.786.181.8

OCRBench73.683.484.162.084.1

CountBenchQA87.570.087.991.287.3

PixMoCount (test)82.532.855.787.089.2

MMMU (val)37.749.940.948.851.4

MathVista (mini)51.061.451.856.563.6

InternVL3.5-2B and Qwen3-VL-2B are similar in size. Molmo2-4B and Qwen3-VL-4B are larger.

The pattern is uneven and worth understanding. Counting is the strongest category. Zyphra reports Zamba2-VL-1.2B at 62.5 on PixMoCount. That compares with 32.8 for InternVL3.5-1B and 17.7 for PerceptionLM-1B. Document understanding also holds up, with DocVQA at 90.9 for the 2.7B model. The model lags larger baselines on knowledge-heavy reasoning, such as MMMU and MathVista.

Why Inference is Faster

Inference is where Zamba2-VL shows its main advantage. Transformer attention scales quadratically with sequence length. Multimodal inputs make sequences long very quickly. A single high-resolution image can add several thousand vision tokens. A short video clip can produce tens of thousands of tokens.

Zamba2-VL avoids the growing KV cache of attention. It inherits near-linear-time prefill and a fixed-size recurrent state. On a 32k-token prefill, it leads on the score-versus-TTFT plot. No Transformer VLM in the comparison matched its score at similar latency. The latency gap is at least an order of magnitude.

The efficiency advantage is largest at the 1.2B and 2.7B scales. That is the range targeted for on-device and edge deployment.

Use Cases With Examples

The practical question is where this fits. Document and form extraction benefits from the strong DocVQA results. Think invoice parsing or receipt digitization at scale. Retail and inventory counting maps to the PixMoCount and CountBenchQA strengths. Grounding support enables pointing to objects in product or UI images. On-device assistants benefit from the low time-to-first-token. The 1.2B model targets phones and edge boxes. Long visual inputs, like multi-page PDFs, gain most from linear-time prefill.

Getting Started

The three models live in the Zyphra Zamba2-VL collection on Hugging Face. Inference runs through Zyphra’s transformers fork, based on transformers v4.57.1. The optimized Mamba2 kernels need a CUDA GPU for good latency.

Install the fork and its core dependencies:

Copy CodeCopiedUse a different Browser

pip install "transformers @ git+https://github.com/Zyphra/transformers.git@zamba2-vl"

pip install qwen-vl-utils==0.0.2

pip install flash_attn

Optimized Mamba2 kernels need two more packages:

Copy CodeCopiedUse a different Browser

pip install --no-build-isolation "causal-conv1d @ git+https://github.com/Zyphra/z-causal-conv1d.git@zamba2-vl"

pip install --no-build-isolation "mamba-ssm @ git+https://github.com/Zyphra/mamba.git@zamba2-vl"

Then load the model and run a single-image query:

Copy CodeCopiedUse a different Browser

from transformers import Zamba2_VLForConditionalGeneration, Zamba2_VLProcessor

import torch

from PIL import Image

from qwen_vl_utils import process_vision_info

import requests

device = "cuda"

processor = Zamba2_VLProcessor.from_pretrained("Zyphra/Zamba2-VL-2.7B", temporal_patch_size=1)

model = Zamba2_VLForConditionalGeneration.from_pretrained(

"Zyphra/Zamba2-VL-2.7B",

device_map=device,

torch_dtype=torch.bfloat16,

attn_implementation="flash_attention_2",

)

url = "http://images.cocodataset.org/val2017/000000039769.jpg"

image = Image.open(requests.get(url, stream=True).raw)

question = "What do you see in the image? Give us some detail."

num_img_tokens = 3400

conversation = [

{"role": "user", "content": [

{"type": "image", "image": image,

"max_pixels": num_img_tokens * 28 * 28, "min_pixels": 10 * 28 * 28},

{"type": "text", "text": question},

]},

]

prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)

images, _ = process_vision_info(conversation)

inputs = processor(text=prompt, images=images, add_special_tokens=True, return_tensors="pt")

inputs = {key: value.to(device) for key, value in inputs.items()}

outputs = model.generate(**inputs, max_new_tokens=100)

print(processor.tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Swap the model ID for Zamba2-VL-1.2B or Zamba2-VL-7B to change scale.

Strengths and Weaknesses

Strengths:

First open VLM family on a fully open hybrid SSM–Transformer LLM, per Zyphra.

About an order of magnitude lower time-to-first-token than comparable Transformer baselines.

Strong visual counting and competitive document understanding.

Three sizes cover edge, mid, and 7B-class deployment.

Apache 2.0 license with public weights and working inference code.

Weaknesses and Challenges:

Released as a research artifact.

Lags larger models on knowledge reasoning like MMMU and MathVista.

Lower OCRBench than same-size Qwen3-VL and InternVL3.5.

Optimized kernels need a CUDA GPU; CPU paths are slow.

Deployment requires self-hosting from the released code.

Key Takeaways

Zamba2-VL ships at 1.2B, 2.7B, and 7B parameters under Apache 2.0.

The backbone pairs Mamba2 state-space layers with a few shared transformer blocks.

Time-to-first-token drops about an order of magnitude versus comparable Transformer VLMs.

Counting and document understanding are strengths; knowledge reasoning lags.

Weights and working inference code are public on Hugging Face and GitHub.

Marktechpost’s Interactive Explainer

#mtp-zamba2vl-explainer *{box-sizing:border-box!important;margin:0;padding:0}

#mtp-zamba2vl-explainer hr,#mtp-zamba2vl-explainer p:empty,#mtp-zamba2vl-explainer del,#mtp-zamba2vl-explainer s{display:none!important}

#mtp-zamba2vl-explainer{

background:#0d0d0d!important;color:#e8e8e8!important;border:1px solid #232323!important;border-radius:14px!important;

font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Helvetica,Arial,sans-serif!important;

max-width:880px;margin:24px auto;overflow:hidden;line-height:1.5;

}

#mtp-zamba2vl-explainer .z2-head{padding:26px 26px 18px;background:#0d0d0d!important;border-bottom:1px solid #1c1c1c!important}

#mtp-zamba2vl-explainer .z2-eyebrow{color:#76B900!important;font-size:11px;letter-spacing:.18em;font-weight:700;text-transform:uppercase}

#mtp-zamba2vl-explainer .z2-title{color:#fff!important;font-size:23px;font-weight:800;margin:10px 0 6px;line-height:1.25}

#mtp-zamba2vl-explainer .z2-sub{color:#9aa0a6!important;font-size:14.5px;max-width:680px}

#mtp-zamba2vl-explainer .z2-sizes{display:flex;gap:8px;margin-top:16px;flex-wrap:wrap}

#mtp-zamba2vl-explainer .z2-size{

background:#151515!important;color:#cfcfcf!important;border:1px solid #2a2a2a!important;border-radius:999px!important;

padding:7px 16px;font-size:13px;font-weight:700;cursor:pointer;font-family:inherit;transition:.15s

}

#mtp-zamba2vl-explainer .z2-size:hover{border-color:#3a3a3a!important}

#mtp-zamba2vl-explainer .z2-size.active{background:#76B900!important;color:#0d0d0d!important;border-color:#76B900!important}

#mtp-zamba2vl-explainer .z2-size-detail{

margin-top:14px;background:#121212!important;border:1px solid #222!important;border-left:3px solid #76B900!important;

border-radius:8px!important;padding:12px 14px;font-size:13.5px;color:#cdd0d3!important;min-height:44px

}

#mtp-zamba2vl-explainer .z2-tabs{display:flex;gap:4px;padding:14px 18px 0;background:#0d0d0d!important;flex-wrap:wrap}

#mtp-zamba2vl-explainer .z2-tab{

background:transparent!important;color:#9aa0a6!important;border:0!important;border-bottom:2px solid transparent!important;

padding:10px 14px;font-size:14px;font-weight:700;cursor:pointer;font-family:inherit

}

#mtp-zamba2vl-explainer .z2-tab:hover{color:#e8e8e8!important}

#mtp-zamba2vl-explainer .z2-tab.active{color:#76B900!important;border-bottom:2px solid #76B900!important}

#mtp-zamba2vl-explainer .z2-panel{display:none;padding:22px 26px 8px}

#mtp-zamba2vl-explainer .z2-panel.active{display:block}

#mtp-zamba2vl-explainer .z2-h{color:#fff!important;font-size:16px;font-weight:800;margin-bottom:4px}

#mtp-zamba2vl-explainer .z2-note{color:#8a9097!important;font-size:12.5px;margin-top:10px}

/* pipeline */

#mtp-zamba2vl-explainer .z2-pipe{display:flex;align-items:stretch;gap:0;margin:14px 0 4px;overflow-x:auto;padding-bottom:6px}

#mtp-zamba2vl-explainer .z2-stage{

background:#151515!important;color:#d6d6d6!important;border:1px solid #2a2a2a!important;border-radius:9px!important;

padding:11px 12px;font-size:12.5px;font-weight:700;cursor:pointer;white-space:nowrap;flex:0 0 auto;transition:.15s;text-align:center

}

#mtp-zamba2vl-explainer .z2-stage:hover{border-color:#3d3d3d!important}

#mtp-zamba2vl-explainer .z2-stage.active{border-color:#76B900!important;background:#16210a!important;color:#cde89a!important}

#mtp-zamba2vl-explainer .z2-arrow{display:flex;align-items:center;color:#3a3a3a!important;padding:0 6px;font-size:16px;flex:0 0 auto}

#mtp-zamba2vl-explainer .z2-stage-detail{

background:#121212!important;border:1px solid #222!important;border-radius:9px!important;padding:14px 16px;font-size:13.5px;color:#cdd0d3!important;margin-top:6px

}

#mtp-zamba2vl-explainer .z2-stage-detail b{color:#fff!important}

/* efficiency lab */

#mtp-zamba2vl-explainer .z2-lab{margin-top:6px}

#mtp-zamba2vl-explainer .z2-presets{display:flex;gap:7px;flex-wrap:wrap;margin:12px 0}

#mtp-zamba2vl-explainer .z2-preset{

background:#151515!important;color:#bdbdbd!important;border:1px solid #2a2a2a!important;border-radius:7px!important;

padding:6px 11px;font-size:12px;font-weight:600;cursor:pointer;font-family:inherit

}

#mtp-zamba2vl-explainer .z2-preset:hover{border-color:#76B900!important;color:#cde89a!important}

#mtp-zamba2vl-explainer .z2-slider{width:100%;margin:6px 0 2px;accent-color:#76B900;height:4px}

#mtp-zamba2vl-explainer .z2-tok{color:#fff!important;font-size:20px;font-weight:800;font-variant-numeric:tabular-nums}

#mtp-zamba2vl-explainer .z2-tok small{color:#8a9097!important;font-size:12.5px;font-weight:600;display:block;margin-top:2px}

#mtp-zamba2vl-explainer .z2-bars{margin-top:16px;display:flex;flex-direction:column;gap:14px}

#mtp-zamba2vl-explainer .z2-bar-row .z2-bl{display:flex;justify-content:space-between;font-size:12.5px;margin-bottom:5px}

#mtp-zamba2vl-explainer .z2-bl .z2-lbl{color:#cfcfcf!important;font-weight:700}

#mtp-zamba2vl-explainer .z2-bl .z2-val{color:#fff!important;font-weight:800;font-variant-numeric:tabular-nums}

#mtp-zamba2vl-explainer .z2-track{background:#1a1a1a!important;border:1px solid #242424!important;border-radius:6px!important;height:16px;overflow:hidden}

#mtp-zamba2vl-explainer .z2-fill{height:100%;width:0;border-radius:5px;transition:width .35s ease}

#mtp-zamba2vl-explainer .z2-fill.tf{background:#E0A050!important}

#mtp-zamba2vl-explainer .z2-fill.z2{background:#76B900!important}

#mtp-zamba2vl-explainer .z2-ratio{

margin-top:16px;background:#16210a!important;border:1px solid #2e4413!important;border-radius:9px!important;padding:12px 14px;

color:#cde89a!important;font-size:14px;font-weight:700;text-align:center

}

#mtp-zamba2vl-explainer .z2-ratio b{color:#76B900!important;font-size:18px}

#mtp-zamba2vl-explainer .z2-callout{

margin-top:14px;background:#101010!important;border:1px solid #222!important;border-radius:9px!important;padding:12px 14px;font-size:13px;color:#b9bdc2!important

}

#mtp-zamba2vl-explainer .z2-callout b{color:#fff!important}

/* benchmarks */

#mtp-zamba2vl-explainer .z2-bench-tabs{display:flex;gap:6px;flex-wrap:wrap;margin:12px 0 18px}

#mtp-zamba2vl-explainer .z2-bt{

background:#151515!important;color:#bdbdbd!important;border:1px solid #2a2a2a!important;border-radius:7px!important;

padding:6px 11px;font-size:12px;font-weight:700;cursor:pointer;font-family:inherit

}

#mtp-zamba2vl-explainer .z2-bt:hover{border-color:#3d3d3d!important}

#mtp-zamba2vl-explainer .z2-bt.active{background:#76B900!important;color:#0d0d0d!important;border-color:#76B900!important}

#mtp-zamba2vl-explainer .z2-brow{margin-bottom:11px}

#mtp-zamba2vl-explainer .z2-brow .z2-bl{display:flex;justify-content:space-between;font-size:12.5px;margin-bottom:4px}

#mtp-zamba2vl-explainer .z2-brow .z2-mname{color:#cfcfcf!important;font-weight:600}

#mtp-zamba2vl-explainer .z2-brow.lead .z2-mname{color:#cde89a!important;font-weight:800}

#mtp-zamba2vl-explainer .z2-brow .z2-score{color:#fff!important;font-weight:800;font-variant-numeric:tabular-nums}

#mtp-zamba2vl-explainer .z2-btrack{background:#1a1a1a!important;border:1px solid #242424!important;border-radius:5px!important;height:13px;overflow:hidden}

#mtp-zamba2vl-explainer .z2-bfill{height:100%;width:0;background:#3a3a3a!important;border-radius:4px;transition:width .45s ease}

#mtp-zamba2vl-explainer .z2-bfill.lead{background:#76B900!important}

/* footer / tagline */

#mtp-zamba2vl-explainer .z2-foot{

margin-top:14px;padding:18px 26px;background:#0a0a0a!important;border-top:1px solid #1c1c1c!important;

display:flex;align-items:center;gap:12px;flex-wrap:wrap;justify-content:space-between

}

#mtp-zamba2vl-explainer .z2-foot .z2-brand{font-size:13.5px;color:#9aa0a6!important}

#mtp-zamba2vl-explainer .z2-foot .z2-brand b{color:#76B900!important;font-weight:800}

#mtp-zamba2vl-explainer .z2-foot a{

color:#0d0d0d!important;background:#76B900!important;text-decoration:none!important;font-weight:800;font-size:12.5px;

padding:8px 15px;border-radius:7px!important;border:1px solid #76B900!important;white-space:nowrap

}

#mtp-zamba2vl-explainer .z2-line{height:1px!important;background:#1c1c1c!important;border:0!important;margin:8px 0}

@media (max-width:640px){

#mtp-zamba2vl-explainer .z2-head,#mtp-zamba2vl-explainer .z2-panel{padding-left:16px;padding-right:16px}

#mtp-zamba2vl-explainer .z2-tabs{padding-left:10px;padding-right:10px}

#mtp-zamba2vl-explainer .z2-title{font-size:19px}

#mtp-zamba2vl-explainer .z2-tab{padding:9px 9px;font-size:13px}

#mtp-zamba2vl-explainer .z2-foot{padding:16px;flex-direction:column;align-items:flex-start}

#mtp-zamba2vl-explainer .z2-pipe{padding-bottom:8px}

}

Interactive Explainer

Zamba2-VL: Hybrid SSM–Transformer Vision-Language Models

Open VLMs at 1.2B, 2.7B, and 7B that replace dense attention with a Mamba2 state-space + Transformer hybrid. Apache 2.0.

1.2B

2.7B

7B

How it works

Why it’s faster

Benchmarks

The pipeline (tap a stage)

Zamba2-VL follows the LLaVA-style template: vision encoder → adapter → language model.

Token-scaling lab

Drag the slider or pick a preset. Attention prefill scales O(n²); the Mamba2 layers scale O(n).

1 hi-res image (~3,400)

A few images (~12,000)

Short video (~24,000)

Long context (~32,000)

3,400 vision tokensabout one high-resolution image

Transformer attention — prefill compute1.0×

Zamba2-VL hybrid — prefill compute1.0×

Transformer KV cache — memory for contextgrows

Zamba2-VL recurrent state — memory for contextfixed

At this length, the hybrid uses about 1.0× less prefill compute

Measured claim: Zyphra reports near-linear-time prefill and a fixed-size recurrent state. On a 32k-token prefill, it reports roughly an order-of-magnitude lower time-to-first-token than the closest Transformer baseline.

Bars above illustrate O(n²) vs O(n) scaling, not measured latency.

Benchmark explorer — Zamba2-VL-2.7B vs baselines

Pick an eval. Green is Zamba2-VL-2.7B. Higher is better.

Source: Zyphra evaluation harness (VLMEvalKit). InternVL3.5-2B and Qwen3-VL-2B are similar in size; Molmo2-4B and Qwen3-VL-4B are larger.

Published by Marktechpost — AI/ML research, model releases, and developer tutorials for engineers and data scientists.

Read more on Marktechpost

(function(){

var root=document.getElementById("mtp-zamba2vl-explainer");

if(!root)return;

var $=function(s){return root.querySelector(s)};

var $all=function(s){return Array.prototype.slice.call(root.querySelectorAll(s))};

/* ---- size selector ---- */

var sizeInfo={

"1.2B":"Edge and on-device tier. Largest relative time-to-first-token advantage. Zyphra reports PixMoCount 62.5, versus 32.8 for InternVL3.5-1B and 17.7 for PerceptionLM-1B.",

"2.7B":"Mid-size generalist, based on the Zamba2-2.7B LLM. Strong visual counting and document understanding; lags larger models on knowledge-heavy reasoning (MMMU, MathVista).",

"7B":"Largest release. Zyphra reports it as competitive with the strongest 7\u20138B open-weight VLMs across nearly every category."

};

function setSize(s){

$all(".z2-size").forEach(function(b){b.classList.toggle("active",b.getAttribute("data-size")===s)});

$("#z2-size-detail").textContent=sizeInfo[s];

}

$all(".z2-size").forEach(function(b){b.addEventListener("click",function(){setSize(b.getAttribute("data-size"))})});

setSize("1.2B");

/* ---- tabs ---- */

$all(".z2-tab").forEach(function(t){

t.addEventListener("click",function(){

var name=t.getAttribute("data-tab");

$all(".z2-tab").forEach(function(x){x.classList.toggle("active",x===t)});

$all(".z2-panel").forEach(functio