OpenRouter、画像認識 LLM への API 送信ガイドを公開

If you want a vision-capable Large Language Model (LLM) to read a screenshot, inspect a chart, or answer questions about a photo, you need to build the request body correctly. This guide covers image input only, meaning a model reading a picture you give it, not generating or editing one.

The pattern is simple. Send a chat message whose content array includes a text part and an image_url part to our image understanding endpoint. After that, the request shape stays the same, and you change only the model field to use any vision-capable model we support.

One model handles Optical Character Recognition (OCR) better, another reads UI screenshots more reliably, another reasons over charts more carefully. Because the input shape doesn’t change, you can test those differences without changing your integration.

This guide covers sending images to multimodal models, choosing between public URLs and base64 uploads, building multimodal RAG pipelines, and the practical limits that matter once you’re in production.

Tl;dr

We support multimodal image understanding through the Chat Completions API.

• Endpoint: POST /api/v1/chat/completions

• Body: a messages array where the user message’s content is a list of parts, one {"type": "text"} and one {"type": "image_url"}.

• Model: any slug with image input (e.g. anthropic/claude-opus-4.8, google/gemini-3-flash-preview). Swap it freely, since the request body is identical.

The basic request: attach an image to a chat call

The request is one user message with two parts in its content array, a text part and an image_url part. The rest of this guide builds on this request.

The message content array

Text-only chat sends content as a plain string. When you add an image, content becomes an array of typed objects instead:

{
  "role": "user",
  "content": [
    { "type": "text", "text": "What's in this image?" },
    { "type": "image_url", "image_url": { "url": "https://example.com/receipt.jpg" } }
  ]
}

Order matters here, so put the text part first. That’s how we parse the array. If your use case genuinely needs the image referenced before any text, move that framing into the system prompt instead of trying to reorder the content array.

base64 data URL vs. hosted image URL: when to use which

The image_url.url field accepts two shapes: a plain public HTTP(S) link, or a base64 data URL formatted as data:image/jpeg;base64,<encoded-bytes>. Which one to use depends on where the file already lives.

If the image is already hosted somewhere public, a CDN, an S3 bucket with a signed link, or your own server, pass the URL. The request stays small, and the provider fetches the bytes on its own.

If the image is local, or it shouldn’t have a public URL at all, such as a user’s uploaded ID or an internal document, encode it as base64 and put it in the request. The request gets larger and the upload takes longer, but the file only leaves your systems through the API call itself.

Base64 has a second advantage. Hosted URLs can fail because of access controls, regional blocks, or expired signed URLs. Those failures can’t happen when the bytes are already in the request. Either format supports PNG, JPEG, WebP, and GIF.

A runnable example in cURL, Python, and TypeScript

Same request, three languages. The only line that changes between models is MODEL.

cURL (hosted URL)

curl https://openrouter.ai/api/v1/chat/completions \
  -H "Authorization: Bearer $OPENROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "anthropic/claude-opus-4.8",
    "messages": [
      { "role": "user", "content": [
        { "type": "text", "text": "What total is on this receipt?" },
        { "type": "image_url", "image_url": { "url": "https://example.com/receipt.jpg" } }
      ]}
    ]
  }'

Python (local file → base64)

import base64, os, requests

def to_data_url(path: str, mime: str = "image/jpeg") -> str:
    with open(path, "rb") as f:
        b64 = base64.b64encode(f.read()).decode("utf-8")
    return f"data:{mime};base64,{b64}"

MODEL = "anthropic/claude-opus-4.8"  # swap this one string for any vision model

resp = requests.post(
    "https://openrouter.ai/api/v1/chat/completions",
    headers={"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}"},
    json={
        "model": MODEL,
        "messages": [{
            "role": "user",
            "content": [
                {"type": "text", "text": "What total is on this receipt?"},
                {"type": "image_url", "image_url": {"url": to_data_url("receipt.jpg")}},
            ],
        }],
    },
)
print(resp.json()["choices"][0]["message"]["content"])

TypeScript (local file → base64)

import { readFile } from "node:fs/promises";

const MODEL = "anthropic/claude-opus-4.8"; // change only this to swap models
const bytes = await readFile("receipt.jpg");
const dataUrl = `data:image/jpeg;base64,${bytes.toString("base64")}`;

const res = await fetch("https://openrouter.ai/api/v1/chat/completions", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.OPENROUTER_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: MODEL,
    messages: [{
      role: "user",
      content: [
        { type: "text", text: "What total is on this receipt?" },
        { type: "image_url", image_url: { url: dataUrl } },
      ],
    }],
  }),
});
const data = await res.json();
console.log(data.choices[0].message.content);

Choosing a vision model

Every vision model on OpenRouter takes the same request shape. To switch models, change the model field and leave everything else the same. This lets you compare models without changing your integration.

What makes a model vision-capable

Not every model on the catalog can read images. A model qualifies as a vision language model (VLM) when it pairs a text model with an image encoder, letting it take in pixels alongside tokens. On OpenRouter, you can check this directly: a model’s architecture lists image under input_modalities if it supports image input. Send an image_url part to a model that doesn’t list it, and the request will fail. Check the catalog first.

Cost, context window, and what each model is good at

The request shape is the same across these models, but the models behave differently. Price, context window, latency, and how well a model handles OCR, charts, or general scene understanding all vary. Test candidate models against your actual images before you commit to one.

*Prices and context windows shift as models change. Treat this table as illustrative and pull live figures from /models before you rely on them.*

Filtering the catalog to vision-capable models

Query the catalog instead of hard-coding a model list.

import requests

models = requests.get("https://openrouter.ai/api/v1/models").json()["data"]
vision = [m["id"] for m in models
          if "image" in m["architecture"]["input_modalities"]]
print(vision)  # models that accept image input

Because the request body is identical across all of them, you can pick any model from this list at request time. Sort it by price, context window, or whatever your app cares about, instead of hard-coding one model up front.

Sending multiple images and long documents

Multiple images in one request

You aren’t limited to one image per request. Add as many image_url parts to the content array as you need. This is how you handle before-and-after comparisons, multi-page scans, or a question that covers several charts at once:

{
  "role": "user",
  "content": [
    { "type": "text", "text": "Which chart shows higher Q4 revenue?" },
    { "type": "image_url", "image_url": { "url": "https://example.com/2025.png" } },
    { "type": "image_url", "image_url": { "url": "https://example.com/2026.png" } }
  ]
}

Practical limits: image count, resolution, and token cost

There’s no universal cap. Limits are set per provider and per model. A few images per request is usually fine, but check the specific model’s endpoint page before you send dozens at once. Every image adds tokens, so cost grows with both image count and resolution, especially when you send full scanned documents.

When to downscale or pre-crop before sending

A phone photo of a receipt is usually 4000 pixels wide. No model needs that much resolution to read the total at the bottom. Shrink the image to the smallest size where the text is still legible. If you already know which part of the image matters, crop to that region. Both steps reduce token cost. Cropping also tends to improve accuracy, because it removes visual content the model would otherwise have to process.

How image tokenization works

Images are split into patches, converted to embeddings, then processed as tokens, so higher resolution means more tokens and higher cost. Here is how that works. An image encoder, typically a Vision Transformer, cuts the image into a grid of fixed-size patches. Each patch is converted into an embedding, a vector that represents that slice of the image. Those embeddings are passed to the language model as tokens, mixed in with your text tokens. The model never sees raw pixels. It sees patch embeddings.

The consequence is simple. More pixels means more patches, and more patches means more tokens on your bill. Downscaling an image reduces the number of patches the encoder produces. Exactly how many tokens a given image costs varies by provider, so check the specific model’s endpoint page if you need precise numbers.

Multimodal RAG: retrieving over documents that contain images

Multimodal RAG indexes images, charts, and scanned pages alongside text, so retrieval can return visual content and pass it to a vision model at answer time. It solves this problem: most real documents aren’t pure text. A quarterly report’s key number might appear only inside a bar chart. A text-only retrieval pipeline relies on OCR, which often reads the chart incorrectly or skips it. If your index never captured that visual content, retrieval can’t find it later.

Indexing strategy A: summarize images to text, then embed

At index time, send every chart, figure, or scanned page through a VLM and ask for a text summary. Embed that summary with your normal text embedding model, and keep a pointer back to the original image. The advantage is that retrieval stays entirely text-based, so it drops into whatever vector store you already run. The trade-off is that retrieval quality depends on the summary. Anything the VLM leaves out at index time can’t be found later.

Indexing strategy B: native multimodal embeddings

Skip the summary step and embed the image directly, using a multimodal embedding model that puts images and text in the same vector space. A text query can then match an image directly. You keep more visual detail this way, but you need a multimodal embedding model in your stack and a vector store that can hold its output. Use strategy A when you want to reuse existing tooling and the documents are relatively simple. Use strategy B when the charts and figures carry detail a summary would likely lose.

Assembling the prompt: feed retrieved text and images into the VLM

The answer step is the same for both indexing strategies. Take your best-matching text and image chunks, then build a single content array with the question, the retrieved text, and the retrieved image_url parts:

def answer(question, retrieved):
    content = [{"type": "text", "text": question}]
    for r in retrieved:
        if r["type"] == "text":
            content.append({"type": "text", "text": r["text"]})
        else:  # image chunk
            content.append({"type": "image_url",
                            "image_url": {"url": r["url"]}})
    return requests.post(
        "https://openrouter.ai/api/v1/chat/completions",
        headers={"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}"},
        json={"model": "anthropic/claude-opus-4.8", "messages":
              [{"role": "user", "content": content}]},
    ).json()

A minimal end-to-end code sketch

The full pipeline works like this: parse the source document into text and image chunks, index each chunk with a summary or a multimodal embedding, retrieve the top matches for an incoming query, assemble the multimodal prompt shown above, and answer with a vision model. Only the parsing, indexing, and retrieval steps are specific to RAG. The final answer call is the same request shown at the top of this guide. If you need the output as structured data rather than prose, add tool calling.

What this approach is not best for

• Real-time video understanding. Don’t stream every frame into a chat request. You can send a few sampled frames through an image array, but that’s a workaround, not a video pipeline. Real video work needs deliberate frame sampling, hard limits on frame count, and latency handling this endpoint isn’t built for.

• Pixel-precise or dense small-text OCR at low resolution. A general-purpose VLM can miss characters that are very small in the source image. If every character has to be correct, upscale or crop first, or use a dedicated OCR pipeline instead.

• Generating or editing images. This guide covers image input only, meaning the model reading a picture you give it. Creating a new image or modifying an existing one uses a different request shape. See our image generation docs for that.

まとめ

To send an image to an LLM, use a content array with a text part and an image_url part. The request shape is the same whether the URL points to a hosted file or carries base64-encoded bytes. Change the model field and the same body works with every model that accepts image input. A multimodal RAG pipeline makes this same request at answer time, after retrieval has picked the right image to send.

Three things to remember:

• One request body works for every vision model. Nothing about the content array changes when you switch models. You only change one string.

• Choose URL or base64 based on where the image lives. For public, already-hosted images, use a URL. For local or private images, use base64. Either way, downscale before sending to keep token costs down.

• RAG adds retrieval in front of the same call. Index your charts and scans next to your text, retrieve the right ones, and send the chosen image to the model using the request shown at the start.

Browse vision-capable models in the catalog and swap between them without touching your integration.

Frequently asked questions

Can I send images to the API?

Yes. Add an image_url or base64 data URL part to the message content array alongside your text, and any vision-capable model can read it. The shape of the request doesn’t change based on which provider ends up serving it. Only the model field does.

Can you do RAG with images?

Yes. Index charts, tables, and scanned pages next to your text, either by summarizing images into text at index time or by embedding them directly with a multimodal embedding model. At query time, retrieve the relevant text and image chunks and send them to a vision model together in one request.

How do multimodal LLMs process images?

An image encoder breaks the image into fixed-size patches, turns each patch into an embedding, and passes those embeddings to the language model as tokens, mixed in with your text tokens. The model reasons over those embeddings, never the raw pixels.

How do multimodal LLMs tokenize images?

The pipeline runs patches to embeddings to tokens. A higher-resolution image produces more patches, which means more tokens and a higher cost. Downscaling before you send an image is the direct way to control that cost.

base64 or URL: which should I use?

If the image is already public and hosted somewhere, use the URL, which keeps the request small. If it’s local, private, or a provider can’t reliably fetch it from wherever it’s hosted, encode it as base64 instead.

How many images can I send in one request?

There’s no single universal number, since it depends on the provider and the model. A few images per request is generally safe, but check the model’s specific endpoint page before sending a large batch, since each image adds to both token count and cost.

Do all models support image input?

No. Only models that list image under input_modalities will accept an image_url part. Query the catalog to find out which models qualify rather than assuming any given model supports it.

How much do images cost in tokens?

It scales with resolution: more pixels produce more patches, and more patches means more tokens. Exactly how a provider counts those tokens varies, so check the model’s endpoint page if you need a precise cost estimate before scaling up.

Can I combine image input with tool calling or structured outputs?

Yes. Add tool calling to a vision request and the model returns typed JSON, like { "total": 42.10 }, instead of a sentence. This is the common pattern for extracting structured fields from receipts, forms, or scanned documents.