Jetson上でのオープンソース視覚言語モデル(VLM)のデプロイ

Back to Articles Deploying Open Source Vision Language Models (VLM) on Jetson

Upvote - Mitesh Patel mitp Follow nvidia Johnny Nuñez Cano johnnynv Follow nvidia Raymond Lo raymondlo84-nvidia Follow nvidia Vision-Language Models (VLMs) mark a significant leap in AI by blending visual perception with semantic reasoning. Moving beyond traditional models constrained by fixed labels, VLMs utilize a joint embedding space to interpret and discuss complex, open-ended environments using natural language.

The rapid evolution of reasoning accuracy and efficiency has made these models ideal for edge devices. The NVIDIA Jetson family, ranging from the high-performance AGX Thor and AGX Orin to the compact Orin Nano Super is purpose-built to drive accelerated applications for physical AI and robotics, providing the optimized runtime necessary for leading open source models.

In this tutorial, we will demonstrate how to deploy the NVIDIA Cosmos Reasoning 2B model across the Jetson lineup using the vLLM framework. We will also guide you through connecting this model to the Live VLM WebUI, enabling a real-time, webcam-based interface for interactive physical AI.

Supported Devices:

Jetson AGX Thor Developer Kit

Jetson AGX Orin (64GB / 32GB)

Jetson Orin Super Nano

JetPack Version:

JetPack 6 (L4T r36.x) — for Orin devices

JetPack 7 (L4T r38.x) — for Thor

Storage: NVMe SSD required

~5 GB for the FP8 model weights

~8 GB for the vLLM container image

Create NVIDIA NGC account(free) to download both the model and vLLM contanier

Jetson AGX Thor

Jetson AGX Orin

Orin Super Nano

nvcr.io/nvidia/vllm:26.01-py3

ghcr.io/nvidia-ai-iot/vllm:r36.4-tegra-aarch64-cu126-22.04

ghcr.io/nvidia-ai-iot/vllm:r36.4-tegra-aarch64-cu126-22.04

FP8 via NGC (volume mount)

FP8 via NGC (volume mount)

FP8 via NGC (volume mount)

Max Model Length

256 tokens (memory-constrained)

GPU Memory Util

The workflow is the same for both devices:

Download the FP8 model checkpoint via NGC CLI

Pull the vLLM Docker image for your device

Launch the container with the model mounted as a volume

Connect Live VLM WebUI to the vLLM endpoint

Step 1: Install the NGC CLI

The NGC CLI lets you download model checkpoints from the NVIDIA NGC Catalog.

Download and install

mkdir -p ~/Projects/CosmosReasoning cd ~/Projects/CosmosReasoning # Download the NGC CLI for ARM64 # Get the latest installer URL from: https://org.ngc.nvidia.com/setup/installers/cli wget -O ngccli_arm64.zip https://api.ngc.nvidia.com/v2/resources/nvidia/ngc-apps/ngc_cli/versions/4.13.0/files/ngccli_arm64.zip unzip ngccli_arm64.zip chmod u+x ngc-cli/ngc # Add to PATH export PATH="$PATH:$(pwd)/ngc-cli"

Configure the CLI

You will be prompted for:

API Key — generate one at NGC API Key setup

CLI output format — choose json

org — press Enter to accept the default

Step 2: Download the Model

Download the FP8 quantized checkpoint. This is used on all Jetson devices:

cd ~/Projects/CosmosReasoning ngc registry model download-version "nim/nvidia/cosmos-reason2-2b:1208-fp8-static-kv8"

This creates a directory called cosmos-reason2-2b_v1208-fp8-static-kv8/

Step 3: Pull the vLLM Docker Image

For Jetson AGX Thor

docker pull nvcr.io/nvidia/vllm:26.01-py3

For Jetson AGX Orin / Orin Super Nano

docker pull ghcr.io/nvidia-ai-iot/vllm:r36.4-tegra-aarch64-cu126-22.04

Step 4: Serve Cosmos Reasoning 2B with vLLM

Option A: Jetson AGX Thor

Thor has ample GPU memory and can run the model with generous context length.

Set the path to your downloaded model and free cached memory on the host:

MODEL_PATH="$HOME/Projects/CosmosReasoning/cosmos-reason2-2b_v1208-fp8-static-kv8" sudo sysctl -w vm.drop_caches=3

Launch the container with the model mounted:

docker run --rm -it \ --runtime nvidia \ --network host \ --ipc host \ -v "$MODEL_PATH:/models/cosmos-reason2-2b:ro" \ -e NVIDIA_VISIBLE_DEVICES=all \ -e NVIDIA_DRIVER_CAPABILITIES=compute,utility \ nvcr.io/nvidia/vllm:26.01-py3 \ bash

Inside the container, activate the environment and serve the model:

vllm serve /models/cosmos-reason2-2b \ --max-model-len 8192 \ --media-io-kwargs '{"video": {"num_frames": -1}}' \ --reasoning-parser qwen3 \ --gpu-memory-utilization 0.8

Note: The --reasoning-parser qwen3

--media-io-kwargs

Wait until you see:

INFO: Uvicorn running on http://0.0.0.0:8000

Option B: Jetson AGX Orin

AGX Orin has enough memory to run the model with the same generous parameters as Thor.

Set the path to your downloaded model and free cached memory on the host:

MODEL_PATH="$HOME/Projects/CosmosReasoning/cosmos-reason2-2b_v1208-fp8-static-kv8" sudo sysctl -w vm.drop_caches=3

1. Launch the container:

docker run --rm -it \ --runtime nvidia \ --network host \ -v "$MODEL_PATH:/models/cosmos-reason2-2b:ro" \ -e NVIDIA_VISIBLE_DEVICES=all \ -e NVIDIA_DRIVER_CAPABILITIES=compute,utility \ ghcr.io/nvidia-ai-iot/vllm:r36.4-tegra-aarch64-cu126-22.04 \ bash

1. Inside the container, activate the environment and serve:

cd /opt/ source venv/bin/activate vllm serve /models/cosmos-reason2-2b \ --max-model-len 8192 \ --media-io-kwargs '{"video": {"num_frames": -1}}' \ --reasoning-parser qwen3 \ --gpu-memory-utilization 0.8

Wait until you see:

INFO: Uvicorn running on http://0.0.0.0:8000

Option C: Jetson Orin Super Nano (memory-constrained)

The Orin Super Nano has significantly less RAM, so we need aggressive memory optimization flags.

Set the path to your downloaded model and free cached memory on the host:

MODEL_PATH="$HOME/Projects/CosmosReasoning/cosmos-reason2-2b_v1208-fp8-static-kv8" sudo sysctl -w vm.drop_caches=3

1. Launch the container:

docker run --rm -it \ --runtime nvidia \ --network host \ -v "$MODEL_PATH:/models/cosmos-reason2-2b:ro" \ -e NVIDIA_VISIBLE_DEVICES=all \ -e NVIDIA_DRIVER_CAPABILITIES=compute,utility \ ghcr.io/nvidia-ai-iot/vllm:r36.4-tegra-aarch64-cu126-22.04 \ bash

1. Inside the container, activate the environment and serve:

cd /opt/ source venv/bin/activate vllm serve /models/cosmos-reason2-2b \ --host 0.0.0.0 \ --port 8000 \ --trust-remote-code \ --enforce-eager \ --max-model-len 256 \ --max-num-batched-tokens 256 \ --gpu-memory-utilization 0.65 \ --max-num-seqs 1 \ --enable-chunked-prefill \ --limit-mm-per-prompt '{"image":1,"video":1}' \ --mm-processor-kwargs '{"num_frames":2,"max_pixels":150528}'

Key flags explained (Orin Super Nano only):

--enforce-eager

Disables CUDA graphs to save memory

--max-model-len 256

Limits context to fit in available memory

--max-num-batched-tokens 256

Matches the model length limit

--gpu-memory-utilization 0.65

Reserves headroom for system processes

--max-num-seqs 1

Single request at a time to minimize memory

--enable-chunked-prefill

Processes prefill in chunks for memory efficiency

--limit-mm-per-prompt

Limits to 1 image and 1 video per prompt

--mm-processor-kwargs

Reduces video frames and image resolution

--VLLM_SKIP_WARMUP=true

Skips warmup to save time and memory

Wait until you see the server is ready:

INFO: Uvicorn running on http://0.0.0.0:8000

Verify the server is running

From another terminal on the Jetson:

curl http://localhost:8000/v1/models

You should see the model listed in the response.

Step 5: Test with a Quick API Call

Before connecting the WebUI, verify the model responds correctly:

curl -s http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "/models/cosmos-reason2-2b", "messages": [ { "role": "user", "content": "What capabilities do you have?" } ], "max_tokens": 128 }' | python3 -m json.tool

Tip: The model name used in the API request must match what vLLM reports. Verify with curl http://localhost:8000/v1/models

Step 6: Connect to Live VLM WebUI

Live VLM WebUI provides a real-time webcam-to-VLM interface. With vLLM serving Cosmos Reasoning 2B, you can stream your webcam and get live AI analysis with reasoning.

Install Live VLM WebUI

The easiest method is pip (Open another terminal):

curl -LsSf https://astral.sh/uv/install.sh | sh source $HOME/.local/bin/env cd ~/Projects/CosmosReasoning uv venv .live-vlm --python 3.12 source .live-vlm/bin/activate uv pip install live-vlm-webui live-vlm-webui

git clone https://github.com/nvidia-ai-iot/live-vlm-webui.git cd live-vlm-webui ./scripts/start_container.sh

Configure the WebUI

Open https://localhost:8090

Accept the self-signed certificate (click Advanced → Proceed)

In the VLM API Configuration section on the left sidebar: Set API Base URL to http://localhost:8000/v1

Click the Refresh button to detect the model

Select the Cosmos Reasoning 2B model from the dropdown

Select your camera and click Start

The WebUI will now stream your webcam frames to Cosmos Reasoning 2B and display the model’s analysis in real-time.

Recommended WebUI settings for Orin

Since Orin runs with a shorter context length, adjust these settings in the WebUI:

Max Tokens: Set to 100–150 (shorter responses complete faster)

Frame Processing Interval: Set to 60+ (gives the model time between frames)

Troubleshooting

Out of memory on Orin

Problem: vLLM crashes with CUDA out-of-memory errors.

Free system memory before starting:

sudo sysctl -w vm.drop_caches=3

Lower --gpu-memory-utilization

Reduce --max-model-len

Make sure no other GPU-intensive processes are running

Model not found in WebUI

Problem: The model doesn’t appear in the Live VLM WebUI dropdown.

Verify vLLM is running: curl http://localhost:8000/v1/models

Make sure the WebUI API Base URL is set to http://localhost:8000/v1

If vLLM and WebUI are in separate containers, use http://<jetson-ip>:8000/v1

Slow inference on Orin

Problem: Each response takes a very long time.

This is expected with the memory-constrained configuration. Cosmos Reasoning 2B FP8 on Orin prioritizes fitting in memory over speed

Reduce max_tokens

Increase the frame interval so the model isn’t constantly processing new frames

vLLM fails to load model

Problem: vLLM reports that the model path doesn’t exist or can’t be loaded.

Verify the NGC download completed successfully: ls ~/Projects/CosmosReasoning/cosmos-reason2-2b_v1208-fp8-static-kv8/

Make sure the volume mount path is correct in your docker run

Check that the model directory is mounted as read-only (:ro

In this tutorial, we showcased how to deploy NVIDIA Cosmos Reasoning 2B model on Jetson family of devices using vLLM.

The combination of Cosmos Reasoning 2B’s chain-of-thought capabilities with Live VLM WebUI’s real-time streaming makes it ideal to prototype and evaluate vision AI applications at the edge.

Additional Resources

Cosmos Reasoning 2B on NVIDIA Build: https://build.nvidia.com/nvidia/cosmos-reason2-2b

NGC Model Catalog: https://catalog.ngc.nvidia.com/

Live VLM WebUI: https://github.com/NVIDIA-AI-IOT/live-vlm-webui

vLLM container for Jetson Thor: https://ghcr.io/nvidia-ai-iot/vllm:r36.4-tegra-aarch64-cu126-22.04

vLLM container for Jetson AGX Orin, and Orin Super Nano: https://nvcr.io/nvidia/vllm:26.01-py3

NGC CLI Installers: https://org.ngc.nvidia.com/setup/installers/cli

Open Models supported on Jetson: https://www.jetson-ai-lab.com/models/

Getting started with Jetson: https://www.jetson-ai-lab.com/tutorials/

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here. Tap or paste here to upload images Comment · Sign up or log in to comment