NVIDIA、テキスト知能を維持する統合型音声・言語モデル「Audex」を発表

NVIDIA has released Audex (Nemotron-Labs-Audex-30B-A3B), a unified audio-text large language model. It understands and generates both audio and speech. It also keeps the text intelligence of its backbone. The checkpoints, along with a smaller Audex-2B, are released under a noncommercial license.

Most multimodal models pay a text tax. When labs add audio or vision output, text benchmarks often drop. NVIDIA research team reports this even for speech-only output models. Audex is designed to avoid that regression.

TL;DR

Audex is a single 30B-A3B MoE model that handles audio in and out.

Audio inputs enter the text embedding space; audio outputs are treated like text tokens.

Text scores match the backbone, with small gains and small losses per benchmark.

Multi-stage SFT plus text-only Cascade RL avoids the usual multimodal text regression.

It is few of the open models that generate general audio beyond speech.

What is Audex?

Audex is a single Mixture-of-Experts (MoE) Transformer decoder. It has 30B total parameters and 3B activated per token. The backbone is Nemotron-Cascade-2-30B-A3B, a text-only MoE LLM. That backbone is a hybrid Mamba-Transformer with 52 layers. It uses 128 routable experts and 6 activated experts.

The design is deliberately simple. Audio inputs are encoded and projected into the text embedding space. Text tokens and quantized audio tokens are then treated uniformly during generation. There is no thinker-talker split and no stacked cascade of models.

Because the design stays simple, Audex runs on standard LLM stacks. These include Megatron-LM for training and vLLM for inference. It supports both an instruct mode and a thinking mode. Context length reaches 1M tokens.

How the Unified Design Works

Three components sit around the LLM backbone:

An audio encoder reads sound. Audex uses AF-Whisper from Audio Flamingo 3. It shares the Whisper Large-v3 architecture and handles 16kHz input.

Two-layer MLP adapters map audio features into the model dimension.

An extended vocabulary holds discrete audio output tokens. The original 131,072 tokens grow to 205,312.

Audex uses two codecs for output. Speech uses X-Codec2 at 50 tokens per second. It applies single-layer finite scalar quantization (FSQ) with a 65,536 codebook.

Non-speech sound uses X-Codec at 200 tokens per second. It uses four flattened residual vector quantization (RVQ) layers. Complex sound gets a larger token budget than speech. The interactive demo below computes these token counts for any duration.

Training

Audex needs no audio pretraining. It starts from the text-only SFT checkpoint. Training then adds capabilities stage by stage.

The multi-stage SFT curriculum runs in order: text SFT, audio warmup, audio generation, then audio understanding. During audio warmup, text token embeddings stay frozen. Unfreezing them degraded text quality in ablations.

NVIDIA research team also tested a single-stage recipe that mixes all data at once. That recipe broke long-context retrieval on NIAH. Multi-stage training avoided this, so it became the default.

After SFT, the research team applies text-only Cascade RL and multi-domain on-policy distillation (MOPD). Audio tasks show marginal or no regression after this text-only RL. Text scores improve at the same time.

The data mix is large. It combines 157.4B audio tokens and 320.5B text tokens. Tasks span ASR, AST, TTS, text-to-audio, and audio understanding.

Benchmark and Performance

On text, Audex tracks its backbone closely. It scores 86.4 on MMLU-Redux against the backbone’s 86.3. It even leads on IMO AnswerBench, 81.1 versus 79.3. Small drops appear on MMLU-Pro and GPQA-Diamond.

Audex also tops the text-only Qwen3.5-35B-A3B on several reasoning, alignment, and instruction-following benchmarks. The comparably sized Qwen3-Omni-30B-A3B-Thinking shows large reasoning drops versus its own backbone.

BenchmarkAudex 30B-A3BQwen3.5-35B-A3BQwen3-Omni-30B-A3B-Thinking

HMMT Feb2592.289.060.4

IMO AnswerBench81.174.859.9

LiveCodeBench v685.374.659.2

ArenaHard v281.665.455.1

IFBench (prompt)77.870.252.4

On speech recognition, Audex leads these open models. It records 6.82 average word error rate on the OpenASR leaderboard. That beats Step-Audio-R1.1-33B and Qwen3-Omni-30B-A3B-Thinking.

On audio understanding the picture is mixed. Audex leads open models on MMAU. It shows gaps on MMAR and MMSU versus the strongest audio LLMs. Audex also generates general audio, which the other leading open models cannot.

Audio benchmarkAudex 30B-A3BStep-Audio-R1.1-33BQwen3-Omni-30B-A3B-Thinking

MMAU75.673.675.4

MMAR63.269.866.4

MMSU63.474.170.2

Audio Entailment95.061.661.6

OpenASR (WER, lower is better)6.827.918.00

BigBenchAudio90.097.6not reported

Audex leads on MMAU, Audio Entailment, and OpenASR word error rate. It trails these open baselines on MMAR, MMSU, and BigBenchAudio.

Use Cases with Examples

Consider a multilingual call center. Audex can transcribe a German call and translate it to English. Its speech translation output lists the source language, transcript, then English translation.

Consider accessibility tooling. A developer can add fixed-voice text-to-speech to a reading app. The Seed-TTS-Eval English word error rate is a low 1.70.

Consider sound design or prototyping. A caption like “birds chirping in a forest” yields a 10-second clip. General audio generation uses an enhancement VAE for 48kHz output.

Consider a voice assistant. Speech-to-speech runs as a cascade, but one checkpoint serves every step. Audex scores 90.0 on BigBenchAudio.

Quick Start Example

Audex follows the ChatML template. The reference container is vLLM 0.20.0. Audio input decoding needs the audio extras.

Audio understanding, ASR, and translation share one audio question-answering format. The <sound> placeholder marks where the audio goes.

Copy CodeCopiedUse a different Browser

[

{

"id": "sample_0",

"sound": "/path/to/audio_0.wav",

"conversations": [

{"from": "human", "value": "<sound>\nDescribe the audio in detail."},

{"from": "gpt", "value": "N/A"}

]

}

]

The model card ships a vLLM audio-QA script for this input format.

Copy CodeCopiedUse a different Browser

add audio codecs, then run audio QA offline

python3 -m pip install "vllm"

python inference_scripts_vllm/audioqa_scripts/run_audioqa_vllm.py \

--model-path "$(pwd)/checkpoint_folder_full" \

--input-json ./inputs.json \

--output-jsonl ./results.jsonl \

--tensor-parallel-size 8

For audio understanding, the research team recommends top_p=0.9 and temperature=0.7. For recognition and translation, use greedy sampling. Generation tasks need classifier-free guidance, shown in the demo’s recipe tab.

Strengths and Weaknesses

Strengths

Marginal or no text regression versus its text-only backbone.

Single unified model, compatible with Megatron-LM and vLLM.

Among the strongest open models, only Audex generates general audio.

Leads Qwen3.5-35B-A3B on several reasoning and alignment tasks.

Weaknesses

The NVIDIA OneWay Noncommercial License limits commercial use.

Audio understanding shows gaps on MMAR and MMSU versus top audio LLMs.

Speech-to-speech is cascaded, not native full-duplex.

Reinforcement learning is text-only; audio-text RL is future work.

Check out the Paper and Model Weights. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us

The post NVIDIA Releases Audex (Nemotron-Labs-Audex-30B-A3B): A Unified Audio-Text LLM That Preserves the Text Intelligence of Its Backbone appeared first on MarkTechPost.