NVIDIA、NeMo Switchyard で AI エージェントのワークロードを複数モデル間でルーティング可能に

Building an AI agent does not end with choosing a single model. Each model has its own strengths, weaknesses, and cost profile, which can shift from one workload to another—or even within the same workload. For example, an agentic task may need classification for one step, reasoning for the next, and a smaller model for routine follow-up tasks. Sending every request to the largest model can increase cost and latency, while sending every request to a smaller model can reduce quality on complex tasks.

Model routing addresses this challenge by orchestrating specialized and frontier models so that each task uses the model best suited to each task. NVIDIA NeMo Switchyard makes this complex engineering problem practical for agent workloads, so developers can route work across models without rebuilding their applications around each provider or model choice.

At runtime, a router evaluates each request and its available context, then sends the work to the model that best suits the task’s requirements, constraints, and policies. Depending on the workload, this system of models may improve accuracy and reduce cost compared with using the most capable model for every request.

NeMo Switchyard provides a library for applying multiple routing approaches. This post explores how NeMo Switchyard enables developers to apply a system-of-models approach and build more efficient, controllable agents better suited for real AI workflows.

*Video 1. Learn how NVIDIA NeMo Switchyard uses live signals to route each agent workflow step across a developer-configured pool of models*

How model routers make decisions

Consider a system of models performing a computer-use task measured by the Terminal-Bench Hard benchmark (Figure 1). While DeepSeek V4 has the highest overall accuracy in this example, it isn’t the best model for every task group. For instance, Kimi K2.6 is better suited to the ML and RL task groups, while Qwen3.5 397B A17B is preferable for math and science. The remaining six task groups should use DeepSeek V4. This approach can also be applied at the individual-task level or across phases of solving a single task.

Cost and completion time further complicate the decision, as shown in Figure 2. Each model has its own costs associated with running or accessing the model, as well as its own verbosity profile, relating not just to tokens but also tool calls.

Building a router requires considering signals from various sources. Each routing algorithm helps determine where to derive those signals.

Broadly, effective routing decisions rely on signals from three areas:

• Model capabilities: Which model(s) can solve the task correctly.

• Model cost profile: The latency and cost associated with each model.

• Infrastructure: System-level signals that enable reliable and seamless handoffs.

To understand the capability and cost signals:

• Look at the request itself. A router can use the classification to match requests based on topic or estimated difficulty. For example, a classifier can identify the topic of a query, match it to a model in the model pool, and route the query appropriately. An embedding model or feature crafter can be used to extract features from the query.

• Look at the model states. A router can examine the model logprobs, cascades, assess agentic trace, model’s residual stream, attention matrices, leverage, etc.

• Look at the system. Pricing, latency, load, and agent-specific signals, including errors, are options. These signals can be used to evaluate model routers or as real-time routing signals.

Importantly, a router must consider not only *which* signals to use, but also *when* *and where* to evaluate them. For instance, for a multi-turn agent task, the router may route each full request to a specific model or route at each step. The whole system may share a pool, or sub-agents may use specialized model pools for the tasks.

The answers to these questions depend on several factors, including the use case, deployment complexity, error tolerance, latency, or throughput constraints. The system also requires infrastructure for a seamless and invisible handoff to the agent/user.

NeMo Switchyard solves these challenges with an intelligent orchestration layer that supports multiple routers. Developers can also bring their own routing algorithms or customization data to NeMo Switchyard.

How NeMo Switchyard enables routing

Routing algorithms produce signals that inform routing decisions across models with different strengths. The system also needs infrastructure that can take a router’s answer, send the request to the chosen model, and carry the response back to the application.

This starts with NeMo switchyard-libsy, the provider-agnostic SDK behind NeMo Switchyard. It represents requests, defines the models available to a system, and manages calls to the selected model. Each model target has a semantic name, while the client behind it maps that name to the provider endpoint and model ID. This separation keeps the routing logic independent of a specific provider.

NeMo Switchyard can carry routing state across an agent’s session when a policy requires it. It can also retain information from earlier turns, such as tool results or an affinity decision, and make that context available for later routing decisions. A route can remain stateless when that history isn’t needed.

This separation is important because model deployments change. A team may update a model, move it to another endpoint, or use a different provider without changing the routing integration. NeMo Switchyard can make a model call through the target client or return the call to the host application. This gives an agent runtime, inference platform, or gateway control over how requests are served while retaining the same routing contract.

The NeMo Switchyard server is a reference for making routing available through common APIs, to simulate an LLM gateway. It accepts OpenAI, Anthropic, and Responses API requests, translates them into the internal NeMo Switchyard request format, and returns the expected response format. It also records the selected model, decision rationale, token usage, latency, and call outcomes, so teams can inspect a running route.

Routing algorithms in NeMo Switchyard

With the infrastructure in place, the next step is looking at the routing approaches that use it to make decisions. NeMo Switchyard offers both tuning-free and tunable routers.

Tuning-free routers

NeMo Switchyard includes several tuning-free routers that make decisions without training on workload-specific data, including the LLM classifier, stage router, and escalation router.

LLM classifier

The LLM classifier uses an LLM as a judge to select a candidate LLM and maintains session affinity with that model across later turns. This avoids repeatedly reclassifying work that has not materially changed throughout the arc of an agent solving a task.

This approach fits headless and domain-specific systems. A team can route coding, mathematical, or healthcare work tasks to selected model targets, while NeMo Switchyard supplies the routing and state-management pieces.

Stage router

A coding agent moves through different stages. Early on, it explores the codebase and recovers from errors. Later it settles into a more mechanical implementation. These stages require different levels of model capability, which the stage router uses to make routing decisions.

For each turn, the stage router examines recent tool activity to decide how much model capability the agent needs. Severe errors, repeated unproductive work, or prolonged exploration push the turn toward the capable model. Steady writes and edits, especially once tests are passed, favor the efficient model. If the signals are inconclusive, the router can consult an LLM judge before falling back to its configured default.

Escalation router

Escalation routing starts each conversation with a lower-cost model. An LLM judge monitors the progress of the task, turn by turn, and moves the session to a more capable model when it detects sustained difficulty.

This approach is designed for multi-turn agent workloads in which a smaller model can handle routine work but may need support after repeated errors, loops, or drift, extending the LLM classifier routing approach from static to adaptive.

Tunable routers

Tunable routers build on this routing foundation by replacing fixed heuristics with signals learned from real-world workload data. Rather than determining which model is most appropriate based on the request text, a tunable router can learn to predict how likely each candidate model is to correctly answer a request.

Prefill router

During training, the prefill router extracts the LLM’s residual stream to estimate the complexity of the query. A shared-trunk MLP uses signals from the residual stream and maps them to accuracy labels for each LLM in the routing pool.

At inference time, the prefill states act as ‌input to the router, and the shared trunk predicts the likelihood that each LLM will successfully complete the task or answer the query.

The router can then apply a policy that blends predicted accuracy with cost, latency, or other deployment constraints. Each candidate model receives a score, and the request is routed to the model with the best tradeoff for that workload. Figure 5 shows that routing is not just about picking the strongest model. It is about choosing the model most likely to meet the required quality level at the right cost.

Improving agent efficiency with NeMo Switchyard

NVIDIA is working with partners across the agent, model, and enterprise application ecosystem to bring NeMo Switchyard model routing into existing developer workflows without a separate setup. These collaborations include:

• Agent workflows: Coding-agent workflows with Cognition, easy-to-configure Hermes Agent model routing with Nous Research, financial software engineering workflows with Ramp, and model-routing evaluation with LangChain.

• Application and infrastructure integrations: LLM application stack as a plug-in with LiteLLM; AI gateway, governance, and API traffic management with Kong; Claude model routing with Classmethod; enterprise automation and connectivity with Boomi Agent Garden.

• Industry-specific AI agents: Formal verification workflows with Cadence in ChipStack AI Super Agent and EDA agent workflows with Siemens.

LangChain benchmarked NeMo Switchyard using its internal deep agents evaluation suite, which includes 145 multi-turn agentic tasks that reflect production workloads, such as customer support dialogue under policy constraints, on-call incident investigation, and multi-step workflow automation across messaging, issue tracking, and email. The suite evaluates tool use, multi-step retrieval, filesystem operations, and long-context summarization, with scenarios drawn from τ²-bench airline, Berkeley Function Calling Leaderboard, FRAMES, and Nexus. Across five runs, routing requests between NVIDIA Nemotron 3.5 Lightning and Claude Opus 4.8 with the escalation router delivered a 74% cost reduction compared with a frontier-only baseline across five runs, sending just 7% of calls to the frontier model, at a measured ~6-point accuracy tradeoff.

Cognition implemented the NeMo Switchyard staged-routing methodology in Devin Desktop and deployed it to NVIDIA internal users for real-world testing. On FrontierCode Main, Cognition’s benchmark for production-grade coding tasks, the implementation routed between Opus 5 and Kimi K2.7. It delivered near-frontier performance, achieving 50.6% at a $3.11 mean cost—within 2.8 percentage points of Opus 5 accuracy at approximately 28% lower mean cost. Together, the benchmark and internal deployment provide a practical case study in model-neutral and adaptive agents, showing how the complementary strengths of different models can be applied dynamically as a task evolves.

Developers can start with partner integrations that bring NeMo Switchyard into familiar agent tools, frameworks, and gateways, or build custom routing into their own agents and LLM gateways using the NeMo Switchyard GitHub instructions.

Orchestration is here to stay

Model routing enables systems of specialized and frontier models to work together, delivering results that are greater than the sum of their parts. However, building a useful and production-ready routing system remains a very difficult engineering challenge.

NeMo Switchyard is fully open source and integrates with technology you already use. Get started with NeMo Switchyard on GitHub, where you can create, test, and contribute routing algorithms tailored to your specific use cases.

As AI systems increasingly combine models, routing is essential for selecting the right model for each task while balancing efficiency, quality, and cost.

*Stay up-to-date on*NVIDIA AI* by subscribing to *NVIDIA news* and following NVIDIA AI on *LinkedIn*, *X*, *Discord*, and *YouTube*.* *Visit the *developer page* for resources to get started. Explore open Nemotron models and datasets on *Hugging Face* and *Blueprints* on *build.nvidia.com. *And* *engage with *Nemotron livestreams*, *tutorials*, and the developer community on *NVIDIA forums* and *Discord*.*