意味的類似性を超えて：NVIDIA NeMo Retrieverの汎用化可能なエージェント型検索パイプラインの紹介

Back to Articles Beyond Semantic Similarity: Introducing NVIDIA NeMo Retriever’s Generalizable Agentic Retrieval Pipeline

Upvote - Radek Osmulski radekosmulski-nvidia Follow nvidia Reza Esfandiarpoor rezaesfandiarpoor Follow nvidia Yauhen Babakhin ybabakhin Follow nvidia Gabriel de Souza Pereira Moreira gmoreira-nv Follow nvidia Bo Liu BoLiu Follow nvidia We are thrilled to announce that NVIDIA NeMo Retriever team has developed a new agentic retrieval pipeline that has officially secured the #1 spot on the ViDoRe v3 pipeline leaderboard. In addition, this exact same pipeline architecture achieved the #2 spot on the highly demanding, reasoning-intensive BRIGHT leaderboard.

In the rapidly evolving landscape of AI retrieval, many solutions are highly specialized, engineered to perform exceptionally well on specific, narrow tasks. However, real-world enterprise applications rarely have the luxury of perfectly curated, single-domain data. They require systems that can seamlessly adapt to a wide variety of challenges—from parsing complex visual layouts to executing deep logical reasoning.

That is why we prioritized generalizability in our design. Rather than relying on dataset-specific heuristics, we built an agentic pipeline that dynamically adapts its search and reasoning strategy to the data at hand. This allows us to deliver state-of-the-art performance across vastly different benchmarks without requiring any underlying architectural changes.

Here is a look at how we built it.

The Motivation: Why Semantic Similarity Isn't Enough

For years, dense retrieval based on semantic similarity has been the standard for finding information. However, as the applications of retrieval expand, finding relevant documents goes beyond semantic similarity alone. Complex document search requires reasoning skills, an understanding of real-world systems, and iterative exploration.

There is a fundamental gap: LLMs are great at thinking and reasoning but cannot process millions of documents at once. Conversely, retrievers can easily sift through millions of documents but possess limited reasoning skills. Agentic retrieval bridges this gap by creating an active, iterative loop between the LLM and the retriever.

How It Works: The Agentic Loop

Agentic retrieval pipeline overview

Our agentic retrieval pipeline relies on a ReACT architecture. Instead of a single "one-and-done" query, the agent iteratively searches, evaluates, and refines its approach.

The agent utilizes built-in tools like think

retrieve (query, top_k)

Generating better queries: The agent dynamically adjusts its search queries based on newly discovered information.

Persistent rephrasing: It continually rephrases queries until useful information is found.

Breaking down complexity: It translates complex, multi-part queries into multiple simpler queries with clear goals.

Finally, to synthesize the iterative discoveries, the agent calls a final_results

Engineering for Speed and Scale

Agentic workflows are notoriously slow and resource-intensive. To make this pipeline viable for leaderboard-scale evaluation, we had to rethink how the LLM agent and the retriever communicate.

Initially, the retriever was exposed to the agent via a Model Context Protocol (MCP) server—a natural choice, since MCP is designed precisely for giving LLMs access to external tools. But in practice, this architecture imposed a compounding tax on experiment velocity. Every run required spinning up a separate MCP server, loading the right dataset corpus into GPU memory, and orchestrating the lifecycle of both client and server. Each of these steps was an opportunity for silent misconfiguration or a server freeze under high-volume requests. The network round-trips added latency to every retrieval call, and the overall cognitive burden of managing the two-process setup made it significantly harder for other teams to adopt and iterate on the pipeline.

To resolve this, we replaced the MCP server with a thread-safe singleton retriever that lives in-process. The singleton loads the model and corpus embeddings once, protects all access with a reentrant lock, and exposes the same retrieve()

Generalization vs. Specialization Across Benchmarks

A common observation in modern retrieval evaluation is that solutions highly optimized for one specific type of task often experience a performance gap when applied to a completely different domain.

NeMo Agentic Retrieval (Opus 4.5 + nemotron-colembed-vl-8b-v2)

Dense retrieval (nemotron-colembed-vl-8b-v2)

INF-X-Retriever (INF-Query-Aligner + nemotron-colembed-vl-8b-v2)

INF-X-Retriever

INF-X-Retriever

NeMo Agentic Retrieval (Opus 4.5 + nemotron-reasoning-3b)

We placed #2 on the reasoning-intensive BRIGHT leaderboard with an NDCG@10 of 50.90. The #1 solution on that leaderboard, INF-X-Retriever, achieves an impressive 63.40. However, to test cross-domain adaptability, we benchmarked the INF-X pipeline (coupled with the same nemotron-colembed-vl-8b-v2

In contrast, our same agentic pipeline (pairing Opus 4.5 with nemotron-colembed-vl-8b-v2

This highlights a core strength of our approach: generalizability. Rather than relying on dataset-specific heuristics or a query-rewriter/aligner, our agentic loop naturally adapts its strategy to the dataset at hand, whether it requires multi-step logical reasoning or parsing complex visual layouts.

Ablation Studies: Open vs. Closed Models

Embedding Model

Average sec/query

Total input tokens (M)

Total output tokens (M)

Average retrieval calls

nemotron-colembed-vl-8b-v2

nemotron-colembed-vl-8b-v2

llama-nemotron-embed-vl-1b-v2

nemotron-colembed-vl-8b-v2

llama-nemotron-embed-vl-1b-v2

Embedding Model

Average sec/query

Total input tokens (M)

Total output tokens (M)

Average retrieval calls

llama-embed-nemotron-reasoning-3b

llama-embed-nemotron-reasoning-3b

llama-nemotron-embed-vl-1b-v2

llama-embed-nemotron-reasoning-3b

llama-nemotron-embed-vl-1b-v2

We conducted extensive ablations to understand the tradeoff between frontier closed models and open-weights alternatives:

Model Choice: On ViDoRe v3, swapping Opus 4.5 for the open gpt-oss-120b

Embeddings: Pairing the agent with specialized embeddings (nemotron-colembed-vl-8b-v2

llama-embed-nemotron-reasoning-3b

It's also interesting to note that the agent can close the gap between stronger and weaker embedding models. For example, on ViDoRe, the gap between the stronger nemotron-colembed-vl-8b-v2

llama-nemotron-embed-vl-1b-v2

llama-embed-nemotron-reasoning-3b

llama-nemotron-embed-vl-1b-v2

The Cost of Autonomy and What's Next

There is no free lunch. Agentic retrieval is more expensive and slower than standard dense retrieval. Looking at our ViDoRe v3 results, the agent averages 136 seconds per query and consumes roughly 760k input and 6.3k output tokens per query. (Note: the sequential latency is measured on a single A100 GPU with a single concurrent Claude API call—i.e., not searching multiple queries at the same time—to reflect true search time in real-world use cases).

However, we believe agentic retrieval is a highly viable approach for high-stakes, complex queries. Our immediate next steps focus on cost reduction: we are actively researching ways to distill these agentic reasoning patterns into smaller, specialized open-weight agents. By fine-tuning smaller models to orchestrate the think

Build Your Own Agentic Pipeline

While our leaderboard-topping runs explored combinations like Claude Opus and gpt-oss

llama-nemotron-embed-vl-1b-v2