MLPerf Inference、マルチターン・エージェント推論をベンチマークに追加へ

Introduction

The MLPerf Inference benchmark suite must evolve alongside AI deployment patterns. Early inference benchmarks focused on image classification, object detection, speech recognition, recommendation, and single-turn language generation. Those workloads remain important, but they no longer cover one of the fastest-growing ways large language models are used in production: multi-turn agentic inference.

For example, a coding assistant is far more complex than a single query. The agent will read an issue, inspect files, run commands, observe failures, edit code, and iterate potentially many times. Similarly, a workflow agent gathers customer information, calls tools, interprets results, asks follow-up questions, and continues until the task is resolved. In both cases, the workload is a trajectory: a sequence of dependent turns where each request includes the conversation history up to that point.

This changes the serving problem in four concrete ways:

Context grows across the trajectory, so prefill and KV-cache pressure increase over time.

KV-cache reuse is now a critical serving optimization for performance and efficiency.

Output lengths vary widely, from compact tool calls to long reasoning traces.

Turn dependencies make throughput a measure of closed-loop progress, not independent request rate.

Figure 1. Illustration of an agentic inference scenario involving multiple turns.

The Agentic Inference benchmark adds this class of workload to the MLPerf Endpoints framework. It keeps the general MLPerf measurement principles intact while defining the workload-specific pieces: model choice, dataset composition, multi-turn load generation, output validation, and constraints for optimizations such as prefix caching and speculative decoding.

Terminology

Turn: one client-issued user or tool request and the model response generated for that request.

Trajectory: the ordered sequence of turns for one task or simulated user; later turns include the accumulated conversation history.

Model selection

The benchmark needs long-context, thinking-capable LLMs that stress the serving behavior of agentic applications: growing context, KV-cache reuse, variable output lengths, and strict turn dependencies. For this benchmark, we chose Kimi K2.6 and Qwen3.6-35B-A3B. Kimi K2.6 brings state-of-the-art coding capability and a large model profile representative of leading agentic systems; Qwen3.6-35B-A3B is more compact and brings a new Gated DeltaNet (GDN) architecture and exceptional coding performance for its size. This gives the benchmark coverage across distinct serving behaviors, architectural choices, and speculative-decoding paths. Both models are evaluated with the same methodology and dataset. To avoid ambiguity, each model produces its own result; the two models are not run together or combined into a single score.

ModelKimi K2.6Qwen3.6-35B-A3B

ArchitectureMoE + MLAMoE + Gated DeltaNet/Attention

Params1T / 32B active35B / 3B active

Context262,144 tokens262,144 tokens

SettingsThinking; temp=1.0; top_p=0.95; preserve_thinkingtemp=1.0; top_p=0.95; top_k=20; presence=1.5; repetition=1.0; preserve_thinking

Spec decodingnvidia/Kimi-K2.6-Eagle3 headNative MTP within the model

Table 1. Model metadata and benchmark settings

Dataset and task selection

The benchmark dataset combines two agentic domains that exercise different infrastructure bottlenecks. Together, they contain 613 multi-turn trajectories: 113 agentic coding trajectories and 500 agentic workflow trajectories. Across the reference dataset, these trajectories contain 30,335 client-issued turns and 30,328 assistant turns. By using real collected traces for benchmarking, the workload simulates real-world behavior in the serving stack, including how speculative decoding and expert-rank balancing behave under realistic multi-turn traffic.

DomainScalePrimary stress

Agentic Coding113 traj.; 15,981 client turnsDeep trajectories; growing context; KV-cache capacity

Agentic Workflow500 traj.; 4,316 client turnsLarge shared prompt; prefix overlap; prefix-cache efficiency

Table 2. Dataset composition and mean token statistics

The agentic coding traces come from the DeepSWE dataset by DataCurve (datacurve.ai), which contains software engineering tasks built around repository-level bugs and feature requests. A typical trace begins with a user request describing an issue, followed by an assistant that investigates the repository through a series of bash commands. The agent searches files, reads code, runs tests, observes errors, edits the implementation, and iterates. These traces are deep: the median trajectory contains dozens of turns, and the conversation history grows steadily as command outputs, file contents, and test logs accumulate.

The Workato agentic workflow traces come from enterprise customer-support and orchestration scenarios. They represent interactions where a simulated customer asks for help and the agent uses tools to retrieve orders, track shipments, check policies, escalate cases, or resolve account questions. These trajectories are usually shallower than the coding traces, but they include a much larger shared system prompt with many tool definitions and business rules. The agentic workflow traces were provided by Workato, the agentic control and execution plane for the enterprise. The data are synthetic traces modeled on Workato’s production experience orchestrating customer support agents for enterprise customers.

The two domains are intentionally different:

Coding traces grow aggressively over many turns.

Workflow traces begin with a large common prefix and grow more slowly.

Coding stresses KV-cache capacity and long-context scheduling.

Workflow stresses shared-prefix reuse and routing locality.

The combined workload prevents systems from optimizing for only one agentic traffic shape and stresses context-aware routing.

Client Design

We introduced multi-turn support to MLPerf Endpoints so the benchmark can drive full end-to-end agentic workloads against standard serving stacks. Submitters simply need to spin up an OpenAI-compatible endpoint using vLLM, SGLang, TensorRT-LLM, or another serving framework, then point the client at that endpoint for full end-to-end benchmarking.

The client handles the multi-turn behavior:

Closed-loop replay: one active conversation issues one turn at a time and waits for the complete model response before the next turn.

Target concurrency: the load generator controls the number of active users or conversations without breaking turn dependencies.

Inter-turn delay: dataset-provided waits before tool or user turns preserve realistic pacing without counting delay as model serving latency.

Conversation-aware routing: a stable X-Session-ID header is sent for each trajectory so routers can preserve KV-cache locality.

Cache salting: deterministic salt markers around the system prompt allow valid within-trajectory reuse while blocking invalid cross-trajectory reuse to ensure the benchmark is representative of production workloads.

Deterministic prompt reconstruction: future prompts are built from the pre-recorded dataset rather than from live model output, which keeps performance runs reproducible while still measuring generated outputs.

Generated token cache clearing: In order to enable fair benchmarking of all platforms, the generated tokens are cleared out of the KV cache by introducing a whitespace character – ensuring that the performance is independent of the system that was used for generating the traces.

Performance metrics

The primary performance chart is a Pareto curve where each point is a fixed-concurrency benchmark result. Submitters need to submit a full Pareto curve as defined in the MLPerf Endpoints rules.

Y-axis: output tokens per second per system, measuring aggregate serving throughput.

X-axis: output tokens per second per user, computed as output tokens per trajectory / (E2E time – total delays).

The two axes represent complementary views of the serving system. The x-axis, output tokens per second per user, captures how quickly an individual agent progresses through its task; moving right means faster task completion. The y-axis, output tokens per second per system, captures the total work completed across all active agents; moving up means greater aggregate throughput and higher system utilization. As concurrency increases, the system may complete more total work while each individual agent progresses more slowly. The Pareto curve makes this tradeoff visible.

Figure 2. Chart illustrating agentic inference performance

Accuracy metrics

Accuracy is enforced as a three-level hierarchy so the Pareto curve measures usable agentic progress rather than responses that are shorter, lower quality, or easier to serve.

OSL mean value: prevents systems from gaining throughput by truncating responses or shifting the answer-length distribution. The check requires generated outputs to preserve the expected mean output sequence length for the workload.

Inline accuracy: catches behavior drift during the exact performance configuration being reported. It operates on the outputs produced by the performance run itself and compares them against ground-truth values preserved in the dataset, including coding tool-call action checks and workflow intent-code checks, so no separate inference pass is required.

Standalone accuracy: verifies task-level capability beyond turn-level similarity. The standalone evaluation will use 200 tasks from the SWE-bench Verified dataset to check whether the submitted model configuration can solve representative tasks end to end.

All three levels must be run for every point on the Pareto curve, and each score must meet a pre-defined threshold. Thresholds are defined separately for each level and for both Kimi K2.6 and Qwen3.6-35B-A3B.

Accuracy levelKimi K2.6Qwen3.6-35B-A3B

Inline accuracy63.08%55.86%

OSL per-turn mean range[404,494][355,434]

Standalone accuracy76.5%67.0%

Table 3. Tentative accuracy values, subject to change.

Reference implementation

Submitters only need to spin up an OpenAI-compatible server using vLLM, SGLang, TensorRT-LLM, or another serving framework, then point the MLPerf Endpoints client at that endpoint to run the full benchmark. The MLPerf Endpoints Agentic Inference README includes the example configs, command lines, and setup steps.

Conclusion

Agentic inference is one of the fastest growing applications of AI today and it’s far more complicated than existing single-turn text generation. Agentic workflows have serving patterns with unique constraints: growing context, strict turn dependencies, tool-mediated delays, shared prefixes, long-tail trajectories, and output quality checks that must run on the same configuration used for performance.

The Agentic Inference benchmark brings those constraints into MLPerf in a reproducible form. By combining deep coding trajectories with shared-prefix workflow trajectories, it measures whether serving systems can make progress for real multi-turn users while still using hardware efficiently.

As agentic applications become a larger share of production LLM traffic, the industry needs benchmarks that measure the systems work that actually matters: preserving cache locality, scheduling long contexts, maintaining output quality, and balancing aggregate throughput with per-user progress. This benchmark is a step toward that measurement standard. If your team is building serving infrastructure for multi-turn agents, join MLCommons to access the reference implementation and submit results

References

MLPerf Endpoints repository.

MLPerf Endpoints Inference Rules and Pareto collection methodology.

MLPerf Endpoints Agentic Inference reference implementation.

SWE-bench Verified benchmark.

Workato agentic workflow and enterprise task definitions.

DeepSWE Dataset by Datacurve.

The post Agentic Inference for MLPerf Inference appeared first on MLCommons.