AI エージェントのツールインターフェース：MCP と CLI の重要性を問う

We ran a controlled 1-to-1 comparison of MCP vs. CLI interfaces on the same backend servers and tools across 50 long-horizon agentic tasks. What we found:

• CLI is not a better tool interface than MCP by default. When the underlying tools are identical, whether CLI outperforms MCP largely depends on the model and the task: CLI helps less capable models on retrieval-heavy tasks, but is neutral or worse on lighter, more targeted tasks. Also, giving the model both interfaces doesn't automatically make it route between them.
• Interface choice matters much less for the latest frontier models. Earlier models were sensitive to it, but Opus 4.8 and GPT-5.5 perform similarly across all of our interface setups.
• CLI helps mainly by compensating for tool-use errors. CLI reduces tool-use errors for less capable models; more capable models don't make those errors, so the bottleneck shifts to planning and coverage, which a better interface can't fix.
• Interface choice still affects cost, even when performance converges. While task success rate for latest frontier models converges across interface setups, the differences that remain are trajectory length, latency, and dollar cost.

Motivation

We set out to answer one question: when the backend operations are exactly the same, does the tool interface still change how well an agent performs?

The Model Context Protocol (MCP) has quickly become a common way to expose structured tools to agents, especially in enterprise-style environments where actions need to be typed, auditable, and permissioned. At the same time, recent work (e.g., Terminal Agents Suffice for Enterprise Automation and MCP is up to 32× more expensive than CLI) has suggested that terminal-style agents can be highly effective, matching or outperforming agents built around structured MCP interfaces.

But this comparison leaves an important question unresolved. In many systems, the terminal agent is not only using a different interface; it may also be operating against a different tool surface, with different API coverage, different documentation, and different opportunities for programmatic composition.

At the same time, other work has shown that MCP performance can be limited by the way tools are exposed: incomplete or ambiguous tool descriptions, large tool inventories, prompt bloat, and difficult tool selection. In other words, when an MCP agent underperforms, it is not obvious whether the problem is MCP as an interface, the specific MCP server implementation, or the size and quality of the tool surface.

Our main finding is that the interface can affect agent performance, but that sensitivity is shrinking quickly as model capability improves.

Experimental Setup: Same Tools, Different Interfaces

We built a total of 50 long-horizon agentic tasks across two realistic, enterprise-style environments, each containing thousands of artifacts and server entities designed to mimic a different type of real-world business.

Environment 1 is lighter, containing 8 servers and 113 tools over a modest data volume of ~2,000 artifacts, where tasks focus on precise lookups and targeted updates across different applications.

Environment 2 is high-cardinality and data-heavy, containing over 30,000 data artifacts across 8 servers and 140 tools, where tasks require broad retrieval, repeated inspection, and gathering evidence across many records.

The agent can use tools to inspect or modify the environment. Our setup allows each tool to be exposed in three different ways:

• As an MCP tool: the agent sees the operation as a structured MCP tool with a tool name, description, and parameter schema.
• As a CLI command: the agent sees the operation as a command-line interface.
• As both: the agent sees both the MCP tool and the CLI command for the same operation, and can choose which interface to use.

The key control is that the CLI command is generated from the MCP tool schemas. Every MCP tool has a corresponding CLI command with the same underlying backend capability from the server. Therefore, the agent is not getting a better tool in one form factor than another.

To keep the CLI realistic, we exposed it the way an agent would encounter one in practice: a lightweight skill pointing to the binary and its --help for discovery, rather than a full embedded command schema.

Tasks from both environments are intentionally hard for current frontier agents. Successful completion of a task usually requires 50 to 100 tool calls, multi-source retrieval, keeping track of intermediate state, long-horizon reasoning and information synthesis, and producing the correct final state rather than just a plausible text response.

Each of the 50 tasks comes with 11 to 49 (median 24) human-created rubric criteria. The final trajectory is evaluated against those criteria using an LLM judge.

Throughout, we use "less capable" and "more capable" to describe the agent along a single axis in two ways: model generation (Opus 4.6 vs 4.8, GPT-5.4 vs 5.5) and reasoning effort (GPT-5.4 with reasoning off vs reasoning on). We used Claude Code harness for Opus 4.6 and Opus 4.8, and OpenAI Agents SDK and the Responses API for GPT-5.4 and GPT-5.5. The reasoning effort is set to “medium” for all four models, except for the setup when we turned off reasoning on GPT-5.4.

With each model, we tested four interface conditions with at least 4 runs per task, per interface condition:

• MCP-only: the agent only sees MCP tools.
• CLI-only: the agent only sees CLI commands.
• Both: the agent can access every tool in both interfaces; CLI and MCP interface expose the same backend operations
• Hybrid: broad search/list operations are routed to CLI, while targeted reads, writes, and state-changing operations remain on MCP.

We found the main result to be conditional.

Less Capable Models Are More Sensitive to Interfaces

The clearest interface effects appear when the model is less capable: With Opus 4.6, interface choice had a clear conditional effect. On the data-heavy Environment 2 tasks, CLI improved task success over MCP: roughly from 8.5% to 17.7%. On the lighter Environment 1 tasks, the direction reversed: MCP was stronger than CLI, roughly from 7.3% to 11.8%. CLI was not uniformly better. It helped most where the task required broad retrieval, repeated inspection, and gathering evidence across many records.

We see the same pattern in GPT-5.4 when reasoning is turned off. With reasoning off, task success is close to 0, so the binary task-level pass metric is no longer a reliable differentiator, therefore we look at the percentage of rubric criteria that were passed for each task instead. CLI shows substantially more progress than MCP: roughly passing 9% more rubric criteria overall and 14% more on Environment 2 tasks alone.

The split between the two environments suggests that CLI is not inherently superior to MCP on all tasks. For less capable models, CLI helps especially on data-heavy tasks that require broader tool calls and retrieval. We think it is because CLI encourages the model to work more iteratively by trying scoped commands, inspecting outputs, and refining the search over multiple steps.

More Capable Models Flatten the Interface Gap

As model capability improves, the performance gap between MCP and CLI interfaces becomes much smaller.

With Opus 4.8, MCP-only, CLI-only, Both, and Hybrid all perform similarly on the full task set. GPT-5.5 shows a similar pattern: MCP, CLI, Both, and Hybrid all performed closely, with a modest edge to CLI and Hybrid. For GPT-5.4 with reasoning turned on, task success rises sharply and the pure MCP-vs-CLI gap collapses to almost zero, although the both interface setup still performed significantly worse than deliberate hybrid routing.

Stronger Models Flatten the Interface Gap

Opus 4.8 · AggregatedOpus 4.8 · Env 1Opus 4.8 · Env 2GPT-5.4 · AggregatedGPT-5.4 · Env 1GPT-5.4 · Env 2GPT-5.5 · AggregatedGPT-5.5 · Env 1GPT-5.5 · Env 20%6%12%18%24%Pass@1 — Task Success (%)