AI21、オープンモデルとフロンティアモデルの組み合わせでコスト削減を提案

In brief

There’s been a resurgence of attention on executor-orchestrator architectures as a way to save on frontier compute. There, a frontier model assesses a task, builds a plan, and orchestrates among smaller, cheaper executor models to arrive at a solution. We think those savings can be even greater by examining how the entire pipeline is built.

In our setup, cheaper models are called first to handle token heavy context exploration and extraction, optimized to surface the highest quality context. By the time the frontier model is called at the end, it has the context in hand to write a correct patch after just one call.

This is the architecture that allowed us to reach a state-of-the-art 80.8% resolve rate on SWE-Bench Pro while spending just $5.99 per task, compared with other hybrid approaches, such as Fireworks AI’s recent worker + advisor experiment, where quality has suffered in the pursuit of cost savings.

Experimental setup and baseline

We use the standard coding-agent benchmark setup, covered in detail in our previous work: the agent receives a natural-language issue, works inside a Docker container holding the target repo, acts through terminal commands, and outputs a git patch scored against hidden tests. Our baseline is a classic ReAct (Reasoning + Acting) loop, with the Docker terminal as its only tool. A single pass with the open source MiniMax-M3 resolves 57% of the full public SWE-Bench Pro set. It’s a solid baseline, but raising it requires more sophisticated orchestration.

Engineering a multi-model pipeline to improve agent performance

An easy answer to raising the resolve rate could have been replacing MiniMax-M3 with a frontier model. But our goal wasn’t just raising quality; it was doing while also reducing compute costs compared to the current state-of-the-art result. A frontier agent would have spent its entire budget on both exploring and generating, driving up costs significantly.

Our solution was to think intentionally about how we built our pipeline.

Previously, we’ve measured how switching the order of the conventional coding agent pipeline improved both accuracy and efficiency; instead of first extracting context and then generating a solution, we generate solutions at a large scale and then exploit those candidates for better context extraction.

Now, we continue examining how effectively designing our coding agent pipeline can dramatically improve quality while reducing the overall cost per task. We make two crucial suggestions: first, assigning each step to the model that’s most efficient at the work required. Second, engineering the agents so that, by the time the frontier model is called for the final write, it has the best context to work with.

You can think of it like a team, with junior, senior, and principal members. Here, our junior model fans out across the repo in parallel, searching, trying fixes, running tests. Not every attempt lands, but collectively they map where the fix lives. A senior model picks up their attempts, digs into the code they touched, and writes up exactly what the real fix depends on: the functions involved, their callers, the tests that cover them. Only then does the principal model sit down and write the patch, once, from a prepared brief.

That’s our pipeline: MiniMax-M3 as the juniors, GPT-5.2 as the senior, and the frontier model (Opus 4.8 or Fable 5) as the principal we bring in for the single step where their skill changes the outcome.

Evaluating our code coverage efficiency: Are we surfacing the right context?

One of the primary sources of efficiency in this pipeline is delivering only the most relevant, high-quality context to the frontier model, so it stays focused while generating the patch. Because solution patches can reach tens of thousands of tokens, irrelevant context adds noise that can steer the model away from the right solution and send it wandering, driving up cost.

So before the final generation (step 3), we checked whether the context we assembled actually contains the code the fix needs. We did so by measuring it against the benchmark’s gold patches: For each task, what fraction of the lines the gold patch touches also appears in our assembled context? We track this in two ways:

Parallel rollouts (junior model). Each rollout is one full attempt by the open drafter (MiniMax-M3) to solve the issue. As we move along the x-axis in Figure 3, the pool grows from 1→10 as more attempts collectively touch on more of the relevant code.

Context extraction (senior model). On top of the candidate patches, GPT-5.2 pulls in the surrounding code the fix depends on. The shaded lift in the chart is what this extraction step adds beyond the rollouts

We measure two kinds of code lines separately:

• Removed lines are the lines the gold patch deletes or modifies, and they live in the original code. Coverage here helps answer the question: How well did we find the right “spot” in the repo to fix? With parallel rollouts and context extraction we reach ~90% coverage on this type of line.

• Added lines are the new code the fix introduces. Coverage here helps answer the question: How well did we propose the right fix? On this type of line, we reach ~71% coverage.

Tracing where the money actually goes

Our pipeline, running with Fable 5, costs $5.99/task. Here’s how that price breaks down across the pipeline:

The core advantage here is shifting the load off the frontier model.

When a solo Opus 4.8 agent tackles the whole task, start to finish, it costs $18.28 (as reported by Fireworks AI). In our pipeline, the frontier models consume only 25% of the budget, with the rest of the work being done by cheaper models. As a result, our whole pipeline runs at ~⅓ of the solo Opus 4.8 agent, while surpassing it in quality.

要点

• Match the model to the stage. Open, weaker models are not only cheaper, they’re also now good enough to effectively generate rollouts and search repos. Save frontier quality for the hard tail of tasks that open, weaker models still struggle to perform well.

• Reuse what you’ve already paid for. Recognize the Juniors’ parallel rollouts as the ‘fix map’ they are. Feed that signal into context extraction rather than discarding it after a vote.

• Gather source context after attempts at solving the tasks. Searching for discriminative context between known patches results in much more focused and effective source context to be fed to the next step.

• Spend frontier compute sparingly. A single frontier generation over well-prepared context beats an agent using a frontier model end-to-end. Use the expensive model on the step where quality matters most.

This is one piece of a broader push to optimize agent performance by composing the right execution strategy across the right models. Next up is training the source context extraction step to reach higher coverage.