Scale Labs、エージェント評価ベンチマーク「TERMINAL-BENCH 3.0」を公開

A lot of technical work happens in the terminal: compiling code, training models, tracing failing pipelines, and keeping entire systems running. TERMINAL-BENCH 3.0 is now live with broader coverage, harder tasks, and frontier tracking. This open-source, community effort benchmark measures how well AI agents can perform by testing them on involved tasks in a command-line environment.

Each task runs inside an isolated Docker sandbox, a clean, self-contained environment sealed off from the real computer, and starts with a plain-language instruction. The agent drives a real shell, reads the output, and works the problem the way a person would. Scoring ignores the agent's own account of what it did and checks the end result instead: did the code compile, did the server come up, did the pipeline return the right answer?

Scale Contributions

Scale contributed the most tasks of any single organization in the launch set, scaling up our expert and engineering teams to create and verify each one. That volume comes from two strengths working together: a very large and highly skilled expert network, and a data operation built to turn that expertise into verified tasks at scale. Where other providers are limited by how many qualified contributors they can reach, we can staff difficult domains deeply and run many workstreams in parallel without giving up quality.

The majority are science-focused domain workflows: end-to-end, tooling-heavy problems that are hard to specify and easy to fake your way through if the checks are loose. The rest cover finance and systems.

Getting the tasks right took precise task specs, verification that catches shortcuts, and a strict definition of what "correct" means. These tasks came out of a network of skilled contributors across domains who know what a right answer looks like in their own field and can build a task and a verifier that holds up to it.

*"The best benchmarks measure the edge of what models can do. TERMINAL-BENCH 3.0 provides a huge diversity of real, hard, economically valuable tasks, each programmatically verified, where the best frontier models can solve less than 40% of the tasks. That rigor is why Scale is one of its largest contributors, and why I am so excited about this partnership." — Aakash Sabharwal, VP of Engineering at Scale AI*

Why TERMINAL-BENCH 3.0

On Terminal-Bench 2.0, frontier models were already clearing around half the tasks, and they got there faster than expected. Once a benchmark gets that close to solved, it stops telling you much. TERMINAL-BENCH 3.0 pushes task difficulty higher so the benchmark keeps measuring frontier capabilities instead of trailing it.

Harbor sits underneath all of this. It's an open-source framework for evaluating and tuning agents and models inside container environments, and TERMINAL-BENCH 3.0 is built as a Harbor-native benchmarks. In practice that means a task contributed to TERMINAL-BENCH 3.0 plugs into a shared, reusable evaluation setup rather than a one-off script. The tasks span roughly 16 categories, from software engineering, security, and scientific computing to data science, machine learning, systems administration, web configuration, hardware and kernel work, games, and debugging, at easy, medium, and hard levels.

What Makes the Tasks Trustworthy

A benchmark is only as good as its tasks, and researchers are careful about how they get in. It's worth walking through, because the process shaped how we worked.

• Automated checks first. Every task PR has to pass a set of automated checks before a human looks at it.
• Human in the loop throughout. Reviewers are involved at every stage of task creation and quality control before anything merges.
• Failure-mode analysis. Agent rollouts are read closely to understand why a run failed. A task is kept only when the agent falls short because of a genuine capability gap, not because the instructions were unclear or the environment was broken.

Failures are the Point

The most useful thing a difficult task does is demonstrate exactly where an agent breaks. Across our tasks, we see consistent failure patterns that go beyond simple knowledge gaps. Models are strong at execution once pointed in the right direction, but they struggle with the step that comes first: reading the problem, understanding its structure, and choosing a method that actually fits the constraints rather than reaching for the most convenient one.

We see agents skip over physical or domain-specific reasoning in favor of generic approaches. We see them derive a correct, nuanced solution during their own analysis and then argue themselves out of it, defaulting to a simpler form because it fits only slightly worse. We see them produce answers that look plausible and well-structured but rest on the wrong assumptions. These aren't execution failures. They're judgment failures, and they're the kind of error that keeps agents out of real professional work.

This is what the benchmark catches, and it's what our task design is built to expose. Understanding these failure modes at a granular level is what lets us build tasks and verifiers that hold up against increasingly capable systems.

What's next

TERMINAL-BENCH 3.0 arrives while terminal agents are improving quickly, and the field needs better measurement to keep up.

If you build or evaluate terminal agents, this is an open project where you can propose a task, review a PR, or just follow along. The conversation is on Discord, and the code and task repositories are on GitHub.

Want data like this?

The tasks we contributed are a small window into the evaluation data Scale builds across scientific and professional domains: hard problems, precise specifications, and verifiers that hold up to expert scrutiny. If you're training or evaluating agents and want to see what data at this bar looks like, request sample tasks.