ProgramBench：ソースコードなしでソフトウェアを再現するエージェント評価ベンチマーク

Can language models rebuild programs from scratch?

Given only a compiled binary and its documentation, agents must architect and implement

a complete codebase that reproduces the original program's behavior.

About ProgramBench

In each task, the agent receives an executable and its documentation, and it must re-implement the given executable. It does not get access to *any* of the executable's source code, it cannot de-compile the executable, and cannot use the internet. There are 200 tasks in total covering different program complexities, ranging from small terminal utilities like jq and ripgrep to massive software projects like the PHP compiler, FFmpeg, and SQLite.

The agent must choose a language, design the architecture, write all source code, and produce a build script. Every design decision is the model's to make.

Once the agent submits a program, our test suite compares the candidate program's behavior against the original program. A candidate program passes only if all tests for that task pass.

Our test suite is generated via agent-driven fuzzing, and it comprises more than 248,000 total behavioral tests for our 200 tasks.

Can tasks in ProgramBench be fully solved at all?

Yes. The agent can run the given program with any input and observe exactly what it does, so there's nothing hidden that can't be discovered through experimentation. The benchmark is hard, but it's solvable by design: all the reference executables pass our test suites. Read more in our blog post.

Why are ProgramBench scores so low?

Building a program from scratch is a fundamentally challenging task. Agents do currently make partial progress on many tasks (see the extended results for details), but fully passing every test is still out of reach.

Agents truly have to architect. This is in part because unlike other whole-repo generation projects, we give no hints or structure to the agent, meaning that the agent truly has to architect its own solutions (see "How is ProgramBench different?").

No harness tuning. Other recent and concurrent work also performed substantial harness tuning for a single or a handful number of tasks. We deliberately avoid this, since headline scores from a tuned harness on a curated handful of tasks can substantially overstate how capable agents really are at building software from scratch. Instead, ProgramBench is evaluated with a single generic harness across the entire task set.

Cleanroom implementation. We take substantial precautions to prevent cheating. Agents run in sandboxed containers without internet access, so they cannot retrieve the original source code or obtain any other form of help.

No decompilation. See "Can tasks be solved with decompilation?"

We review related work in section 6 of the paper. We also discuss cheating in the FAQ below and in section 4.1.

Is your agent scaffold sufficient to solve all tasks?

Widely adopted baseline. We use mini-SWE-agent because it is both widely adopted as a baseline by other benchmarks (SWE-bench Verified, SWE-bench Multilingual, Terminal-bench) and deliberately minimal in its scaffolding, reducing confounds between model capability and harness design. Most other agents (like Claude Code with apparently several 100k lines of code) are also constantly changing in non-transparent ways, while mini-SWE-agent will allow for apples-to-apples performance comparison of models for the foreseeable future.

Almost no runtime limitations. With very few exceptions, models submit their solutions deliberately rather than exceeding our generous time or step limits, and they never exhaust their context window. Because we do not limit total cost, our runs have cost up to $5k (for Sonnet 4.5).

Varying degree of difficulty. ProgramBench deliberately includes tasks from various degrees of difficulty, from very short repositories of only a few thousand lines of code to extremely large ones. We believe that the extremely low scores are therefore more of a signal of inadequate model capabilities rather than an indicator that only multi-agent systems can solve our tasks. Nonetheless, we would be excited to be one of the first systematic benchmarks that includes tasks that can only be solved by multi-agent systems.

Kicking off a new scaffold race. We believe that mini-SWE-agent is the right choice of baseline and that it can absolutely solve (some of) the tasks. However, we'd be more than excited if ProgramBench kicks off a new scaffold race! We will be opening submissions soon.

Can agents cheat?

Agents run in sandboxed containers with no internet access, execute-only permissions on the binary, and no access to decompilation tools. In early trials without these restrictions, models found shortcuts like cloning source repositories from GitHub or downloading code through package managers. Read more in our blog post and in section 4.1 of the paper.

Why and how do you block decompilation?

The executable that is given to the agent only has execution, not read permissions. That means that any operation that is not execution (such as running a decompiler, disassembler, objdump, strings, or hexdump) will fail.

We do this because we want ProgramBench to answer the question "How well can LMs build programs from scratch", rather than "How well can LMs patch together bits of decompiled code".

How is the leaderboard sorted? What's the primary metric?

The primary metric that should be reported for ProgramBench are fully resolved instances. We currently report "almost resolved" (more than 95% of test cases pass) as an additional point of reference while the scores of our primary metric are low. The leaderboard is sorted by fully resolved first, almost resolved second, and finally the average test pass rate.

For a detailed understanding of model performance, we recommend the plot at the detailed leaderboard. See also: "Have you considered other metrics?"

How do I submit to the leaderboard?

A public submission portal is coming soon. Follow the authors (John, Kilian) for updates.

Why do you not allow internet?

We have extensively studied different inference settings, including allowing internet. We find that allowing internet leads to an abundance of cheating that requires LM as a judge to flag and disqualify solutions. This makes the benchmark less reliable, especially because defining exactly what cheating means in the context of obtaining source online is not as clear cut as it might seem.

However, except for instances that contained cheating, we did not observe a dramatic improvement of scores when allowing internet.

Find our ablations in section 4.1 of the paper and John's explanation.

Have you considered other metrics? E.g., average number of tests passed?

Yes, we've decided on our current resolved metric after a lot of thought. Our initial question was "Can LMs build programs from scratch?", and the most relevant metric is the fraction of programs that can be fully built. Reporting an average test pass rate would be extremely misleading, because every instance includes very simple tests (such as checking for the existence of flags, checking what happens if you call the executable with --help etc.).

We've also thought about using a more relaxed metric like "almost resolved". However, relaxing to ≥95% of tests solved or even 99% of tests solved is also problematic. First, for some of our tasks, we have almost 15k tests. Even 1% of that is still 100 tests. And even a single failed test can indicate severe issues with a program. Therefore "almost resolved" only serves as additional orientation, until the main "resolved" metric has enough signal to differentiate all models.

However, all auxiliary metrics are still useful for diagnosing and improving models and scaffolds! They're just not the right metric as a benchmark. Check the extended results for more information. See also: "How is the leaderboard sorted?"

[junegunn/fzf

:cherry_blossom: A command-line fuzzy finder

79,721

go

Best score: 82%](https://programbench.com/task/junegunn__fzf.b56d614/)

[jesseduffield/lazygit

simple terminal UI for git commands

76,901

go

Best score: 56%](https://programbench.com/task/jesseduffield__lazygit.1d0db51/)

[BurntSushi/ripgrep

ripgrep recursively searches directories for a regex pattern while respecting your gitignore

62,855

rs

Best score: 80%](https://programbench.com/task/burntsushi__ripgrep.3b7fd44/)

[FFmpeg/FFmpeg

Mirror of https://git.ffmpeg.org/ffmpeg.git

59,217

c

Best score: 5%](https://programbench.com/task/ffmpeg__ffmpeg.360a402/)

[sharkdp/bat

A cat(1) clone with wings.

58,487

rs

Best score: 33%](https://programbench.com/task/sharkdp__bat.f822bd0/)

[typst/typst

A markup-based typesetting system that is powerful and easy to learn.

52,957

rs

Best score: 28%](https://programbench.com/task/typst__typst.88356d0/)

[jgm/pandoc

Universal markup converter

43,632

hs

Best score: 14%](https://programbench.com/task/jgm__pandoc.5caad90/)

[sharkdp/fd

A simple, fast and user-friendly alternative to 'find'

42,668

rs

Best score: 78%](https://programbench.com/task/sharkdp__fd.40d8eb3/)

[php/php-src

The PHP Interpreter

40,030

c

Best score: 5%](https://programbench.com/task/php__php-src.c891263/)

[duckdb/duckdb

DuckDB is an analytical in-process SQL database management system

37,657

cpp

Best score: 12%](https://programbench.com/task/duckdb__duckdb.bdb65ec/)

[ajeetdsouza/zoxide

A smarter cd command. Supports all major shells.

35,994

rs

Best score: 76%](https://programbench.com/task/ajeetdsouza__zoxide.67ca1bc/)

[jqlang/jq

Command-line JSON processor

34,541

c

Best score: 90%](https://programbench.com/task/jqlang__jq.b33a763/)

[dandavison/delta

A syntax-highlighting pager for git, diff, grep, rg --json, and blame output

30,445

rs

Best score: 37%](https://programbench.com/task/dandavison__delta.acd758f/)

[sharkdp/hyperfine

A command-line benchmarking tool

27,960

rs

Best score: 54%](https://programbench.com/task/sharkdp__hyperfine.327d5f4/)

[ggreer/the_silver_searcher

A code-searching tool similar to ack, but faster.

27,080

c

Best score: 59%](https://programbench.com/task/ggreer__the_silver_searcher.a61f178/)

[facebook/zstd

Zstandard - Fast real-time compression algorithm

27,013

c

Best score: 69%](https://programbench.com/task/facebook__zstd.1168da0/)

[facebookresearch/fastText

Library for fast text representation and classification.

26,511

cpp

Best score: 76%](https://programbench.com/task/facebookresearch__fasttext.1142dc4/)

[robertdavidgraham/masscan

TCP port scanner, spews SYN packets asynchronously, scanning entire Internet in under 5 minutes.

25,544

c

Best score: 57%](https://programbench.com/task/robertdavidgraham__masscan.b99d433/)

[tree-sitter/tree-sitter

An incremental parsing system for programming tools

24,953

rs

Best score: 37%](https://programbench.com/task/tree-sitter__tree-sitter.5e23cca/)

[FiloSottile/age

A simple, modern and secure encryption tool (and Go library) with small explicit keys, no config options, and UNIX-style composability.

22,077

go

Best score: 63%](https://programbench.com/task/filosottile__age.706dfc1/)

[rust-lang/mdBook

Create book from markdown files. Like Gitbook but implemented in Rust

21,541

rs

Best score: 55%](https://programbench.com/task/rust-lang__mdbook.37273ba/)

[jarun/nnn

n³ The unorthodox terminal file manager

21,506

c

Best score: 98%](https://programbench.com/task/jarun__nnn.cb2c535/)

[antonmedv/fx

Terminal JSON viewer & processor

20,433

go

Best score: 76%](https://programbench.com/task/antonmedv__fx.86d0d34/)

[mikefarah/yq

yq is a portable command-line YAML, JSON, XML, CSV, TOML, HCL and properties processor

15,281

go

Best score: 39%](https://programbench.com/task/mikefarah__yq.602586d/)

[Y2Z/monolith

⬛️ CLI tool and library for saving complete web pages as a single HTML file

15,024

rs

Best score: 51%](https://programbench.com/task/y2z__monolith.8702e66/)

<