PaperBench

What it measures

Evaluates whether coding agents can replicate 20 Spotlight/Oral ICML machine learning papers from scratch (writing all code and running experiments).

Reading the measurement

Partial credit can distinguish a useful approach from an empty attempt, but it is not the same as a completed task. Read the rubric, full-success count and score distribution together. For mathematical work, preserve the difference between expert review and acceptance by a formal proof checker.

Published evaluation

Mean replication score across papers using BasicAgent. Source error bars are one standard error of the mean; Code-Dev is a separate protocol.

Where this fits in an evaluation suite

The agent wrapper is part of the system. Pin tools, dependencies, timeouts, tests, and the repository revision. Compare contest coding separately from repository maintenance.

Start with a capability hypothesis, then choose a second test with a different failure mode. Keep the checkpoint, reasoning effort, agent tools, sampling policy, and scoring procedure fixed when measuring a training intervention. Record failures and missing outputs in the denominator.

A reproducible comparison

Mean replication score across papers using BasicAgent. Source error bars are one standard error of the mean; Code-Dev is a separate protocol.

Publish the exact run configuration, item coverage, score units, and available uncertainty. If a source supplies only an aggregate or a single run, preserve that limit. A new model name or a rounded score alone does not establish a capability improvement.