CORE-Bench: what it measures

One row per model and compatible score measure. Mean = equal-weight arithmetic average of imported reports. Distinct configurations reported for the same model in a source table remain separate. Reports can overlap; the range is not a confidence interval. Unbundle to inspect each observation.
		Evidence	What the result shows
GPT-4o →OpenAITest set · Easy · Agent Architecture: CORE-Agent	60% 1 report	CORE-Bench · CORE-Bench-Easy[1]Test set · Easy · Agent Architecture: CORE-Agent	Leads this displayed comparison. Source notes Source cell: 60.00%
GPT-4o-mini →OpenAITest set · Easy · Agent Architecture: CORE-Agent	44.44% 1 report	CORE-Bench · CORE-Bench-Easy[1]Test set · Easy · Agent Architecture: CORE-Agent	#2 of 4 in this displayed comparison; 15.56 pp below the leader.
GPT-4o →OpenAITest set · Easy · Agent Architecture: AutoGPT	35.56% 1 report	CORE-Bench · CORE-Bench-Easy[1]Test set · Easy · Agent Architecture: AutoGPT	#3 of 4 in this displayed comparison; 24.44 pp below the leader.
GPT-4o-mini →OpenAITest set · Easy · Agent Architecture: AutoGPT	8.89% 1 report	CORE-Bench · CORE-Bench-Easy[1]Test set · Easy · Agent Architecture: AutoGPT	#4 of 4 in this displayed comparison; 51.11 pp below the leader.
GPT-4o →OpenAITest set · Hard · Agent Architecture: CORE-Agent	21.48% 1 report	CORE-Bench · CORE-Bench-Hard[1]Test set · Hard · Agent Architecture: CORE-Agent	Leads this displayed comparison.
GPT-4o-mini →OpenAITest set · Hard · Agent Architecture: CORE-Agent	16.3% 1 report	CORE-Bench · CORE-Bench-Hard[1]Test set · Hard · Agent Architecture: CORE-Agent	#2 of 4 in this displayed comparison; 5.18 pp below the leader. Source notes Source cell: 16.30%
GPT-4o →OpenAITest set · Hard · Agent Architecture: AutoGPT	6.67% 1 report	CORE-Bench · CORE-Bench-Hard[1]Test set · Hard · Agent Architecture: AutoGPT	#3 of 4 in this displayed comparison; 14.81 pp below the leader.
GPT-4o-mini →OpenAITest set · Hard · Agent Architecture: AutoGPT	2.22% 1 report	CORE-Bench · CORE-Bench-Hard[1]Test set · Hard · Agent Architecture: AutoGPT	#4 of 4 in this displayed comparison; 19.26 pp below the leader.
GPT-4o →OpenAITest set · Medium · Agent Architecture: CORE-Agent	57.78% 1 report	CORE-Bench · CORE-Bench-Medium[1]Test set · Medium · Agent Architecture: CORE-Agent	Leads this displayed comparison.
GPT-4o →OpenAITest set · Medium · Agent Architecture: AutoGPT	37.78% 1 report	CORE-Bench · CORE-Bench-Medium[1]Test set · Medium · Agent Architecture: AutoGPT	#2 of 4 in this displayed comparison; 20 pp below the leader.
GPT-4o-mini →OpenAITest set · Medium · Agent Architecture: CORE-Agent	32.59% 1 report	CORE-Bench · CORE-Bench-Medium[1]Test set · Medium · Agent Architecture: CORE-Agent	#3 of 4 in this displayed comparison; 25.19 pp below the leader.
GPT-4o-mini →OpenAITest set · Medium · Agent Architecture: AutoGPT	2.22% 1 report	CORE-Bench · CORE-Bench-Medium[1]Test set · Medium · Agent Architecture: AutoGPT	#4 of 4 in this displayed comparison; 55.56 pp below the leader.

One row per model and compatible score measure. Mean = equal-weight arithmetic average of imported reports. Distinct configurations reported for the same model in a source table remain separate. Reports can overlap; the range is not a confidence interval. Unbundle to inspect each observation.

Evidence

What the result shows

GPT-4o →OpenAITest set · Easy · Agent Architecture: CORE-Agent

60%

1 report

CORE-Bench · CORE-Bench-Easy[1]Test set · Easy · Agent Architecture: CORE-Agent

Leads this displayed comparison.

Source notes

Source cell: 60.00%

GPT-4o-mini →OpenAITest set · Easy · Agent Architecture: CORE-Agent

44.44%

1 report

CORE-Bench · CORE-Bench-Easy[1]Test set · Easy · Agent Architecture: CORE-Agent

#2 of 4 in this displayed comparison; 15.56 pp below the leader.

GPT-4o →OpenAITest set · Easy · Agent Architecture: AutoGPT

35.56%

1 report

CORE-Bench · CORE-Bench-Easy[1]Test set · Easy · Agent Architecture: AutoGPT

#3 of 4 in this displayed comparison; 24.44 pp below the leader.

GPT-4o-mini →OpenAITest set · Easy · Agent Architecture: AutoGPT

8.89%

1 report

CORE-Bench · CORE-Bench-Easy[1]Test set · Easy · Agent Architecture: AutoGPT

#4 of 4 in this displayed comparison; 51.11 pp below the leader.

GPT-4o →OpenAITest set · Hard · Agent Architecture: CORE-Agent

21.48%

1 report

CORE-Bench · CORE-Bench-Hard[1]Test set · Hard · Agent Architecture: CORE-Agent

Leads this displayed comparison.

GPT-4o-mini →OpenAITest set · Hard · Agent Architecture: CORE-Agent

16.3%

1 report

CORE-Bench · CORE-Bench-Hard[1]Test set · Hard · Agent Architecture: CORE-Agent

#2 of 4 in this displayed comparison; 5.18 pp below the leader.

Source notes

Source cell: 16.30%

GPT-4o →OpenAITest set · Hard · Agent Architecture: AutoGPT

6.67%

1 report

CORE-Bench · CORE-Bench-Hard[1]Test set · Hard · Agent Architecture: AutoGPT

#3 of 4 in this displayed comparison; 14.81 pp below the leader.

GPT-4o-mini →OpenAITest set · Hard · Agent Architecture: AutoGPT

2.22%

1 report

CORE-Bench · CORE-Bench-Hard[1]Test set · Hard · Agent Architecture: AutoGPT

#4 of 4 in this displayed comparison; 19.26 pp below the leader.

GPT-4o →OpenAITest set · Medium · Agent Architecture: CORE-Agent

57.78%

1 report

CORE-Bench · CORE-Bench-Medium[1]Test set · Medium · Agent Architecture: CORE-Agent

Leads this displayed comparison.

GPT-4o →OpenAITest set · Medium · Agent Architecture: AutoGPT

37.78%

1 report

CORE-Bench · CORE-Bench-Medium[1]Test set · Medium · Agent Architecture: AutoGPT

#2 of 4 in this displayed comparison; 20 pp below the leader.

GPT-4o-mini →OpenAITest set · Medium · Agent Architecture: CORE-Agent

32.59%

1 report

CORE-Bench · CORE-Bench-Medium[1]Test set · Medium · Agent Architecture: CORE-Agent

#3 of 4 in this displayed comparison; 25.19 pp below the leader.

GPT-4o-mini →OpenAITest set · Medium · Agent Architecture: AutoGPT

2.22%

1 report

CORE-Bench · CORE-Bench-Medium[1]Test set · Medium · Agent Architecture: AutoGPT

#4 of 4 in this displayed comparison; 55.56 pp below the leader.

Sources

CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark — published result table ↗

Zachary S. Siegel, Sayash Kapoor, Nitya Nadgir, Benedikt Stroebl, Arvind Narayanan · 2024

Dates & evaluation details

CORE-Bench · CORE-Bench-Easy · Test set · 2026-10-26

Published model comparison, 2024. Test set · Easy. CORE-Agent results average three runs; AutoGPT results use one run. Difficulty levels expose different information and remain separate. Table 5; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.

CORE-Bench · CORE-Bench-Medium · Test set · 2026-10-26

Published model comparison, 2024. Test set · Medium. CORE-Agent results average three runs; AutoGPT results use one run. Difficulty levels expose different information and remain separate. Table 5; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.

CORE-Bench · CORE-Bench-Hard · Test set · 2026-10-26

Published model comparison, 2024. Test set · Hard. CORE-Agent results average three runs; AutoGPT results use one run. Difficulty levels expose different information and remain separate. Table 5; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.

TRACE THE SCORE · Per-model notes use published breakdowns where available.

What it measures

Computational reproducibility benchmark testing whether agents can reproduce scientific paper results from messy repositories.

Reading the measurement

Look beyond a plausible code sample to the executable artifact. Tests, repository context, language coverage and time limits shape the result. Separate first-attempt correctness from success after debugging, retries or agent assistance.

Published evaluation

CORE-Agent results average three runs; AutoGPT results use one run. Difficulty levels expose different information and remain separate.

Where this fits in an evaluation suite

The agent wrapper is part of the system. Pin tools, dependencies, timeouts, tests, and the repository revision. Compare contest coding separately from repository maintenance.

Start with a capability hypothesis, then choose a second test with a different failure mode. Keep the checkpoint, reasoning effort, agent tools, sampling policy, and scoring procedure fixed when measuring a training intervention. Record failures and missing outputs in the denominator.

A reproducible comparison

CORE-Agent results average three runs; AutoGPT results use one run. Difficulty levels expose different information and remain separate.

Publish the exact run configuration, item coverage, score units, and available uncertainty. If a source supplies only an aggregate or a single run, preserve that limit. A new model name or a rounded score alone does not establish a capability improvement.

PRIMARY EVIDENCE & FURTHER READING

CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark — published result table ↗
Zachary S. Siegel, Sayash Kapoor, Nitya Nadgir, Benedikt Stroebl, Arvind Narayanan · 2024

Reviewed 2026-10-26. Explanation and interpretation by RLTasks. Benchmark ownership remains with its original authors.

Sources

What it measures

Reading the measurement

Published evaluation

Where this fits in an evaluation suite

A reproducible comparison

Continue exploring

AA-3D

Terminal-Bench 4.0

FrontierCode 1.1

Terminal-Bench 2.1