DS-1000: what it measures

One row per model and compatible score measure. Mean = equal-weight arithmetic average of imported reports. Distinct configurations reported for the same model in a source table remain separate. Reports can overlap; the range is not a confidence interval. Unbundle to inspect each observation.
		Evidence	What the result shows
Codex-002 →	41.25% 2 reports · range 39.2–43.3%	DS-1000 · Overall pass@1[1] View 2 reports 43.3% · DS-1000 · Overall pass@1 · reported completion and insertion settings · DS-1000 · reported completion and insertion settings [1] 39.2% · DS-1000 · Overall pass@1 · reported completion and insertion settings · DS-1000 · reported completion and insertion settings [1]	Published observations span 4.1 percentage points. Configuration and source differences contribute to this spread; it is not measured run-to-run uncertainty. Leads this displayed comparison.
Codex-001 →	20.2% 1 report	DS-1000 · Overall pass@1[1]DS-1000 · reported completion and insertion settings	#2 of 5 in this displayed comparison; 21.05 pp below the leader. Source notes Source cell: 20.2
Codex-Cushman →	18.1% 1 report	DS-1000 · Overall pass@1[1]DS-1000 · reported completion and insertion settings	#3 of 5 in this displayed comparison; 23.15 pp below the leader. Source notes Source cell: 18.1
CodeGen-6B →	8.4% 1 report	DS-1000 · Overall pass@1[1]DS-1000 · reported completion and insertion settings	#4 of 5 in this displayed comparison; 32.85 pp below the leader. Source notes Source cell: 08.4
InCoder-6B →	7.45% 2 reports · range 7.4–7.5%	DS-1000 · Overall pass@1[1] View 2 reports 7.5% · DS-1000 · Overall pass@1 · reported completion and insertion settings · DS-1000 · reported completion and insertion settings [1] 7.4% · DS-1000 · Overall pass@1 · reported completion and insertion settings · DS-1000 · reported completion and insertion settings [1]	Published observations span 0.1 percentage points. Configuration and source differences contribute to this spread; it is not measured run-to-run uncertainty. #5 of 5 in this displayed comparison; 33.8 pp below the leader.

One row per model and compatible score measure. Mean = equal-weight arithmetic average of imported reports. Distinct configurations reported for the same model in a source table remain separate. Reports can overlap; the range is not a confidence interval. Unbundle to inspect each observation.

Evidence

What the result shows

Codex-002 →

41.25%

2 reports · range 39.2–43.3%

DS-1000 · Overall pass@1[1]

View 2 reports

43.3% · DS-1000 · Overall pass@1 · reported completion and insertion settings · DS-1000 · reported completion and insertion settings [1]
39.2% · DS-1000 · Overall pass@1 · reported completion and insertion settings · DS-1000 · reported completion and insertion settings [1]

Published observations span 4.1 percentage points. Configuration and source differences contribute to this spread; it is not measured run-to-run uncertainty.

Leads this displayed comparison.

Codex-001 →

20.2%

1 report

DS-1000 · Overall pass@1[1]DS-1000 · reported completion and insertion settings

#2 of 5 in this displayed comparison; 21.05 pp below the leader.

Source notes

Source cell: 20.2

Codex-Cushman →

18.1%

1 report

DS-1000 · Overall pass@1[1]DS-1000 · reported completion and insertion settings

#3 of 5 in this displayed comparison; 23.15 pp below the leader.

Source notes

Source cell: 18.1

CodeGen-6B →

8.4%

1 report

DS-1000 · Overall pass@1[1]DS-1000 · reported completion and insertion settings

#4 of 5 in this displayed comparison; 32.85 pp below the leader.

Source notes

Source cell: 08.4

InCoder-6B →

7.45%

2 reports · range 7.4–7.5%

DS-1000 · Overall pass@1[1]

View 2 reports

7.5% · DS-1000 · Overall pass@1 · reported completion and insertion settings · DS-1000 · reported completion and insertion settings [1]
7.4% · DS-1000 · Overall pass@1 · reported completion and insertion settings · DS-1000 · reported completion and insertion settings [1]

Published observations span 0.1 percentage points. Configuration and source differences contribute to this spread; it is not measured run-to-run uncertainty.

#5 of 5 in this displayed comparison; 33.8 pp below the leader.

Sources

DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation — published result table ↗

Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer et al. · 2022

All 10 authors

Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Scott Wen-tau Yih, Daniel Fried, Sida Wang, Tao Yu

Dates & evaluation details

DS-1000 · Overall pass@1 · reported completion and insertion settings · 2026-10-26

Published model comparison, 2022. DS-1000 · reported completion and insertion settings. Overall pass@1; DS-1000 · reported completion and insertion settings. Model configurations and missing observations follow the original publication. Table 5; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.

TRACE THE SCORE · Per-model notes use published breakdowns where available.

What it measures

1,000 realistic data-science coding problems across NumPy, Pandas, SciPy, Scikit-Learn, Matplotlib, PyTorch, and TensorFlow.

Reading the measurement

The number of permitted attempts changes the question. Pass@1 estimates whether the first submission succeeds; pass@k asks whether at least one of several candidates succeeds. Match sampling, execution limits and candidate selection before reading a gap as better capability.

Published evaluation

Overall pass@1; DS-1000 · reported completion and insertion settings. Model configurations and missing observations follow the original publication.

Where this fits in an evaluation suite

The agent wrapper is part of the system. Pin tools, dependencies, timeouts, tests, and the repository revision. Compare contest coding separately from repository maintenance.

Start with a capability hypothesis, then choose a second test with a different failure mode. Keep the checkpoint, reasoning effort, agent tools, sampling policy, and scoring procedure fixed when measuring a training intervention. Record failures and missing outputs in the denominator.

A reproducible comparison

Overall pass@1; DS-1000 · reported completion and insertion settings. Model configurations and missing observations follow the original publication.

Publish the exact run configuration, item coverage, score units, and available uncertainty. If a source supplies only an aggregate or a single run, preserve that limit. A new model name or a rounded score alone does not establish a capability improvement.

PRIMARY EVIDENCE & FURTHER READING

DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation — published result table ↗
Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer et al. · 2022
All 10 authors
Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Scott Wen-tau Yih, Daniel Fried, Sida Wang, Tao Yu

Reviewed 2026-10-26. Explanation and interpretation by RLTasks. Benchmark ownership remains with its original authors.

Sources

What it measures

Reading the measurement

Published evaluation

Where this fits in an evaluation suite

A reproducible comparison

Continue exploring

AA-3D

Terminal-Bench 4.0

FrontierCode 1.1

Terminal-Bench 2.1