CruxEval (Input & Output)

800 Python functions testing code execution reasoning: predicting output from input (CruxEval-O) and input from output (CruxEval-I).26 Oct 2026

One row per model and compatible score measure. Mean = equal-weight arithmetic average of imported reports. Distinct configurations reported for the same model in a source table remain separate. Reports can overlap; the range is not a confidence interval. Unbundle to inspect each observation.
		Evidence	What the result shows
Qwen3-235B-A22B Base →Qwen	79% 1 report	CruxEval (Input & Output) · CRUX-O accuracy[1]Qwen3 report · base-model comparison	Leads this displayed comparison. Source notes Source cell: 79.00
Llama-4-Maverick Base →Meta	77% 1 report	CruxEval (Input & Output) · CRUX-O accuracy[1]Qwen3 report · base-model comparison	#2 of 21 in this displayed comparison; 2 pp below the leader. Source notes Source cell: 77.00
DeepSeek V3 Base →DeepSeek	76.6% 1 report	CruxEval (Input & Output) · CRUX-O accuracy[1]Qwen3 report · base-model comparison	#3 of 21 in this displayed comparison; 2.4 pp below the leader. Source notes Source cell: 76.60
Qwen3-32B Base →Qwen	72.5% 1 report	CruxEval (Input & Output) · CRUX-O accuracy[1]Qwen3 report · base-model comparison	#4 of 21 in this displayed comparison; 6.5 pp below the leader. Source notes Source cell: 72.50
Qwen2.5-Plus Base →Qwen	68.5% 1 report	CruxEval (Input & Output) · CRUX-O accuracy[1]Qwen3 report · base-model comparison	#5 of 21 in this displayed comparison; 10.5 pp below the leader. Source notes Source cell: 68.50
Qwen2.5-32B Base →Qwen	67.8% 1 report	CruxEval (Input & Output) · CRUX-O accuracy[1]Qwen3 report · base-model comparison	#6 of 21 in this displayed comparison; 11.2 pp below the leader. Source notes Source cell: 67.80
Qwen2.5-72B Base →Qwen	66.2% 2 reports	CruxEval (Input & Output) · CRUX-O accuracy[1] View 2 reports 66.2% · CruxEval (Input & Output) · CRUX-O accuracy · Qwen3 report · base-model comparison · Qwen3 report · base-model comparison [1] 66.2% · CruxEval (Input & Output) · CRUX-O accuracy · Qwen3 report · base-model comparison · Qwen3 report · base-model comparison [1]	The imported reports agree at the published precision. Repeated citations may describe the same underlying evaluation. #7 of 21 in this displayed comparison; 12.8 pp below the leader.
Qwen3-8B Base →Qwen	62% 1 report	CruxEval (Input & Output) · CRUX-O accuracy[1]Qwen3 report · base-model comparison	#8 of 21 in this displayed comparison; 17 pp below the leader. Source notes Source cell: 62.00
Llama-4-Scout Base →Meta	61.9% 1 report	CruxEval (Input & Output) · CRUX-O accuracy[1]Qwen3 report · base-model comparison	#9 of 21 in this displayed comparison; 17.1 pp below the leader. Source notes Source cell: 61.90
Qwen2.5-14B Base →Qwen	61.1% 1 report	CruxEval (Input & Output) · CRUX-O accuracy[1]Qwen3 report · base-model comparison	#10 of 21 in this displayed comparison; 17.9 pp below the leader. Source notes Source cell: 61.10
Gemma-3-27B Base →Google	60% 1 report	CruxEval (Input & Output) · CRUX-O accuracy[1]Qwen3 report · base-model comparison	#11 of 21 in this displayed comparison; 19 pp below the leader. Source notes Source cell: 60.00
Qwen3-4B Base →Qwen	55% 1 report	CruxEval (Input & Output) · CRUX-O accuracy[1]Qwen3 report · base-model comparison	#12 of 21 in this displayed comparison; 24 pp below the leader. Source notes Source cell: 55.00
Qwen2.5-7B Base →Qwen	48.5% 2 reports	CruxEval (Input & Output) · CRUX-O accuracy[1] View 2 reports 48.5% · CruxEval (Input & Output) · CRUX-O accuracy · Qwen3 report · base-model comparison · Qwen3 report · base-model comparison [1] 48.5% · CruxEval (Input & Output) · CRUX-O accuracy · Qwen3 report · base-model comparison · Qwen3 report · base-model comparison [1]	The imported reports agree at the published precision. Repeated citations may describe the same underlying evaluation. #13 of 21 in this displayed comparison; 30.5 pp below the leader.
Llama-3-8B Base →Meta	36.8% 1 report	CruxEval (Input & Output) · CRUX-O accuracy[1]Qwen3 report · base-model comparison	#14 of 21 in this displayed comparison; 42.2 pp below the leader. Source notes Source cell: 36.80
Qwen2.5-3B Base →Qwen	36.5% 1 report	CruxEval (Input & Output) · CRUX-O accuracy[1]Qwen3 report · base-model comparison	#15 of 21 in this displayed comparison; 42.5 pp below the leader. Source notes Source cell: 36.50
Qwen3-1.7B Base →Qwen	36.4% 1 report	CruxEval (Input & Output) · CRUX-O accuracy[1]Qwen3 report · base-model comparison	#16 of 21 in this displayed comparison; 42.6 pp below the leader. Source notes Source cell: 36.40
Gemma-3-4B Base →Google	34% 1 report	CruxEval (Input & Output) · CRUX-O accuracy[1]Qwen3 report · base-model comparison	#17 of 21 in this displayed comparison; 45 pp below the leader. Source notes Source cell: 34.00
Qwen2.5-1.5B Base →Qwen	29.6% 1 report	CruxEval (Input & Output) · CRUX-O accuracy[1]Qwen3 report · base-model comparison	#18 of 21 in this displayed comparison; 49.4 pp below the leader. Source notes Source cell: 29.60
Qwen3-0.6B Base →Qwen	27% 1 report	CruxEval (Input & Output) · CRUX-O accuracy[1]Qwen3 report · base-model comparison	#19 of 21 in this displayed comparison; 52 pp below the leader. Source notes Source cell: 27.00
Qwen2.5-0.5B Base →Qwen	12.1% 1 report	CruxEval (Input & Output) · CRUX-O accuracy[1]Qwen3 report · base-model comparison	#20 of 21 in this displayed comparison; 66.9 pp below the leader. Source notes Source cell: 12.10
Gemma-3-1B Base →Google	3.8% 1 report	CruxEval (Input & Output) · CRUX-O accuracy[1]Qwen3 report · base-model comparison	#21 of 21 in this displayed comparison; 75.2 pp below the leader. Source notes Source cell: 3.80

Sources

Qwen3 Technical Report — published result table ↗
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng et al. · 2025
All 60 authors
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, Zihan Qiu
Dates & evaluation details
CruxEval (Input & Output) · CRUX-O accuracy · Qwen3 report · base-model comparison · 2026-10-26
Published model comparison, 2025. Qwen3 report · base-model comparison. Base-model results from the Qwen3 technical report. MultiPL-E and McEval are distinct suites; this observation covers only the explicitly named measure. Table 3; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.
CruxEval (Input & Output) · CRUX-O accuracy · Qwen3 report · base-model comparison · 2026-10-26
Published model comparison, 2025. Qwen3 report · base-model comparison. Base-model results from the Qwen3 technical report. MultiPL-E and McEval are distinct suites; this observation covers only the explicitly named measure. Table 4; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.
CruxEval (Input & Output) · CRUX-O accuracy · Qwen3 report · base-model comparison · 2026-10-26
Published model comparison, 2025. Qwen3 report · base-model comparison. Base-model results from the Qwen3 technical report. MultiPL-E and McEval are distinct suites; this observation covers only the explicitly named measure. Table 6; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.
CruxEval (Input & Output) · CRUX-O accuracy · Qwen3 report · base-model comparison · 2026-10-26
Published model comparison, 2025. Qwen3 report · base-model comparison. Base-model results from the Qwen3 technical report. MultiPL-E and McEval are distinct suites; this observation covers only the explicitly named measure. Table 7; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.
CruxEval (Input & Output) · CRUX-O accuracy · Qwen3 report · base-model comparison · 2026-10-26
Published model comparison, 2025. Qwen3 report · base-model comparison. Base-model results from the Qwen3 technical report. MultiPL-E and McEval are distinct suites; this observation covers only the explicitly named measure. Table 8; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.

TRACE THE SCORE · Per-model notes use published breakdowns where available.

What it measures

800 Python functions testing code execution reasoning: predicting output from input (CruxEval-O) and input from output (CruxEval-I).

Reading the measurement

The number of permitted attempts changes the question. Pass@1 estimates whether the first submission succeeds; pass@k asks whether at least one of several candidates succeeds. Match sampling, execution limits and candidate selection before reading a gap as better capability.

Published evaluation

Base-model results from the Qwen3 technical report. MultiPL-E and McEval are distinct suites; this observation covers only the explicitly named measure.

Where this fits in an evaluation suite

The agent wrapper is part of the system. Pin tools, dependencies, timeouts, tests, and the repository revision. Compare contest coding separately from repository maintenance.

Start with a capability hypothesis, then choose a second test with a different failure mode. Keep the checkpoint, reasoning effort, agent tools, sampling policy, and scoring procedure fixed when measuring a training intervention. Record failures and missing outputs in the denominator.

A reproducible comparison

Base-model results from the Qwen3 technical report. MultiPL-E and McEval are distinct suites; this observation covers only the explicitly named measure.

Publish the exact run configuration, item coverage, score units, and available uncertainty. If a source supplies only an aggregate or a single run, preserve that limit. A new model name or a rounded score alone does not establish a capability improvement.

PRIMARY EVIDENCE & FURTHER READING

Qwen3 Technical Report — published result table ↗
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng et al. · 2025
All 60 authors
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, Zihan Qiu

Reviewed 2026-10-26. Explanation and interpretation by RLTasks. Benchmark ownership remains with its original authors.

Continue exploring

All code & engineering benchmarks →

Sources

What it measures

Reading the measurement

Published evaluation

Where this fits in an evaluation suite

A reproducible comparison

Continue exploring

AA-3D

Terminal-Bench 4.0

FrontierCode 1.1

Terminal-Bench 2.1