What it measures

1,140 practical software engineering tasks requiring complex function calls across 139 Python libraries (BigCodeBench-Hard = 148).

Reading the measurement

The number of permitted attempts changes the question. Pass@1 estimates whether the first submission succeeds; pass@k asks whether at least one of several candidates succeeds. Match sampling, execution limits and candidate selection before reading a gap as better capability.

Published evaluation

Original greedy-decoding score, converted from a fraction to a percentage. This is the original suite, not BigCodeBench-Hard; calibrated and sampled pass@k columns are not substituted.

Where this fits in an evaluation suite

The agent wrapper is part of the system. Pin tools, dependencies, timeouts, tests, and the repository revision. Compare contest coding separately from repository maintenance.

Start with a capability hypothesis, then choose a second test with a different failure mode. Keep the checkpoint, reasoning effort, agent tools, sampling policy, and scoring procedure fixed when measuring a training intervention. Record failures and missing outputs in the denominator.

A reproducible comparison

Original greedy-decoding score, converted from a fraction to a percentage. This is the original suite, not BigCodeBench-Hard; calibrated and sampled pass@k columns are not substituted.

Publish the exact run configuration, item coverage, score units, and available uncertainty. If a source supplies only an aggregate or a single run, preserve that limit. A new model name or a rounded score alone does not establish a capability improvement.

PRIMARY EVIDENCE & FURTHER READING

  1. BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions — published result table ↗

    Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari et al. · 2024

    All 33 authors

    Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Liu, Zijian Wang, Binyuan Hui, Niklas Muennighoff, David Lo, Daniel Fried, Xiaoning Du, Harm de Vries, Leandro Von Werra

Reviewed 2026-10-26. Explanation and interpretation by RLTasks. Benchmark ownership remains with its original authors.