o1-mini

Sources

Every plotted observation links to its evaluation. Reported configurations, benchmark versions and task subsets can differ; similarity describes shared scores, not training lift.

1. Aider code editing benchmark ↗

   Paul Gauthier, Aider contributors

2. Aider refactoring benchmark ↗

   Paul Gauthier, Aider contributors

3. ARC Prize - Leaderboard ↗

   ARC Prize Foundation

4. JudgeBench: A Benchmark for Evaluating LLM-based Judges — published result table ↗

   Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y. Tang, Alejandro Cuadron, Chenguang Wang et al. · 2024

   All 8 authors

   Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y. Tang, Alejandro Cuadron, Chenguang Wang, Raluca Ada Popa, Ion Stoica

5. ProcessBench: Identifying Process Errors in Mathematical Reasoning — published result table ↗

   Zheng, Chujie, Zhang, Zhenru, Zhang, Beichen, Lin, Runji, Lu, Keming, Yu, Bowen et al. · 2024

   All 9 authors

   Zheng, Chujie, Zhang, Zhenru, Zhang, Beichen, Lin, Runji, Lu, Keming, Yu, Bowen, Liu, Dayiheng, Zhou, Jingren, Lin, Junyang

6. HARP: A challenging human-annotated math reasoning benchmark — published result table ↗

   Yue, Albert S., Madaan, Lovish, Moskovitz, Ted, Strouse, DJ, Singh, Aaditya K. · 2024

7. CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings — published result table ↗

   Shanghaoran Quan, Jiaxi Yang, Bowen Yu, Bo Zheng, Dayiheng Liu, An Yang et al. · 2025

   All 17 authors

   Shanghaoran Quan, Jiaxi Yang, Bowen Yu, Bo Zheng, Dayiheng Liu, An Yang, Xuancheng Ren, Bofei Gao, Yibo Miao, Yunlong Feng, Zekun Wang, Jian Yang, Zeyu Cui, Yang Fan, Yichang Zhang, Binyuan Hui, Junyang Lin

8. PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models — published result table ↗

   Song, Mingyang, Su, Zhaochen, Qu, Xiaoye, Zhou, Jiawei, Cheng, Yu · 2025

9. ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning — published result table ↗

   Bill Yuchen Lin, Ronan Le Bras, Kyle Richardson, Ashish Sabharwal, Radha Poovendran, Peter Clark et al. · 2025

   All 7 authors

   Bill Yuchen Lin, Ronan Le Bras, Kyle Richardson, Ashish Sabharwal, Radha Poovendran, Peter Clark, Yejin Choi

10. Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models ↗

    Zhang, Andy K., Perry, Neil, Dulepet, Riya, Ji, Joey, Menders, Celeste, Lin, Justin W. et al. · 2024

    All 27 authors

    Zhang, Andy K., Perry, Neil, Dulepet, Riya, Ji, Joey, Menders, Celeste, Lin, Justin W., Jones, Eliot, Hussein, Gashon, Liu, Samantha, Jasper, Donovan, Peetathawatchai, Pura, Glenn, Ari, Sivashankar, Vikram, Zamoshchin, Daniel, Glikbarg, Leo, Askaryar, Derek, Yang, Mike, Zhang, Teddy, Alluri, Rishi, Tran, Nathan, Sangpisit, Rinnara, Yiorkadjis, Polycarpos, Osele, Kenny, Raghupathi, Gautham, Boneh, Dan, Ho, Daniel E., Liang, Percy

11. ARC-AGI-1 ↗

    Epoch AI

12. ARC-AGI-2 ↗

    Epoch AI

13. FrontierMath Tiers 1-3 (v1, old) ↗

    Epoch AI

14. GPQA Diamond ↗

    Epoch AI

15. MATH Level 5 ↗

    Epoch AI

16. OTIS Mock AIME 2024-2025 ↗

    Epoch AI

17. SimpleBench ↗

    Epoch AI

18. AI benchmarks & capabilities ↗

    Epoch AI

19. Writing benchmark — incorporating mandatory story elements ↗

    lechmazur

20. WeirdML ↗

    Håvard Tveit Ihle

21. LiveBench — model evaluation leaderboard ↗

    LiveBench Team

22. Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models — published result table ↗

    Gao, Bofei, Song, Feifan, Yang, Zhe, Cai, Zefan, Miao, Yibo, Dong, Qingxiu et al. · 2024

    All 20 authors

    Gao, Bofei, Song, Feifan, Yang, Zhe, Cai, Zefan, Miao, Yibo, Dong, Qingxiu, Li, Lei, Ma, Chenghao, Chen, Liang, Xu, Runxin, Tang, Zhengyang, Wang, Benyou, Zan, Daoguang, Quan, Shanghaoran, Zhang, Ge, Sha, Lei, Zhang, Yichang, Ren, Xuancheng, Liu, Tianyu, Chang, Baobao

23. PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change — published result table ↗

    Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, Subbarao Kambhampati · 2022

24. Simple Evals — model benchmark results ↗

    OpenAI · 2025