Find the task behind the score.

Research mathematics evaluated through scoped progress, theorem use, counterexamples, and proof audits.

1. ErdősBench: research mathematics, scoped progress and proof audits ↗

   Ulam AI · 2026

Synthetic olympiad mathematics with reference solutions and partial credit for the proof itself.

1. SimoBench | Ulam ↗

   Ulam

Open mathematical problems test literature search, proof construction, proof criticism, and formalization. The published comparison reports separate research rubric ratings rather than a solved percentage.

1. Open Mathematical Problems as an AI Reasoning Benchmark ↗

   Przemyslaw Chojecki · 2026

157 exact-answer mathematical problems across AIME, AIME Hard, graduate and researcher tiers, with a reference scorer and fixed answer contract.

1. AIME++ Sample — dataset card and release documentation ↗

   Ulam AI · 2026

Three public research-math trajectory samples combine proof-verification units, negative attempts, preference pairs and review annotations. Partial and conditional research claims retain their stated status.

1. Verified Research Reasoning Trajectories for RLVR — dataset card and release documentation ↗

   Ulam AI · 2026

Ten public olympiad trajectory samples include golden solutions, verification units, generated attempts, preference pairs and adversarial checks for inspecting a post-training data format.

1. Verified Math Olympiad Reasoning Trajectories for RLVR — dataset card and release documentation ↗

   Ulam AI · 2026

Expert-written math problems covering advanced undergrad through early career research.

1. FrontierMath Tiers 1-3 (v2) ↗

   Epoch AI

2. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

The hardest FrontierMath tier tests advanced mathematical problem solving in a particularly demanding slice of the suite.

1. GPT-6 Astra: A new generation of intelligence ↗

   OpenAI

Open research problems are tracked through computer-checkable constructions and documented mathematical advances. Public first-solver credits are kept distinct from controlled benchmark success rates.

1. FrontierMath: Open Problems — public problem and solution registry ↗

   Epoch AI · 2026

68 unsolved Erdős problems where AI systems must produce formally verified solutions in Lean 4.

1. Publisher directory ↗
2. FrontierMath Erdős ↗

   Epoch AI

3. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

A mathematical evaluation is included in recent frontier-model science and math comparisons.

1. Gemini 4 Argon Model Card ↗

   Google DeepMind · 2026

A benchmark of graduate-level mathematics problems where models must write Lean 4 proofs that pass formal verification.

1. Epoch AI · benchmark and methodology ↗
2. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

3. ProofBench — evaluation methodology and results ↗

   Vals AI · 2026

Olympiad-related mathematics is evaluated through answer-oriented scoring in frontier model reports.

1. moonshotai/Kimi-K2.6 · Hugging Face ↗

   Moonshot AI

A new annual contest set tests mathematical reasoning through short integer answers.

1. moonshotai/Kimi-K2.6 · Hugging Face ↗

   Moonshot AI

Contest mathematics with short integer answers provides a compact test of multi-step problem solving.

1. AIME 2025 Benchmark | Artificial Analysis ↗

   Artificial Analysis

30 problems from the 2024 AIME (standard benchmark in DeepSeek-R1, o1, Qwen2.5/3 whitepapers).

1. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning — published result table ↗

   DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang et al. · 2025

   All 200 authors

   DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, Zhen Zhang

45 competition-style math problems from OTIS, harder than MATH Level 5 but easier than FrontierMath.

1. OTIS Mock AIME 2024-2025 ↗

   Epoch AI

2. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

Harder post-AIME olympiad benchmark designed to avoid saturation of AIME 2024/2025 (used in DAPO, VAPO, SAO/GLM-5.2).

1. Seed-Thinking-v1.5: Advancing Superb Reasoning Models with Reinforcement Learning ↗

   ByteDance Seed · 2025

The February Harvard–MIT Mathematics Tournament provides a dated contest-math evaluation.

1. moonshotai/Kimi-K2.6 · Hugging Face ↗

   Moonshot AI

Standard 2025 competition math holdout used in OpenAI, DeepSeek, and Qwen evaluations.

1. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models — published result table ↗

   DeepSeek-AI, Liu, Aixin, Mei, Aoxue, Lin, Bangcai, Xue, Bing, Wang, Bingxuan et al. · 2025

   All 264 authors

   DeepSeek-AI, Liu, Aixin, Mei, Aoxue, Lin, Bangcai, Xue, Bing, Wang, Bingxuan, Xu, Bingzheng, Wu, Bochao, Zhang, Bowei, Lin, Chaofan, Dong, Chen, Lu, Chengda, Zhao, Chenggang, Deng, Chengqi, Xu, Chenhao, Ruan, Chong, Dai, Damai, Guo, Daya, Yang, Dejian, Chen, Deli, Li, Erhang, Zhou, Fangqi, Lin, Fangyun, Dai, Fucong, Hao, Guangbo, Chen, Guanting, Li, Guowei, Zhang, H., Xu, Hanwei, Li, Hao, Liang, Haofen, Wei, Haoran, Zhang, Haowei, Luo, Haowen, Ji, Haozhe, Ding, Honghui, Tang, Hongxuan, Cao, Huanqi, Gao, Huazuo, Qu, Hui, Zeng, Hui, Huang, Jialiang, Li, Jiashi, Xu, Jiaxin, Hu, Jiewen, Chen, Jingchang, Xiang, Jingting, Yuan, Jingyang, Cheng, Jingyuan, Zhu, Jinhua, Ran, Jun, Jiang, Junguang, Qiu, Junjie, Li, Junlong, Song, Junxiao, Dong, Kai, Gao, Kaige, Guan, Kang, Huang, Kexin, Zhou, Kexing, Huang, Kezhao, Yu, Kuai, Wang, Lean, Zhang, Lecong, Wang, Lei, Zhao, Liang, Yin, Liangsheng, Guo, Lihua, Luo, Lingxiao, Ma, Linwang, Wang, Litong, Zhang, Liyue, Di, M. S., Xu, M. Y, Zhang, Mingchuan, Zhang, Minghua, Tang, Minghui, Zhou, Mingxu, Huang, Panpan, Cong, Peixin, Wang, Peiyi, Wang, Qiancheng, Zhu, Qihao, Li, Qingyang, Chen, Qinyu, Du, Qiushi, Xu, Ruiling, Ge, Ruiqi, Zhang, Ruisong, Pan, Ruizhe, Wang, Runji, Yin, Runqiu, Xu, Runxin, Shen, Ruomeng, Zhang, Ruoyu, Liu, S. H., Lu, Shanghao, Zhou, Shangyan, Chen, Shanhuang, Cai, Shaofei, Chen, Shaoyuan, Hu, Shengding, Liu, Shengyu, Hu, Shiqiang, Ma, Shirong, Wang, Shiyu, Yu, Shuiping, Zhou, Shunfeng, Pan, Shuting, Zhou, Songyang, Ni, Tao, Yun, Tao, Pei, Tian, Ye, Tian, Yue, Tianyuan, Zeng, Wangding, Liu, Wen, Liang, Wenfeng, Pang, Wenjie, Luo, Wenjing, Gao, Wenjun, Zhang, Wentao, Gao, Xi, Wang, Xiangwen, Bi, Xiao, Liu, Xiaodong, Wang, Xiaohan, Chen, Xiaokang, Zhang, Xiaokang, Nie, Xiaotao, Cheng, Xin, Liu, Xin, Xie, Xin, Liu, Xingchao, Yu, Xingkai, Li, Xingyou, Yang, Xinyu, Li, Xinyuan, Chen, Xu, Su, Xuecheng, Pan, Xuehai, Lin, Xuheng, Fu, Xuwei, Wang, Y. Q., Zhang, Yang, Xu, Yanhong, Ma, Yanru, Li, Yao, Li, Yao, Zhao, Yao, Sun, Yaofeng, Wang, Yaohui, Qian, Yi, Yu, Yi, Zhang, Yichao, Ding, Yifan, Shi, Yifan, Xiong, Yiliang, He, Ying, Zhou, Ying, Zhong, Yinmin, Piao, Yishi, Wang, Yisong, Chen, Yixiao, Tan, Yixuan, Wei, Yixuan, Ma, Yiyang, Liu, Yiyuan, Yang, Yonglun, Guo, Yongqiang, Wu, Yongtong, Wu, Yu, Cheng, Yuan, Ou, Yuan, Xu, Yuanfan, Wang, Yuduan, Gong, Yue, Wu, Yuhan, Zou, Yuheng, Li, Yukun, Xiong, Yunfan, Luo, Yuxiang, You, Yuxiang, Liu, Yuxuan, Zhou, Yuyang, Wu, Z. F., Ren, Z. Z., Zhao, Zehua, Ren, Zehui, Sha, Zhangli, Fu, Zhe, Xu, Zhean, Xie, Zhenda, Zhang, Zhengyan, Hao, Zhewen, Gou, Zhibin, Ma, Zhicheng, Yan, Zhigang, Shao, Zhihong, Huang, Zhixian, Wu, Zhiyu, Li, Zhuoshu, Zhang, Zhuping, Xu, Zian, Wang, Zihao, Gu, Zihui, Zhu, Zijia, Li, Zilin, Zhang, Zipeng, Xie, Ziwei, Gao, Ziyi, Pan, Zizheng, Yao, Zongqing, Feng, Bei, Li, Hui, Cai, J. L., Ni, Jiaqi, Xu, Lei, Li, Meng, Tian, Ning, Chen, R. J., Jin, R. L., Li, S. S., Zhou, Shuang, Sun, Tianyu, Li, X. Q., Jin, Xiangyue, Shen, Xiaojin, Chen, Xiaosha, Song, Xinnan, Zhou, Xinyi, Zhu, Y. X., Huang, Yanping, Li, Yaohui, Zheng, Yi, Zhu, Yuchen, Ma, Yunxian, Huang, Zhen, Xu, Zhipeng, Zhang, Zhongyu, Ji, Dongjie, Liang, Jian, Guo, Jianzhong, Chen, Jin, Xia, Leyi, Wang, Miaojun, Li, Mingming, Zhang, Peng, Chen, Ruyi, Sun, Shangmian, Wu, Shaoqing, Ye, Shengfeng, Wang, T., Xiao, W. L., An, Wei, Wang, Xianzu, Sun, Xiaowen, Wang, Xiaoxiang, Tang, Ying, Zha, Yukun, Zhang, Zekai, Ju, Zhe, Zhang, Zhen, Qu, Zihua

640+ William Lowell Putnam Mathematical Competition problems (1962–2023) formalized in Lean 4, Isabelle, and Coq.

1. PutnamBench: Evaluating Neural Theorem-Provers on the Putnam Mathematical Competition — published result table ↗

   George Tsoukalas, Jasper Lee, John Jennings, Jimmy Xin, Michelle Ding, Michael Jennings et al. · 2024

   All 8 authors

   George Tsoukalas, Jasper Lee, John Jennings, Jimmy Xin, Michelle Ding, Michael Jennings, Amitayush Thakur, Swarat Chaudhuri

A mathematical-reasoning benchmark built from Putnam competition problems, with original questions and functional variations that test whether a model can adapt its solution when a problem changes. Problem accuracy is separate from teacher-forced token prediction.

1. Putnam-AXIOM: A Functional and Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs — published result table ↗

   Gulati, Aryan, Miranda, Brando, Chen, Eric, Xia, Emily, Fronsdal, Kai, Dumont, Bruno et al. · 2025

   All 8 authors

   Gulati, Aryan, Miranda, Brando, Chen, Eric, Xia, Emily, Fronsdal, Kai, Dumont, Bruno, Obbad, Elyas, Koyejo, Sanmi

4,428 olympiad-level math problems across 33 sub-domains and 4 difficulty tiers.

1. Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models — published result table ↗

   Gao, Bofei, Song, Feifan, Yang, Zhe, Cai, Zefan, Miao, Yibo, Dong, Qingxiu et al. · 2024

   All 20 authors

   Gao, Bofei, Song, Feifan, Yang, Zhe, Cai, Zefan, Miao, Yibo, Dong, Qingxiu, Li, Lei, Ma, Chenghao, Chen, Liang, Xu, Runxin, Tang, Zhengyang, Wang, Benyou, Zan, Daoguang, Quan, Shanghaoran, Zhang, Ge, Sha, Lei, Zhang, Yichang, Ren, Xuancheng, Liu, Tianyu, Chang, Baobao

Bilingual, multimodal olympiad-level mathematics benchmark spanning IMO, regional olympiads, and Gaokao hard problems.

1. OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems — published result table ↗

   Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen et al. · 2024

   All 14 authors

   Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, Maosong Sun

Chinese National Mathematical Olympiad problems reported in DeepSeek-R1, Qwen, and Kimi whitepapers.

1. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning — published result table ↗

   DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang et al. · 2025

   All 200 authors

   DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, Zhen Zhang

Formalized mathematical statements test whether a system can construct proofs accepted by a proof assistant.

1. MiniF2F: a cross-system benchmark for formal Olympiad-level mathematics — published result table ↗

   Zheng, Kunhao, Han, Jesse Michael, Polu, Stanislas · 2021

2. Primary benchmark documentation or model evaluation report ↗

Undergraduate pure mathematics autoformalization and formal theorem proving benchmark in Lean.

1. ProofNet: Autoformalizing and Formally Proving Undergraduate-Level Mathematics — published result table ↗

   Zhangir Azerbayev, Bartosz Piotrowski, Hailey Schoelkopf, Edward W. Ayers, Dragomir Radev, Jeremy Avigad · 2023

IMO Shortlist algebra and number theory problems formalized in Lean.

1. FIMO: A Challenge Formal Dataset for Automated Theorem Proving — published result table ↗

   Liu, Chengwu, Shen, Jianhao, Xin, Huajian, Liu, Zhengying, Yuan, Ye, Wang, Haiming et al. · 2023

   All 12 authors

   Liu, Chengwu, Shen, Jianhao, Xin, Huajian, Liu, Zhengying, Yuan, Ye, Wang, Haiming, Ju, Wei, Zheng, Chuanyang, Yin, Yichun, Li, Lin, Zhang, Ming, Liu, Qun

98,734 theorems extracted from Mathlib with novel-premise split testing formal theorem proving.

1. LeanDojo: Theorem Proving with Retrieval-Augmented Language Models — published result table ↗

   Yang, Kaiyu, Swope, Aidan M., Gu, Alex, Chalamala, Rahul, Song, Peiyang, Yu, Shixing et al. · 2023

   All 9 authors

   Yang, Kaiyu, Swope, Aidan M., Gu, Alex, Chalamala, Rahul, Song, Peiyang, Yu, Shixing, Godil, Saad, Prenger, Ryan, Anandkumar, Anima

A compact selection of competition mathematics problems supports repeatable reasoning checks.

1. MATH 500 — evaluation methodology and results ↗

   Vals AI · 2026

2. MATH-500 Benchmark | Artificial Analysis ↗

   Artificial Analysis

Grade-school mathematical word problems test multi-step arithmetic reasoning.

1. Primary benchmark documentation or model evaluation report ↗
2. Epoch AI · benchmark and methodology ↗
3. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

Perturbed and symbolic template versions of GSM8K testing robustness to Distractor clauses and numerical changes.

1. GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers — published result table ↗

   Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, Wei Bi · 2024

250 GSM8K problems professionally translated into 10 typologically diverse languages.

1. MGSM — evaluation methodology and results ↗

   Vals AI · 2026

Multilingual mathematical reasoning benchmark across 18 languages and 4 difficulty levels.

1. PolyMath: Evaluating Mathematical Reasoning in Multilingual Contexts — published result table ↗

   Yiming Wang, Pei Zhang, Jialong Tang, Haoran Wei, Baosong Yang, Rui Wang et al. · 2025

   All 16 authors

   Yiming Wang, Pei Zhang, Jialong Tang, Haoran Wei, Baosong Yang, Rui Wang, Chenshu Sun, Feitong Sun, Jiran Zhang, Junxuan Wu, Qiqian Cang, Yichang Zhang, Fei Huang, Junyang Lin, Fei Huang, Jingren Zhou

2,818 university-level math problems spanning calculus, linear algebra, differential equations, probability, and abstract algebra.

1. MathScale: Scaling Instruction Tuning for Mathematical Reasoning — published result table ↗

   Zhengyang Tang, Xingxing Zhang, Benyou Wang, Furu Wei · 2024

Human-annotated problems from US national math competitions, with 4,780 automatically checkable short-answer questions spanning six difficulty levels. The paper also studies multiple-choice prompting and inference-time sampling.

1. HARP: A challenging human-annotated math reasoning benchmark — published result table ↗

   Yue, Albert S., Madaan, Lovish, Moskovitz, Ted, Strouse, DJ, Singh, Aaditya K. · 2024

387 high-school to university and olympiad mathematics problems crafted by professors to avoid contamination.

1. MathOdyssey: Benchmarking Mathematical Problem-Solving Skills in Large Language Models Using Odyssey Math Data — published result table ↗

   Meng Fang, Xiangpeng Wan, Fei Lu, Fei Xing, Kai Zou · 2024

10-problem public inspection release of FrontierMath with full Inspect log traces.

1. A Rosetta Stone for AI Benchmarks — released Epoch evaluation runs ↗

   Epoch AI · 2025

Mathematical reasoning is tested through diagrams, charts, scientific figures and other visual contexts.

1. Primary benchmark documentation or model evaluation report ↗

Mathematical problems with visual context require reasoning over both the statement and the supplied figures.

1. moonshotai/Kimi-K2.6 · Hugging Face ↗

   Moonshot AI

Complex terminal work spans coding, science, infrastructure, and other technical tasks, with executable checks.

1. Introducing Kimi K3.1 Pro ↗

   Moonshot AI · 2026

2. Terminal-Bench 4.0 — evaluation methodology and results ↗

   Vals AI · 2026

3. Terminal-Bench 4.0 Benchmark | Artificial Analysis ↗

   Artificial Analysis

4. Epoch AI · benchmark and methodology ↗
5. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

A refreshed terminal task suite repairs environment and instruction issues from the earlier release.

1. Terminal-Bench 2.1 — evaluation methodology and results ↗

   Vals AI · 2026

2. Terminal-Bench 2.1 Benchmark | Artificial Analysis ↗

   Artificial Analysis

An agent completes technical work in a terminal under executable task checks.

1. Terminal-Bench 2.0 — evaluation methodology and results ↗

   Vals AI · 2026

2. moonshotai/Kimi-K2.6 · Hugging Face ↗

   Moonshot AI

3. Terminal-Bench 2.0 ↗

   Epoch AI

4. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

A difficult terminal-work evaluation probes multi-step technical execution.

1. Terminal-Bench Hard Benchmark | Artificial Analysis ↗

   Artificial Analysis

Human-reviewed repository issues test whether an agent can implement a repair in an existing Python codebase.

1. SWE-bench Verified — evaluation methodology and results ↗

   Vals AI · 2026

2. Primary benchmark documentation or model evaluation report ↗
3. SWE-bench Verified ↗

   Epoch AI

4. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

Long-horizon software-engineering tasks extend repository evaluation beyond small isolated edits.

1. Primary benchmark documentation or model evaluation report ↗
2. Epoch AI · benchmark and methodology ↗
3. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

Updated 2026 long-horizon repository software engineering benchmark with Full and Hard splits.

1. SWE-Bench Pro V2 — evaluation methodology and results ↗

   Scale Labs / SEAL · 2026

2. SWE-Bench Pro V2 — evaluation methodology and results ↗

   Scale Labs / SEAL · 2026

Repository-level software repair broadens evaluation across programming languages.

1. moonshotai/Kimi-K2.6 · Hugging Face ↗

   Moonshot AI

517 visual software engineering issues (UI rendering, charting, front-end layout, diagram specs) across JS/TS/HTML/CSS repos.

1. SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains? — published result table ↗

   John Yang, Carlos E. Jimenez, Alex L. Zhang, Kilian Lieret, Joyce Yang, Xindi Wu et al. · 2024

   All 13 authors

   John Yang, Carlos E. Jimenez, Alex L. Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R. Narasimhan, Diyi Yang, Sida I. Wang, Ofir Press

2. SWE-bench Multimodal — evaluation methodology and results ↗

   Vals AI · 2026

Full 2,294-issue benchmark and 300-issue SWE-bench Lite subset.

1. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — published result table ↗

   Jimenez, Carlos E., Yang, John, Wettig, Alexander, Yao, Shunyu, Pei, Kexin, Press, Ofir et al. · 2023

   All 7 authors

   Jimenez, Carlos E., Yang, John, Wettig, Alexander, Yao, Shunyu, Pei, Kexin, Press, Ofir, Narasimhan, Karthik

Continuously updated contamination-free stream of newly closed GitHub issues created after model training cutoffs.

1. SWE-bench Goes Live! — published result table ↗

   Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie et al. · 2025

   All 15 authors

   Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, Junhao Wang, Maoquan Wang, Yufan Huang, Shengyu Fu, Elsie Nallipogu, Qingwei Lin, Yingnong Dang, Saravan Rajmohan, Dongmei Zhang

1,400+ real freelance Upwork software engineering tasks ($1M total payout value) spanning IC implementation and SWE Management.

1. SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? — published result table ↗

   Miserendino, Samuel, Wang, Michele, Patwardhan, Tejal, Heidecke, Johannes · 2025

1,632 repository issues spanning 7 languages (Java, TypeScript, JavaScript, Go, Rust, C, C++).

1. Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving — published result table ↗

   Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin et al. · 2025

   All 19 authors

   Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, Siyao Liu, Yongsheng Xiao, Liangqiang Chen, Yuyu Zhang, Jing Su, Tianyu Liu, Rui Long, Kai Shen, Liang Xiang

Evaluates coding agents on multi-file architectural refactoring while preserving existing behavioral contracts.

1. SWE Atlas - Refactoring — evaluation methodology and results ↗

   Scale Labs / SEAL · 2026

Evaluates coding agents on writing production-grade test suites that catch regressions and edge cases.

1. SWE Atlas - Test Writing — evaluation methodology and results ↗

   Scale Labs / SEAL · 2026

Evaluates deep architectural comprehension and multi-file dependency reasoning in large codebases.

1. Scale Labs — evaluation methodology and results ↗

   Scale Labs / SEAL · 2026

Original engineering problems test coding agents on substantial changes in existing repositories.

1. Introducing Kimi K3.1 Pro ↗

   Moonshot AI · 2026

2. Primary benchmark documentation or model evaluation report ↗
3. Epoch AI · benchmark and methodology ↗
4. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

Extended engineering and research challenges test how much useful work an agent can complete within a long fixed budget.

1. Primary benchmark documentation or model evaluation report ↗
2. Epoch AI · benchmark and methodology ↗
3. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

Repository-level engineering is evaluated against the standard of work maintainers could accept.

1. FrontierCode 1.1 Benchmark | Artificial Analysis ↗

   Artificial Analysis

2. Epoch AI · benchmark and methodology ↗
3. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

Extended software projects test whether an agent can keep making useful progress through long engineering sessions.

1. Introducing Kimi K3.1 Pro ↗

   Moonshot AI · 2026

2. Primary benchmark documentation or model evaluation report ↗

Agents reconstruct programs from observed behavior rather than receiving a conventional repository repair issue.

1. Introducing Kimi K3.1 Pro ↗

   Moonshot AI · 2026

2. ProgramBench — evaluation methodology and results ↗

   Vals AI · 2026

Long-horizon clean-room program implementation benchmark administered by Epoch AI.

1. Publisher directory ↗
2. MirrorCode ↗

   Epoch AI

3. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

A natural-language software request is used to assess whether an agent can produce a working repository.

1. deepseek-ai/DeepSeek-V4.1-Flash-1017 · Hugging Face ↗

   DeepSeek AI

Challenges agents to write core Python libraries (like simpy, minitorch, pydantic) from scratch starting from commit 0 and API docs.

1. Commit0: Library Generation from Scratch — published result table ↗

   Zhao, Wenting, Jiang, Nan, Lee, Celine, Chiu, Justin T, Cardie, Claire, Gallé, Matthias et al. · 2024

   All 7 authors

   Zhao, Wenting, Jiang, Nan, Lee, Celine, Chiu, Justin T, Cardie, Claire, Gallé, Matthias, Rush, Alexander M

Compare app-building outcomes across the original suite and the 1–100 collection, with each task set labeled separately.

1. Vibe Code Bench — evaluation methodology and results ↗

   Vals AI · 2026

2. Gemini 4 Argon Model Card ↗

   Google DeepMind · 2026

3. Vibe Code Bench 1-100 — evaluation methodology and results ↗

   Vals AI · 2026

Compare app-building outcomes across the original suite and the 1–100 collection, with each task set labeled separately.

1. Vibe Code Bench — evaluation methodology and results ↗

   Vals AI · 2026

2. Gemini 4 Argon Model Card ↗

   Google DeepMind · 2026

3. Vibe Code Bench 1-100 — evaluation methodology and results ↗

   Vals AI · 2026

Evaluates agents on translating/reimplementing complete working programs from one programming language to another.

1. Code Migration — evaluation methodology and results ↗

   Vals AI · 2026

2. Code Migration v1 — evaluation methodology and results ↗

   Vals AI · 2026

A language-model agent turns a written brief into an interactive three.js scene inside an offline environment.

1. AA-3D: Interactive 3D Experience Benchmark | Artificial Analysis ↗

   Artificial Analysis

Visual-to-code tasks assess whether a model can construct a CAD representation matching a target shape.

1. Introducing Kimi K3.1 Pro ↗

   Moonshot AI · 2026

Language-model workflows produce graphic-design artifacts from direct or reference-based briefs.

1. Introducing Kimi K3.1 Pro ↗

   Moonshot AI · 2026

An agentic-coding evaluation appears alongside terminal and repository benchmarks in Anthropic’s release.

1. Introducing Claude Opus 5.5 ↗

   Anthropic

2. Epoch AI · benchmark and methodology ↗
3. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

Kimi’s internal coding evaluation tests workflows selected by the model developer.

1. Kimi K3: Open Frontier Intelligence — official model card ↗

   Moonshot AI · 2026

2. Introducing Kimi K3.1 Pro ↗

   Moonshot AI · 2026

Kimi reports an internal, difficult security-capability evaluation.

1. Introducing Kimi K3.1 Pro ↗

   Moonshot AI · 2026

An agent must improve small language models within a constrained training-compute budget.

1. Introducing Kimi K3.1 Pro ↗

   Moonshot AI · 2026

2. Primary benchmark documentation or model evaluation report ↗
3. PostTrainBench v1.1 ↗

   Epoch AI

4. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

Agents train and evaluate machine-learning systems on competition-style engineering tasks.

1. MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering — published result table ↗

   Chan, Jun Shern, Chowdhury, Neil, Jaffe, Oliver, Aung, James, Sherburn, Dane, Mays, Evan et al. · 2024

   All 12 authors

   Chan, Jun Shern, Chowdhury, Neil, Jaffe, Oliver, Aung, James, Sherburn, Dane, Mays, Evan, Starace, Giulio, Liu, Kevin, Maksin, Leon, Patwardhan, Tejal, Weng, Lilian, Mądry, Aleksander

Evaluates whether coding agents can replicate 20 Spotlight/Oral ICML machine learning papers from scratch (writing all code and running experiments).

1. PaperBench: Evaluating AI's Ability to Replicate AI Research — published result table ↗

   Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin et al. · 2025

   All 13 authors

   Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, Tejal Patwardhan

Computational reproducibility benchmark testing whether agents can reproduce scientific paper results from messy repositories.

1. CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark — published result table ↗

   Zachary S. Siegel, Sayash Kapoor, Nitya Nadgir, Benedikt Stroebl, Arvind Narayanan · 2024

GPU kernel-generation tasks evaluate whether a model can replace reference operations with correct, efficient implementations.

1. KernelBench: Can LLMs Write Efficient GPU Kernels? — published result table ↗

   Ouyang, Anne, Guo, Simon, Arora, Simran, Zhang, Alex L., Hu, William, Ré, Christopher et al. · 2025

   All 7 authors

   Ouyang, Anne, Guo, Simon, Arora, Simran, Zhang, Alex L., Hu, William, Ré, Christopher, Mirhoseini, Azalia

2. Primary benchmark documentation or model evaluation report ↗

Date-bounded contest problems test coding and related code-repair or execution skills.

1. LiveCodeBench — evaluation methodology and results ↗

   Vals AI · 2026

2. LiveCodeBench Benchmark | Artificial Analysis ↗

   Artificial Analysis

Hard division Codeforces/ICPC competitive programming extension of LiveCodeBench.

1. LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming? — published result table ↗

   Zihan Zheng, Zerui Cheng, Zeyu Shen, Shang Zhou, Kaiyuan Liu, Hansen He et al. · 2025

   All 19 authors

   Zihan Zheng, Zerui Cheng, Zeyu Shen, Shang Zhou, Kaiyuan Liu, Hansen He, Dongruixuan Li, Stanley Wei, Hangyi Hao, Jianzhu Yao, Peiyao Sheng, Zixuan Wang, Wenhao Chai, Aleksandra Korolova, Peter Henderson, Sanjeev Arora, Pramod Viswanath, Jingbo Shang, Saining Xie

Programming problems are solved against an online-judge-style evaluation, including a named Python setting in model reports.

1. moonshotai/Kimi-K2.6 · Hugging Face ↗

   Moonshot AI

Official International Olympiad in Informatics problems evaluated under full subtask grading and submission limits.

1. IOI — evaluation methodology and results ↗

   Vals AI · 2026

Evaluates LLMs directly under live Codeforces Div. 1/2/3 contest conditions with human-calibrated Elo ratings.

1. CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings — published result table ↗

   Shanghaoran Quan, Jiaxi Yang, Bowen Yu, Bo Zheng, Dayiheng Liu, An Yang et al. · 2025

   All 17 authors

   Shanghaoran Quan, Jiaxi Yang, Bowen Yu, Bo Zheng, Dayiheng Liu, An Yang, Xuancheng Ren, Bofei Gao, Yibo Miao, Yunlong Feng, Zekun Wang, Jian Yang, Zeyu Cui, Yang Fan, Yichang Zhang, Binyuan Hui, Junyang Lin

307 USA Computing Olympiad problems across Bronze, Silver, Gold, and Platinum divisions.

1. Can Language Models Solve Olympiad Programming? — published result table ↗

   Shi, Quan, Tang, Michael, Narasimhan, Karthik, Yao, Shunyu · 2024

Competitive programming dataset from Codeforces, Description2Code, and CodeNet with high-coverage test cases.

1. Competition-Level Code Generation with AlphaCode — published result table ↗

   Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond et al. · 2022

   All 26 authors

   Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d'Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, Oriol Vinyals

Programming and refactoring tasks test edits to real code. Polyglot and refactoring results retain their own task sets and attempt budgets.

1. Aider Polyglot ↗

   Epoch AI

2. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

3. Aider code editing benchmark ↗

   Paul Gauthier, Aider contributors

4. Aider refactoring benchmark ↗

   Paul Gauthier, Aider contributors

1,140 practical software engineering tasks requiring complex function calls across 139 Python libraries (BigCodeBench-Hard = 148).

1. BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions — published result table ↗

   Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari et al. · 2024

   All 33 authors

   Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Liu, Zijian Wang, Binyuan Hui, Niklas Muennighoff, David Lo, Daniel Fried, Xiaoning Du, Harm de Vries, Leandro Von Werra

Function-completion problems evaluate whether generated Python programs satisfy a behavioral test suite.

1. The Llama 3 Herd of Models — published result table ↗

   Grattafiori, Aaron, Dubey, Abhimanyu, Jauhri, Abhinav, Pandey, Abhinav, Kadian, Abhishek, Al-Dahle, Ahmad et al. · 2024

   All 561 authors

   Grattafiori, Aaron, Dubey, Abhimanyu, Jauhri, Abhinav, Pandey, Abhinav, Kadian, Abhishek, Al-Dahle, Ahmad, Letman, Aiesha, Mathur, Akhil, Schelten, Alan, Vaughan, Alex, Yang, Amy, Fan, Angela, Goyal, Anirudh, Hartshorn, Anthony, Yang, Aobo, Mitra, Archi, Sravankumar, Archie, Korenev, Artem, Hinsvark, Arthur, Rao, Arun, Zhang, Aston, Rodriguez, Aurelien, Gregerson, Austen, Spataru, Ava, Roziere, Baptiste, Biron, Bethany, Tang, Binh, Chern, Bobbie, Caucheteux, Charlotte, Nayak, Chaya, Bi, Chloe, Marra, Chris, McConnell, Chris, Keller, Christian, Touret, Christophe, Wu, Chunyang, Wong, Corinne, Ferrer, Cristian Canton, Nikolaidis, Cyrus, Allonsius, Damien, Song, Daniel, Pintz, Danielle, Livshits, Danny, Wyatt, Danny, Esiobu, David, Choudhary, Dhruv, Mahajan, Dhruv, Garcia-Olano, Diego, Perino, Diego, Hupkes, Dieuwke, Lakomkin, Egor, AlBadawy, Ehab, Lobanova, Elina, Dinan, Emily, Smith, Eric Michael, Radenovic, Filip, Guzmán, Francisco, Zhang, Frank, Synnaeve, Gabriel, Lee, Gabrielle, Anderson, Georgia Lewis, Thattai, Govind, Nail, Graeme, Mialon, Gregoire, Pang, Guan, Cucurell, Guillem, Nguyen, Hailey, Korevaar, Hannah, Xu, Hu, Touvron, Hugo, Zarov, Iliyan, Ibarra, Imanol Arrieta, Kloumann, Isabel, Misra, Ishan, Evtimov, Ivan, Zhang, Jack, Copet, Jade, Lee, Jaewon, Geffert, Jan, Vranes, Jana, Park, Jason, Mahadeokar, Jay, Shah, Jeet, van der Linde, Jelmer, Billock, Jennifer, Hong, Jenny, Lee, Jenya, Fu, Jeremy, Chi, Jianfeng, Huang, Jianyu, Liu, Jiawen, Wang, Jie, Yu, Jiecao, Bitton, Joanna, Spisak, Joe, Park, Jongsoo, Rocca, Joseph, Johnstun, Joshua, Saxe, Joshua, Jia, Junteng, Alwala, Kalyan Vasuden, Prasad, Karthik, Upasani, Kartikeya, Plawiak, Kate, Li, Ke, Heafield, Kenneth, Stone, Kevin, El-Arini, Khalid, Iyer, Krithika, Malik, Kshitiz, Chiu, Kuenley, Bhalla, Kunal, Lakhotia, Kushal, Rantala-Yeary, Lauren, van der Maaten, Laurens, Chen, Lawrence, Tan, Liang, Jenkins, Liz, Martin, Louis, Madaan, Lovish, Malo, Lubo, Blecher, Lukas, Landzaat, Lukas, de Oliveira, Luke, Muzzi, Madeline, Pasupuleti, Mahesh, Singh, Mannat, Paluri, Manohar, Kardas, Marcin, Tsimpoukelli, Maria, Oldham, Mathew, Rita, Mathieu, Pavlova, Maya, Kambadur, Melanie, Lewis, Mike, Si, Min, Singh, Mitesh Kumar, Hassan, Mona, Goyal, Naman, Torabi, Narjes, Bashlykov, Nikolay, Bogoychev, Nikolay, Chatterji, Niladri, Zhang, Ning, Duchenne, Olivier, Çelebi, Onur, Alrassy, Patrick, Zhang, Pengchuan, Li, Pengwei, Vasic, Petar, Weng, Peter, Bhargava, Prajjwal, Dubal, Pratik, Krishnan, Praveen, Koura, Punit Singh, Xu, Puxin, He, Qing, Dong, Qingxiao, Srinivasan, Ragavan, Ganapathy, Raj, Calderer, Ramon, Cabral, Ricardo Silveira, Stojnic, Robert, Raileanu, Roberta, Maheswari, Rohan, Girdhar, Rohit, Patel, Rohit, Sauvestre, Romain, Polidoro, Ronnie, Sumbaly, Roshan, Taylor, Ross, Silva, Ruan, Hou, Rui, Wang, Rui, Hosseini, Saghar, Chennabasappa, Sahana, Singh, Sanjay, Bell, Sean, Kim, Seohyun Sonia, Edunov, Sergey, Nie, Shaoliang, Narang, Sharan, Raparthy, Sharath, Shen, Sheng, Wan, Shengye, Bhosale, Shruti, Zhang, Shun, Vandenhende, Simon, Batra, Soumya, Whitman, Spencer, Sootla, Sten, Collot, Stephane, Gururangan, Suchin, Borodinsky, Sydney, Herman, Tamar, Fowler, Tara, Sheasha, Tarek, Georgiou, Thomas, Scialom, Thomas, Speckbacher, Tobias, Mihaylov, Todor, Xiao, Tong, Karn, Ujjwal, Goswami, Vedanuj, Gupta, Vibhor, Ramanathan, Vignesh, Kerkez, Viktor, Gonguet, Vincent, Do, Virginie, Vogeti, Vish, Albiero, Vítor, Petrovic, Vladan, Chu, Weiwei, Xiong, Wenhan, Fu, Wenyin, Meers, Whitney, Martinet, Xavier, Wang, Xiaodong, Wang, Xiaofang, Tan, Xiaoqing Ellen, Xia, Xide, Xie, Xinfeng, Jia, Xuchao, Wang, Xuewei, Goldschlag, Yaelle, Gaur, Yashesh, Babaei, Yasmine, Wen, Yi, Song, Yiwen, Zhang, Yuchen, Li, Yue, Mao, Yuning, Coudert, Zacharie Delpierre, Yan, Zheng, Chen, Zhengxing, Papakipos, Zoe, Singh, Aaditya, Srivastava, Aayushi, Jain, Abha, Kelsey, Adam, Shajnfeld, Adam, Gangidi, Adithya, Victoria, Adolfo, Goldstand, Ahuva, Menon, Ajay, Sharma, Ajay, Boesenberg, Alex, Baevski, Alexei, Feinstein, Allie, Kallet, Amanda, Sangani, Amit, Teo, Amos, Yunus, Anam, Lupu, Andrei, Alvarado, Andres, Caples, Andrew, Gu, Andrew, Ho, Andrew, Poulton, Andrew, Ryan, Andrew, Ramchandani, Ankit, Dong, Annie, Franco, Annie, Goyal, Anuj, Saraf, Aparajita, Chowdhury, Arkabandhu, Gabriel, Ashley, Bharambe, Ashwin, Eisenman, Assaf, Yazdan, Azadeh, James, Beau, Maurer, Ben, Leonhardi, Benjamin, Huang, Bernie, Loyd, Beth, De Paola, Beto, Paranjape, Bhargavi, Liu, Bing, Wu, Bo, Ni, Boyu, Hancock, Braden, Wasti, Bram, Spence, Brandon, Stojkovic, Brani, Gamido, Brian, Montalvo, Britt, Parker, Carl, Burton, Carly, Mejia, Catalina, Liu, Ce, Wang, Changhan, Kim, Changkyu, Zhou, Chao, Hu, Chester, Chu, Ching-Hsiang, Cai, Chris, Tindal, Chris, Feichtenhofer, Christoph, Gao, Cynthia, Civin, Damon, Beaty, Dana, Kreymer, Daniel, Li, Daniel, Adkins, David, Xu, David, Testuggine, Davide, David, Delia, Parikh, Devi, Liskovich, Diana, Foss, Didem, Wang, Dingkang, Le, Duc, Holland, Dustin, Dowling, Edward, Jamil, Eissa, Montgomery, Elaine, Presani, Eleonora, Hahn, Emily, Wood, Emily, Le, Eric-Tuan, Brinkman, Erik, Arcaute, Esteban, Dunbar, Evan, Smothers, Evan, Sun, Fei, Kreuk, Felix, Tian, Feng, Kokkinos, Filippos, Ozgenel, Firat, Caggioni, Francesco, Kanayet, Frank, Seide, Frank, Florez, Gabriela Medina, Schwarz, Gabriella, Badeer, Gada, Swee, Georgia, Halpern, Gil, Herman, Grant, Sizov, Grigory, Guangyi, Zhang, Lakshminarayanan, Guna, Inan, Hakan, Shojanazeri, Hamid, Zou, Han, Wang, Hannah, Zha, Hanwen, Habeeb, Haroun, Rudolph, Harrison, Suk, Helen, Aspegren, Henry, Goldman, Hunter, Zhan, Hongyuan, Damlaj, Ibrahim, Molybog, Igor, Tufanov, Igor, Leontiadis, Ilias, Veliche, Irina-Elena, Gat, Itai, Weissman, Jake, Geboski, James, Kohli, James, Lam, Janice, Asher, Japhet, Gaya, Jean-Baptiste, Marcus, Jeff, Tang, Jeff, Chan, Jennifer, Zhen, Jenny, Reizenstein, Jeremy, Teboul, Jeremy, Zhong, Jessica, Jin, Jian, Yang, Jingyi, Cummings, Joe, Carvill, Jon, Shepard, Jon, McPhie, Jonathan, Torres, Jonathan, Ginsburg, Josh, Wang, Junjie, Wu, Kai, U, Kam Hou, Saxena, Karan, Khandelwal, Kartikay, Zand, Katayoun, Matosich, Kathy, Veeraraghavan, Kaushik, Michelena, Kelly, Li, Keqian, Jagadeesh, Kiran, Huang, Kun, Chawla, Kunal, Huang, Kyle, Chen, Lailin, Garg, Lakshya, A, Lavender, Silva, Leandro, Bell, Lee, Zhang, Lei, Guo, Liangpeng, Yu, Licheng, Moshkovich, Liron, Wehrstedt, Luca, Khabsa, Madian, Avalani, Manav, Bhatt, Manish, Mankus, Martynas, Hasson, Matan, Lennie, Matthew, Reso, Matthias, Groshev, Maxim, Naumov, Maxim, Lathi, Maya, Keneally, Meghan, Liu, Miao, Seltzer, Michael L., Valko, Michal, Restrepo, Michelle, Patel, Mihir, Vyatskov, Mik, Samvelyan, Mikayel, Clark, Mike, Macey, Mike, Wang, Mike, Hermoso, Miquel Jubert, Metanat, Mo, Rastegari, Mohammad, Bansal, Munish, Santhanam, Nandhini, Parks, Natascha, White, Natasha, Bawa, Navyata, Singhal, Nayan, Egebo, Nick, Usunier, Nicolas, Mehta, Nikhil, Laptev, Nikolay Pavlovich, Dong, Ning, Cheng, Norman, Chernoguz, Oleg, Hart, Olivia, Salpekar, Omkar, Kalinli, Ozlem, Kent, Parkin, Parekh, Parth, Saab, Paul, Balaji, Pavan, Rittner, Pedro, Bontrager, Philip, Roux, Pierre, Dollar, Piotr, Zvyagina, Polina, Ratanchandani, Prashant, Yuvraj, Pritish, Liang, Qian, Alao, Rachad, Rodriguez, Rachel, Ayub, Rafi, Murthy, Raghotham, Nayani, Raghu, Mitra, Rahul, Parthasarathy, Rangaprabhu, Li, Raymond, Hogan, Rebekkah, Battey, Robin, Wang, Rocky, Howes, Russ, Rinott, Ruty, Mehta, Sachin, Siby, Sachin, Bondu, Sai Jayesh, Datta, Samyak, Chugh, Sara, Hunt, Sara, Dhillon, Sargun, Sidorov, Sasha, Pan, Satadru, Mahajan, Saurabh, Verma, Saurabh, Yamamoto, Seiji, Ramaswamy, Sharadh, Lindsay, Shaun, Lindsay, Shaun, Feng, Sheng, Lin, Shenghao, Zha, Shengxin Cindy, Patil, Shishir, Shankar, Shiva, Zhang, Shuqiang, Zhang, Shuqiang, Wang, Sinong, Agarwal, Sneha, Sajuyigbe, Soji, Chintala, Soumith, Max, Stephanie, Chen, Stephen, Kehoe, Steve, Satterfield, Steve, Govindaprasad, Sudarshan, Gupta, Sumit, Deng, Summer, Cho, Sungmin, Virk, Sunny, Subramanian, Suraj, Choudhury, Sy, Goldman, Sydney, Remez, Tal, Glaser, Tamar, Best, Tamara, Koehler, Thilo, Robinson, Thomas, Li, Tianhe, Zhang, Tianjun, Matthews, Tim, Chou, Timothy, Shaked, Tzook, Vontimitta, Varun, Ajayi, Victoria, Montanez, Victoria, Mohan, Vijai, Kumar, Vinay Satish, Mangla, Vishal, Ionescu, Vlad, Poenaru, Vlad, Mihailescu, Vlad Tiberiu, Ivanov, Vladimir, Li, Wei, Wang, Wenchen, Jiang, Wenwen, Bouaziz, Wes, Constable, Will, Tang, Xiaocheng, Wu, Xiaojian, Wang, Xiaolan, Wu, Xilun, Gao, Xinbo, Kleinman, Yaniv, Chen, Yanjun, Hu, Ye, Jia, Ye, Qi, Ye, Li, Yenda, Zhang, Yilin, Zhang, Ying, Adi, Yossi, Nam, Youngjin, Yu, Wang, Zhao, Yu, Hao, Yuchen, Qian, Yundi, Li, Yunlu, He, Yuzi, Rait, Zach, DeVito, Zachary, Rosnbrick, Zef, Wen, Zhaoduo, Yang, Zhenyu, Zhao, Zhiwei, Ma, Zhiyu

2. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation — published result table ↗

   Liu, Jiawei, Xia, Chunqiu Steven, Wang, Yuyao, Zhang, Lingming · 2023

3. Simple Evals — model benchmark results ↗

   OpenAI · 2025

4. Primary benchmark documentation or model evaluation report ↗

Expanded tests strengthen a collection of short Python programming problems.

1. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation — published result table ↗

   Liu, Jiawei, Xia, Chunqiu Steven, Wang, Yuyao, Zhang, Lingming · 2023

2. Primary benchmark documentation or model evaluation report ↗

Massively multilingual code generation benchmarks translating HumanEval/MBPP and native problems across up to 40 programming languages.

1. Qwen3 Technical Report — published result table ↗

   An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng et al. · 2025

   All 60 authors

   An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, Zihan Qiu

1,000 realistic data-science coding problems across NumPy, Pandas, SciPy, Scikit-Learn, Matplotlib, PyTorch, and TensorFlow.

1. DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation — published result table ↗

   Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer et al. · 2022

   All 10 authors

   Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Scott Wen-tau Yih, Daniel Fried, Sida Wang, Tao Yu

800 Python functions testing code execution reasoning: predicting output from input (CruxEval-O) and input from output (CruxEval-I).

1. Qwen3 Technical Report — published result table ↗

   An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng et al. · 2025

   All 60 authors

   An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, Zihan Qiu

Enterprise text-to-SQL and agentic database querying benchmarks over Snowflake, BigQuery, Postgres, and multi-thousand-column schemas.

1. Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows — published result table ↗

   Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin Su et al. · 2024

   All 16 authors

   Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin Su, Zhaoqing Suo, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, Victor Zhong, Caiming Xiong, Ruoxi Sun, Qian Liu, Sida Wang, Tao Yu

2. Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs — published result table ↗

   Li, Jinyang, Hui, Binyuan, Qu, Ge, Yang, Jiaxi, Li, Binhua, Li, Bowen et al. · 2023

   All 18 authors

   Li, Jinyang, Hui, Binyuan, Qu, Ge, Yang, Jiaxi, Li, Binhua, Li, Bowen, Wang, Bailin, Qin, Bowen, Cao, Rongyu, Geng, Ruiying, Huo, Nan, Zhou, Xuanhe, Ma, Chenhao, Li, Guoliang, Chang, Kevin C. C., Huang, Fei, Cheng, Reynold, Li, Yongbin

Multi-file repository-level code completion and cross-file dependency retrieval benchmarks.

1. RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems — published result table ↗

   Tianyang Liu, Canwen Xu, Julian McAuley · 2023

Evaluates speech-to-text and audio models on transcribing exact code symbols, identifiers, and structured values in spoken engineering workflows.

1. VoiceCodeBench — evaluation methodology and results ↗

   Vals AI · 2026

Evaluates whether AI security agents can discover vulnerabilities in real codebases and write verified patches.

1. CyberBench — evaluation methodology and results ↗

   Vals AI · 2026

In the GPT-6 Astra report, this name refers to software reverse engineering rather than site-reliability incident handling.

1. SRE Bench — evaluation methodology and results ↗

   Vals AI · 2026

2. GPT-6 Astra: A new generation of intelligence ↗

   OpenAI

A controlled security evaluation measures model capability on vulnerability-related software tasks.

1. Introducing Kimi K3.1 Pro ↗

   Moonshot AI · 2026

2. GPT-6 Astra: A new generation of intelligence ↗

   OpenAI

3. ExploitBench v0.1 ↗

   Epoch AI

4. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

Security-oriented software evaluation appears in recent model capability reports.

1. GPT-6 Astra: A new generation of intelligence ↗

   OpenAI

A controlled security benchmark appears in MiniMax’s recent model capability report.

1. MiniMax M3.1 Flash: Redefining Agent Performance and Cost - MiniMax Research ↗

   MiniMax

A controlled software-security evaluation contributes to recent frontier-model capability comparisons.

1. Gemini 4 Argon Model Card ↗

   Google DeepMind · 2026

Published agent submissions are evaluated on 200 cybersecurity competition challenges across six categories. Compare the full-set result with category breakdowns while preserving the agent and attempt budget.

1. NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security — official leaderboard ↗

   Minghao Shao, Sofija Jancheska, Meet Udeshi, Brendan Dolan-Gavitt, Haoran Xi, Kimberly Milner et al. · 2024

   All 13 authors

   Minghao Shao, Sofija Jancheska, Meet Udeshi, Brendan Dolan-Gavitt, Haoran Xi, Kimberly Milner, Boyuan Chen, Max Yin, Siddharth Garg, Prashanth Krishnamurthy, Farshad Khorrami, Ramesh Karri, Muhammad Shafique

2. NYU CTF Bench — Cotool evaluation, February 2026 ↗

   Cotool · 2026

Assistant questions require coordinating reasoning with tools and external information.

1. GAIA: a benchmark for General AI Assistants — published result table ↗

   Mialon, Grégoire, Fourrier, Clémentine, Swift, Craig, Wolf, Thomas, LeCun, Yann, Scialom, Thomas · 2023

2. Primary benchmark documentation or model evaluation report ↗

Professional software workflows test whether an agent can deliver the requested result in a real application environment.

1. Introducing Kimi K3.1 Pro ↗

   Moonshot AI · 2026

Difficult information-seeking questions test a browsing agent’s ability to locate and combine evidence.

1. Introducing Kimi K3.1 Pro ↗

   Moonshot AI · 2026

2. Primary benchmark documentation or model evaluation report ↗

Chinese web-ecosystem counterpart to BrowseComp testing deep multi-hop search across Chinese sources.

1. BrowseComp-ZH: Benchmarking Web Browsing Ability of Large Language Models in Chinese — published result table ↗

   Peilin Zhou, Bruce Leon, Xiang Ying, Can Zhang, Yifan Shao, Qichen Ye et al. · 2025

   All 16 authors

   Peilin Zhou, Bruce Leon, Xiang Ying, Can Zhang, Yifan Shao, Qichen Ye, Dading Chong, Zhiling Jin, Chenxuan Xie, Meng Cao, Yuxin Gu, Sixin Hong, Jing Ren, Jian Chen, Chao Liu, Yining Hua

Web research tasks require collecting a complete set of answers rather than finding a single plausible fact.

1. moonshotai/Kimi-K2.6 · Hugging Face ↗

   Moonshot AI

Broad information gathering asks an agent to identify many entities and their attributes across web sources.

1. moonshotai/Kimi-K2.6 · Hugging Face ↗

   Moonshot AI

Independently evaluates native model search vs. Exa, Keenable, Parallel, and Tavily on private legal and financial research tasks.

1. Vals Web Search Index — evaluation methodology and results ↗

   Vals AI · 2026

Fact-seeking questions that remain difficult when search returns conflicting, noisy or unhelpful results. Seal-0 is the main challenge set; Seal-Hard and LongSeal probe additional reasoning and document-selection conditions.

1. SealQA: Raising the Bar for Reasoning in Search-Augmented Language Models — published result table ↗

   Pham, Thinh, Nguyen, Nguyen, Zunjare, Pratibha, Chen, Weiyuan, Tseng, Yu-Min, Vu, Tu · 2025

The OSWorld version reported in recent launches measures computer-use agents on extended application workflows.

1. Introducing Kimi K3.1 Pro ↗

   Moonshot AI · 2026

2. Introducing Claude Opus 5.5 ↗

   Anthropic

3. Epoch AI · benchmark and methodology ↗
4. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

Long-horizon computer workflows extend evaluation across applications, files, and realistic task sequences.

1. OSWorld 2.0: Benchmarking computer-use agents on long-horizon real-world tasks ↗

   OSWorld Team

2. Epoch AI · benchmark and methodology ↗
3. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

Language-model agents act in real computer environments across applications, using reproducible setup and evaluation scripts.

1. Primary benchmark documentation or model evaluation report ↗
2. Epoch AI · benchmark and methodology ↗
3. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

150+ real Windows OS desktop tasks across Edge, Office, File Explorer, VS Code, and Windows Settings.

1. Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale — published result table ↗

   Bonatti, Rogerio, Zhao, Dan, Bonacci, Francesco, Dupont, Dillon, Abdali, Sara, Li, Yinheng et al. · 2024

   All 12 authors

   Bonatti, Rogerio, Zhao, Dan, Bonacci, Francesco, Dupont, Dillon, Abdali, Sara, Li, Yinheng, Lu, Yadong, Wagle, Justin, Koishida, Kazuhito, Bucker, Arthur, Jang, Lawrence, Hui, Zack

116 dynamic mobile GUI control tasks across 20 real Android apps with randomized task parameters.

1. AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents — published result table ↗

   Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair et al. · 2024

   All 15 authors

   Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, Oriana Riva

Evaluates visual UI element grounding (clicking exact bounding boxes from natural-language instructions across mobile, desktop, and web).

1. OS-ATLAS: A Foundation Action Model for Generalist GUI Agents — published result table ↗

   Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia et al. · 2024

   All 11 authors

   Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, Yu Qiao

A model must locate the interface element referred to by a user instruction on a professional screenshot.

1. GPT-6 Astra: A new generation of intelligence ↗

   OpenAI

Self-hosted websites support realistic multi-step web tasks with checks on whether the intended state was reached.

1. WebArena: A Realistic Web Environment for Building Autonomous Agents — published result table ↗

   Zhou, Shuyan, Xu, Frank F., Zhu, Hao, Zhou, Xuhui, Lo, Robert, Sridhar, Abishek et al. · 2023

   All 12 authors

   Zhou, Shuyan, Xu, Frank F., Zhu, Hao, Zhou, Xuhui, Lo, Robert, Sridhar, Abishek, Cheng, Xianyi, Ou, Tianyue, Bisk, Yonatan, Fried, Daniel, Alon, Uri, Neubig, Graham

2. Primary benchmark documentation or model evaluation report ↗

910 multimodal web tasks requiring visual inspection of images, memes, product photos, and layout state.

1. VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks — published result table ↗

   Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang et al. · 2024

   All 10 authors

   Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, Daniel Fried

Enterprise browser automation benchmark over live ServiceNow instances testing forms, dashboards, workflows, and knowledge bases.

1. WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks? — published result table ↗

   Drouin, Alexandre, Gasse, Maxime, Caccia, Massimo, Laradji, Issam H., Del Verme, Manuel, Marty, Tom et al. · 2024

   All 12 authors

   Drouin, Alexandre, Gasse, Maxime, Caccia, Massimo, Laradji, Issam H., Del Verme, Manuel, Marty, Tom, Boisvert, Léo, Thakkar, Megh, Cappart, Quentin, Vazquez, David, Chapados, Nicolas, Lacoste, Alexandre

Generalist web navigation benchmark across 137 real-world websites (Online-Mind2Web evaluates against live websites).

1. An Illusion of Progress? Assessing the Current State of Web Agents — published result table ↗

   Xue, Tianci, Qi, Weijian, Shi, Tianneng, Song, Chan Hee, Gou, Boyu, Song, Dawn et al. · 2025

   All 8 authors

   Xue, Tianci, Qi, Weijian, Shi, Tianneng, Song, Chan Hee, Gou, Boyu, Song, Dawn, Sun, Huan, Su, Yu

Agents produce professional work from large collections of business files, with checks for correctness, analysis, and presentation.

1. Introducing Kimi K3.1 Pro ↗

   Moonshot AI · 2026

2. AA-Briefcase v1.1: Agentic Knowledge Work Benchmark | Artificial Analysis ↗

   Artificial Analysis

Agents complete occupational work and deliver documents, analyses, and other professional artifacts.

1. Introducing Kimi K3.1 Pro ↗

   Moonshot AI · 2026

2. GDPval-AA v2.1 Benchmark | Artificial Analysis ↗

   Artificial Analysis

Professional-service workflows require sustained work across files and tools.

1. Introducing Kimi K3.1 Pro ↗

   Moonshot AI · 2026

2. moonshotai/Kimi-K2.6 · Hugging Face ↗

   Moonshot AI

3. APEX-Agents ↗

   Epoch AI

4. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

Spreadsheet and document questions test the quantitative work expected of an analyst.

1. AA-AnalystAgent Benchmark | Artificial Analysis ↗

   Artificial Analysis

Business workflows require an agent to coordinate real tools and leave the simulated business systems in the required final state.

1. Primary benchmark documentation or model evaluation report ↗

Agents navigate enterprise tools and change state across business workflows.

1. EnterpriseOps-Gym-AA Benchmark | Artificial Analysis ↗

   Artificial Analysis

Agents investigate Kubernetes incidents from offline snapshots of telemetry and system structure.

1. ITBench-AA Benchmark | Artificial Analysis ↗

   Artificial Analysis

A benchmark measuring how well AI agents can complete real, economically valuable remote freelance projects end-to-end.

1. Remote Labor Index ↗

   Epoch AI

2. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

3. Scale Labs — evaluation methodology and results ↗

   Scale Labs / SEAL · 2026

Autonomous LLM AI research & development scored against published papers, human researchers, and frontier models.

1. Vals RSI Index — evaluation methodology and results ↗

   Vals AI · 2026

Economic task-automation benchmark across white-collar occupational workflows.

1. Epoch Automation Reports ↗

   Epoch AI · 2026

Simulated software company environment where agents interact with RocketChat, OwnCloud, GitLab, Plane, and NPC coworkers.

1. Benchmark reference catalog · research notes ↗
2. The Agent Company ↗

   Epoch AI

3. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

Interactive world of 9 day-to-day apps (Gmail, Spotify, Venmo, Amazon, Splitwise, Todoist) and 457 APIs governed by 100 simulated users.

1. AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents — published result table ↗

   Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li et al. · 2024

   All 9 authors

   Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, Niranjan Balasubramanian

Customer-service agents converse with simulated users while using tools and following domain policy.

1. $\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains — published result table ↗

   Yao, Shunyu, Shinn, Noah, Razavi, Pedram, Narasimhan, Karthik · 2024

2. Primary benchmark documentation or model evaluation report ↗

Both agent and simulated user can act on the shared state in technical-support tasks.

1. 𝜏²-Bench Telecom Benchmark | Artificial Analysis ↗

   Artificial Analysis

Berkeley’s function-calling evaluation covers tool invocation and broader agentic interactions.

1. Primary benchmark documentation or model evaluation report ↗
2. Epoch AI · benchmark and methodology ↗
3. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

Multi-tool tasks examine how an agent discovers, selects, and coordinates capabilities exposed through MCP.

1. Scale Labs — evaluation methodology and results ↗

   Scale Labs / SEAL · 2026

2. Introducing Kimi K3.1 Pro ↗

   Moonshot AI · 2026

Agents work through MCP-connected application and filesystem tasks with programmatic verification.

1. Primary benchmark documentation or model evaluation report ↗

Long-horizon tasks require coordinating tools and completing a practical workflow.

1. moonshotai/Kimi-K2.6 · Hugging Face ↗

   Moonshot AI

Tool-based agent evaluations report both repeatability and success across multiple attempts.

1. moonshotai/Kimi-K2.6 · Hugging Face ↗

   Moonshot AI

Evaluates how effectively coding/terminal agents read and apply modular SKILL.md / tool-skill packages.

1. SkillsBench — evaluation methodology and results ↗

   Vals AI · 2026

Agents must use operational documentation while working through a tool-based task.

1. Introducing Kimi K3.1 Pro ↗

   Moonshot AI · 2026

Presentation-generation tasks assess language-model agents producing slides from practical requests.

1. Introducing Kimi K3.1 Pro ↗

   Moonshot AI · 2026

Meta’s reported professional-tool-use evaluation measures a slice of work performed through tools.

1. Muse Wonder | Meta ↗

   Meta AI

Meta uses this evaluation to study whether an agent continues working toward a difficult objective.

1. Introducing Muse Wonder ↗

   Meta AI · 2026

Meta’s internal composite summarizes personal and knowledge-work capability.

1. Muse Wonder | Meta ↗

   Meta AI

Evaluates whether agents recognize missing/ambiguous requirements and ask targeted clarifying questions before acting.

1. HiL-Bench (Human-in-Loop Benchmark) — evaluation methodology and results ↗

   Scale Labs / SEAL · 2026

Evaluates computer-use agents playing 6 commercial video games using only raw screen pixels, keyboard, and mouse (half held out).

1. CUA-bench — evaluation methodology and results ↗

   Vals AI · 2026

Evaluates whether an AI agent can design rockets, launch, orbit, and run a space program in *Kerbal Space Program*.

1. Time Horizon Index: KSP — evaluation methodology and results ↗

   Vals AI · 2026

A test of AI systems' learning capabilities, measuring whether their scores improve across repeated playthroughs of the relatively obscure game Earthborne Rangers.

1. Earthborne Rangers (EBR-bench) ↗

   Epoch AI

2. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

Multi-table No-Limit Texas Hold'em agent evaluation measuring net profit against competing LLM agents.

1. Poker Agent — evaluation methodology and results ↗

   Vals AI · 2026

Benchmark for Agentic LLM and VLM Reasoning On Games (NetHack, MiniHack, BabyAI, Crafter, TextWorld, Baba Is You).

1. Benchmark reference catalog · research notes ↗
2. Epoch AI · benchmark and methodology ↗
3. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

Track task accuracy alongside the estimated duration of work a model can complete at a specified success rate. Minutes and task success remain separate measures.

1. METR Time Horizons ↗

   Epoch AI

2. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

3. Metr Autonomy Task Horizon Suite — methodology and published results ↗

   METR · 2026

Evaluates model routers against single models across identical agentic finance, coding, legal, and tax tasks.

1. Vals Router Index — evaluation methodology and results ↗

   Vals AI · 2026

Audits open-weight model serving quality across Together AI, Fireworks, Baseten, and first-party APIs.

1. Vals Inference Provider Index — evaluation methodology and results ↗

   Vals AI · 2026

Foundational multi-environment agent benchmark (OS, DB, KG, Web, AlfWorld) and 16,000-API tool-use benchmark.

1. AgentBench: Evaluating LLMs as Agents — published result table ↗

   Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai et al. · 2023

   All 22 authors

   Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, Jie Tang

214 realistic time-consuming web tasks (real estate, travel, shopping, local services) requiring synthesis across multiple pages.

1. AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks? — published result table ↗

   Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, Jonathan Berant · 2024

Next-generation dynamic, stateful multi-turn environment extension of GAIA.

1. Gaia2: Benchmarking LLM Agents on Dynamic and Asynchronous Environments — published result table ↗

   Froger, Romain, Andrews, Pierre, Bettini, Matteo, Budhiraja, Amar, Cabral, Ricardo Silveira, Do, Virginie et al. · 2026

   All 24 authors

   Froger, Romain, Andrews, Pierre, Bettini, Matteo, Budhiraja, Amar, Cabral, Ricardo Silveira, Do, Virginie, Garreau, Emilien, Gaya, Jean-Baptiste, Laurençon, Hugo, Lecanu, Maxime, Malkan, Kunal, Mekala, Dheeraj, Ménard, Pierre, Bertran, Gerard Moreno-Torres, Piterbarg, Ulyana, Plekhanov, Mikhail, Rita, Mathieu, Rusakov, Andrey, Vorotilov, Vladislav, Wang, Mengjue, Yu, Ian, Benhalloum, Amine, Mialon, Grégoire, Scialom, Thomas

Difficult agentic work across research, engineering and professional domains, grounded in specialist-built environments.

1. GDPx v1.0: The General Agentic Benchmark — SemiAnalysis ↗

   SemiAnalysis · 2026

2. GDPx: Measuring Frontier Model Capabilities Across the Most Valuable Domains ↗

   Max Kan, Andrew Megalaa, Dylan Patel · 2026

Scientific research evaluated across biology, chemistry, physics, astronomy, Earth science, and engineering.

1. STEMBench — Six-Domain Model Evaluation | Ulam ↗

   Ulam

The v2.1 release lists 20,000 open and partially solved physics problems, with literature triage, difficulty levels and dependencies between research questions.

1. UnsolvedPhysics — dataset card and release documentation ↗

   Ulam AI · 2026

The source release lists 25,000 chemistry research questions with current-status labels, linked literature and difficulty tiers for retrieval and research planning.

1. UnsolvedChemistry — dataset card and release documentation ↗

   Ulam AI · 2026

Version 1.5 lists 18,000 open or partially solved biology questions across 16 categories, with archived audit records and linked scientific sources.

1. UnsolvedBiology — dataset card and release documentation ↗

   Ulam AI · 2026

Version 1.1 lists 35,000 open and partially solved questions spanning astronomy, astrophysics, cosmology, planetary science and heliophysics.

1. UnsolvedAstronomy — dataset card and release documentation ↗

   Ulam AI · 2026

Version 1.3 lists 20,000 open and partially solved Earth and environmental science questions, including a curated horizon-scan subset and archived review records.

1. UnsolvedEarth — dataset card and release documentation ↗

   Ulam AI · 2026

Version 2.0 lists 20,000 open or partially resolved engineering questions across 16 domains, including a 2,000-problem curated subset.

1. UnsolvedEngineering — dataset card and release documentation ↗

   Ulam AI · 2026

Research workflows are completed in a terminal and checked with task-specific tests.

1. Introducing Kimi K3.1 Pro ↗

   Moonshot AI · 2026

2. Terminal-Bench-Science — evaluation methodology and results ↗

   Vals AI · 2026

3. Terminal-Bench-Science 0.1 Benchmark | Artificial Analysis ↗

   Artificial Analysis

Composite index averaging MysteryMechanism, BioMysteryBench, and Terminal-Bench-Science.

1. Vals Science Index — evaluation methodology and results ↗

   Vals AI · 2026

Evaluates whether agents can rediscover sealed mathematical and physical governing laws through bounded active experiments.

1. MysteryMechanism — evaluation methodology and results ↗

   Vals AI · 2026

Biological investigations test agentic reasoning through a scientific problem and its supporting evidence.

1. Introducing Kimi K3.1 Pro ↗

   Moonshot AI · 2026

2. BioMysteryBench — evaluation methodology and results ↗

   Vals AI · 2026

Research-level physics challenges are attempted with code execution in an offline Linux sandbox.

1. CritPt v2 Benchmark | Artificial Analysis ↗

   Artificial Analysis

The original physics benchmark tests difficult research reasoning across composite challenges.

1. CritPt Benchmark | Artificial Analysis ↗

   Artificial Analysis

2. CritPt v1 ↗

   Epoch AI

3. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

Scientist-authored coding problems require translating domain knowledge into numerical implementations.

1. SciCode Benchmark | Artificial Analysis ↗

   Artificial Analysis

2. Epoch AI · benchmark and methodology ↗
3. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

A demanding subset of expert-written science questions tests detailed reasoning in specialist domains.

1. Introducing Kimi K3.1 Pro ↗

   Moonshot AI · 2026

2. GPQA Diamond — evaluation methodology and results ↗

   Vals AI · 2026

3. GPQA Diamond Benchmark | Artificial Analysis ↗

   Artificial Analysis

4. GPQA Diamond ↗

   Epoch AI

5. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

The broader GPQA science evaluation complements the more commonly reported Diamond subset.

1. GPQA: A Graduate-Level Google-Proof Q&A Benchmark — published result table ↗

   Rein, David, Hou, Betty Li, Stickland, Asa Cooper, Petty, Jackson, Pang, Richard Yuanzhe, Dirani, Julien et al. · 2023

   All 8 authors

   Rein, David, Hou, Betty Li, Stickland, Asa Cooper, Petty, Jackson, Pang, Richard Yuanzhe, Dirani, Julien, Michael, Julian, Bowman, Samuel R.

2. GPQA: A Graduate-Level Google-Proof Q&A Benchmark ↗

   Rein, David, Hou, Betty Li, Stickland, Asa Cooper, Petty, Jackson, Pang, Richard Yuanzhe, Dirani, Julien et al. · 2023

   All 8 authors

   Rein, David, Hou, Betty Li, Stickland, Asa Cooper, Petty, Jackson, Pang, Richard Yuanzhe, Dirani, Julien, Michael, Julian, Bowman, Samuel R.

26,529 graduate-level questions across 285 disciplines (expanding beyond natural sciences into engineering, medicine, agronomy, and humanities).

1. SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines — published result table ↗

   M-A-P Team, Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng et al. · 2025

   All 97 authors

   M-A-P Team, Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, King Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, Chujie Zheng, Kaixin Deng, Shawn Gavin, Shian Jia, Sichao Jiang, Yiyan Liao, Rui Li, Qinrui Li, Sirun Li, Yizhi Li, Yunwen Li, David Ma, Yuansheng Ni, Haoran Que, Qiyao Wang, Zhoufutu Wen, Siwei Wu, Tyshawn Hsing, Ming Xu, Zhenzhu Yang, Zekun Moore Wang, Junting Zhou, Yuelin Bai, Xingyuan Bu, Chenglin Cai, Liang Chen, Yifan Chen, Chengtuo Cheng, Tianhao Cheng, Keyi Ding, Siming Huang, Yun Huang, Yaoru Li, Yizhe Li, Zhaoqun Li, Tianhao Liang, Chengdong Lin, Hongquan Lin, Yinghao Ma, Tianyang Pang, Zhongyuan Peng, Zifan Peng, Qige Qi, Shi Qiu, Xingwei Qu, Shanghaoran Quan, Yizhou Tan, Zili Wang, Chenqing Wang, Hao Wang, Yiya Wang, Yubo Wang, Jiajun Xu, Kexin Yang, Ruibin Yuan, Yuanhao Yue, Tianyang Zhan, Chun Zhang, Jinyang Zhang, Xiyue Zhang, Xingjian Zhang, Yue Zhang, Yongchi Zhao, Xiangyu Zheng, Chenghua Zhong, Yang Gao, Zhoujun Li, Dayiheng Liu, Qian Liu, Tianyu Liu, Shiwen Ni, Junran Peng, Yujia Qin, Wenbo Su, Guoyin Wang, Shi Wang, Jian Yang, Min Yang, Meng Cao, Xiang Yue, Zhaoxiang Zhang, Wangchunshu Zhou, Jiaheng Liu, Qunshu Lin, Wenhao Huang, Ge Zhang

Scientific tasks evaluate reasoning and computational work with domain-relevant tools.

1. Primary benchmark documentation or model evaluation report ↗

2,400+ questions evaluating literature retrieval (LitQA2), protocol troubleshooting (ProtocolQA), figure/table interpretation (FigQA/TableQA), DNA cloning (CloningScenarios), and sequence manipulation (SeqQA).

1. LAB-Bench: Measuring Capabilities of Language Models for Biology Research — published result table ↗

   Jon M. Laurent, Joseph D. Janizek, Michael Ruzo, Michaela M. Hinks, Michael J. Hammerling, Siddharth Narayanan et al. · 2024

   All 9 authors

   Jon M. Laurent, Joseph D. Janizek, Michael Ruzo, Michaela M. Hinks, Michael J. Hammerling, Siddharth Narayanan, Manvitha Ponnapati, Andrew D. White, Samuel G. Rodriques

Real-world bioinformatics agent benchmark requiring R/Python analysis over raw sequencing and multi-omics datasets.

1. BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biology — published result table ↗

   Mitchener, Ludovico, Laurent, Jon M, Andonian, Alex, Tenmann, Benjamin, Narayanan, Siddharth, Wellawatte, Geemi P et al. · 2025

   All 9 authors

   Mitchener, Ludovico, Laurent, Jon M, Andonian, Alex, Tenmann, Benjamin, Narayanan, Siddharth, Wellawatte, Geemi P, White, Andrew, Sani, Lorenzo, Rodriques, Samuel G

2. BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biology — published result table ↗

   Mitchener, Ludovico, Laurent, Jon M, Andonian, Alex, Tenmann, Benjamin, Narayanan, Siddharth, Wellawatte, Geemi P et al. · 2025

   All 9 authors

   Mitchener, Ludovico, Laurent, Jon M, Andonian, Alex, Tenmann, Benjamin, Narayanan, Siddharth, Wellawatte, Geemi P, White, Andrew, Sani, Lorenzo, Rodriques, Samuel G

Life-science questions probe specialist reasoning and the use of relevant scientific knowledge.

1. GPT-6 Astra: A new generation of intelligence ↗

   OpenAI

Specialist genetics evaluation appears in recent frontier-model science reports.

1. GPT-6 Astra: A new generation of intelligence ↗

   OpenAI

OpenAI’s internal medicinal-chemistry evaluation measures a specialist slice of scientific reasoning.

1. GPT-6 Astra: A new generation of intelligence ↗

   OpenAI

82 expert-curated computational drug discovery tasks executed by coding agents.

1. DrugDiscoveryBench — evaluation methodology and results ↗

   Scale Labs / SEAL · 2026

Wet-lab biological protocol mistake identification benchmark.

1. BioLP-bench: Measuring understanding of biological lab protocols by large language models ↗

   Igor Ivanov · 2024

2,700+ curated chemistry questions spanning organic, inorganic, physical, analytical, computational, and chemical safety.

1. Are large language models superhuman chemists? — published result table ↗

   Adrian Mirza, Nawaf Alampara, Sreekanth Kunchapu, Martiño Ríos-García, Benedict Emoekabu, Aswanth Krishnan et al. · 2024

   All 35 authors

   Adrian Mirza, Nawaf Alampara, Sreekanth Kunchapu, Martiño Ríos-García, Benedict Emoekabu, Aswanth Krishnan, Tanya Gupta, Mara Schilling-Wilhelmi, Macjonathan Okereke, Anagha Aneesh, Amir Mohammad Elahi, Mehrdad Asgari, Juliane Eberhardt, Hani M. Elbeheiry, María Victoria Gil, Maximilian Greiner, Caroline T. Holick, Christina Glaubitz, Tim Hoffmann, Abdelrahman Ibrahim, Lea C. Klepsch, Yannik Köster, Fabian Alexander Kreth, Jakob Meyer, Santiago Miret, Jan Matthias Peschel, Michael Ringleb, Nicole Roesner, Johanna Schreiber, Ulrich S. Schubert, Leanne M. Stafast, Dinga Wonanke, Michael Pieler, Philippe Schwaller, Kevin Maik Jablonka

5,520 undergraduate physics problems (in English and Chinese) across 13 physics courses with custom symbolic equivalence verification.

1. UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models — published result table ↗

   Xin Xu, Qiyun Xu, Tong Xiao, Tianhao Chen, Yuchen Yan, Jiaxin Zhang et al. · 2025

   All 9 authors

   Xin Xu, Qiyun Xu, Tong Xiao, Tianhao Chen, Yuchen Yan, Jiaxin Zhang, Shizhe Diao, Can Yang, Yang Wang

Olympiad-level physics problems test scientific reasoning with bilingual text and visual inputs. The published suite covers mathematics and physics; chemistry is not a reported component.

1. OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems — published result table ↗

   Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen et al. · 2024

   All 14 authors

   Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, Maosong Sun

102 data-driven scientific discovery coding tasks extracted from peer-reviewed papers across bioinformatics, computational chemistry, GIS, and psychology.

1. ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery — published result table ↗

   Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu et al. · 2024

   All 20 authors

   Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu-Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su, Huan Sun

Virtual interactive environment where agents formulate hypotheses, design controlled experiments, gather data, and discover scientific laws.

1. DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents — published result table ↗

   Jansen, Peter, Côté, Marc-Alexandre, Khot, Tushar, Bransom, Erin, Mishra, Bhavana Dalvi, Majumder, Bodhisattwa Prasad et al. · 2024

   All 8 authors

   Jansen, Peter, Côté, Marc-Alexandre, Khot, Tushar, Bransom, Erin, Mishra, Bhavana Dalvi, Majumder, Bodhisattwa Prasad, Tafjord, Oyvind, Clark, Peter

24 real-time MuJoCo physics tasks (13 robot arm, 11 humanoid) controlled purely from camera feeds and raw joint motor commands within 3 minutes.

1. Robo-bench — evaluation methodology and results ↗

   Vals AI · 2026

Spatial 3D geometric reasoning and multi-part assembly benchmark administered by Epoch AI.

1. Publisher directory ↗
2. Furniture Assembly Benchmark ↗

   Epoch AI

3. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

Materials-discovery models are compared on structure and energy predictions using Matbench and Matbench Discovery protocols. These are scientific prediction models, rather than a general language-model question-answering test.

1. Matbench Discovery — materials-discovery leaderboard ↗

   Janosh Riebesell

217+ deep mutational scanning assays (2.7M+ variants) evaluating protein fitness and design models.

1. RITA: a Study on Scaling Up Generative Protein Sequence Models — published result table ↗

   Daniel Hesslow, Niccoló Zanichelli, Pascal Notin, Iacopo Poli, Debora Marks · 2022

Generalist biomedical agent evaluation suite across genetics, pharmacology, pathology, and clinical microbiology.

1. Biomni-R0: Using RL to Hill-Climb Biomedical Reasoning Agents to Expert-Level ↗

   Ryan Li, Kexin Huang, Shiyi Cao, Yuanhao Qu, Jure Leskovec · 2025

Visual-Spatial Intelligence benchmark measuring 3D room layout, object distance, relative direction, and camera motion from video.

1. Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces — published result table ↗

   Yang, Jihan, Yang, Shusheng, Gupta, Anjali W., Han, Rilyn, Fei-Fei, Li, Xie, Saining · 2024

Multimodal physical world understanding benchmark across mechanics, optics, thermodynamics, and material properties.

1. PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding — published result table ↗

   Chow, Wei, Mao, Jiageng, Li, Boyi, Seita, Daniel, Guizilini, Vitor, Wang, Yue · 2025

College-level textbook problems in mathematics, chemistry, and physics requiring open-ended numerical derivations.

1. SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models — published result table ↗

   Wang, Xiaoxuan, Hu, Ziniu, Lu, Pan, Zhu, Yanqiao, Zhang, Jieyu, Subramaniam, Satyen et al. · 2023

   All 10 authors

   Wang, Xiaoxuan, Hu, Ziniu, Lu, Pan, Zhu, Yanqiao, Zhang, Jieyu, Subramaniam, Satyen, Loomba, Arjun R., Zhang, Shichang, Sun, Yizhou, Wang, Wei

Nine evaluations connect knowledge work, coding, science and general capability in one versioned score.

1. Artificial Analysis Intelligence Index v5.0 — Artificial Analysis ↗

   Artificial Analysis

2. Artificial Analysis Intelligence Benchmarking Methodology — v5.0 ↗

   Artificial Analysis

A common capability scale fitted across many benchmarks, with uncertainty estimates and a historical view of model progress.

1. AI benchmarks & capabilities ↗

   Epoch AI

2. Epoch AI · ECI methodology ↗
3. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

Expert-authored questions stretch academic reasoning and knowledge across many disciplines.

1. Introducing Kimi K3.1 Pro ↗

   Moonshot AI · 2026

2. Humanity's Last Exam Benchmark | Artificial Analysis ↗

   Artificial Analysis

3. Humanity's Last Exam ↗

   Epoch AI

4. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

A named challenging subset of Humanity’s Last Exam appears in recent model launches.

1. Introducing Kimi K3.1 Pro ↗

   Moonshot AI · 2026

2. Scale Labs — evaluation methodology and results ↗

   Scale Labs / SEAL · 2026

Continuously refreshed dynamic split of Humanity's Last Exam to prevent saturation and leakage.

1. HLE with Gemini 3 Pro — FUSE HLE-Verifications ↗

   FUSE-verifiers · 2026

Few-shot grid transformations test the discovery and application of unfamiliar rules.

1. Primary benchmark documentation or model evaluation report ↗
2. ARC-AGI-1 ↗

   Epoch AI

3. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

Novel grid-transformation problems test abstraction and adaptation from a small number of examples.

1. Primary benchmark documentation or model evaluation report ↗
2. ARC-AGI-2 ↗

   Epoch AI

3. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

Interactive environments assess how an agent explores, learns rules, and adapts to unfamiliar tasks.

1. Primary benchmark documentation or model evaluation report ↗

Intuitive visual, spatial, and physical common-sense reasoning across complex image and video scenarios.

1. Humanity's Sixth Sense — evaluation methodology and results ↗

   Scale Labs / SEAL · 2026

Challenging visual questions probe failures that remain after ordinary image understanding becomes strong.

1. Introducing Kimi K3.1 Pro ↗

   Moonshot AI · 2026

Visual problem solving tests whether a model can reason over image structure rather than rely only on familiar textual knowledge.

1. Introducing Kimi K3.1 Pro ↗

   Moonshot AI · 2026

2. deepseek-ai/DeepSeek-V4.1-Flash-1017 · Hugging Face ↗

   DeepSeek AI

Specialist charts test visual reasoning grounded in professional practice.

1. Introducing Kimi K3.1 Pro ↗

   Moonshot AI · 2026

2. Primary benchmark documentation or model evaluation report ↗
3. Epoch AI · benchmark and methodology ↗
4. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

Scientific charts require models to read visual information and reason over what the figure actually shows.

1. Introducing Kimi K3.1 Pro ↗

   Moonshot AI · 2026

2. moonshotai/Kimi-K2.6 · Hugging Face ↗

   Moonshot AI

College-level questions combine language, visual material and subject knowledge across disciplines.

1. Primary benchmark documentation or model evaluation report ↗

Academic questions require visual understanding as well as language reasoning, with measures against text-only shortcuts.

1. Introducing Kimi K3.1 Pro ↗

   Moonshot AI · 2026

2. MMMU Pro — evaluation methodology and results ↗

   Vals AI · 2026

3. MMMU-Pro Benchmark | Artificial Analysis ↗

   Artificial Analysis

A challenging subset of BIG-bench tests varied reasoning skills beyond straightforward knowledge recall.

1. Primary benchmark documentation or model evaluation report ↗
2. Epoch AI · benchmark and methodology ↗
3. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

Replacement for saturated BBH: 23 drastically harder tasks where frontier non-thinking models score <15% and o1/R1 reasoners are stress-tested.

1. BIG-Bench Extra Hard — published result table ↗

   Kazemi, Mehran, Fatemi, Bahare, Bansal, Hritik, Palowitch, John, Anastasiou, Chrysovalantis, Mehta, Sanket Vaibhav et al. · 2025

   All 20 authors

   Kazemi, Mehran, Fatemi, Bahare, Bansal, Hritik, Palowitch, John, Anastasiou, Chrysovalantis, Mehta, Sanket Vaibhav, Jain, Lalit K., Aglietti, Virginia, Jindal, Disha, Chen, Peter, Dikkala, Nishanth, Tyen, Gladys, Liu, Xin, Shalit, Uri, Chiappa, Silvia, Olszewska, Kate, Tay, Yi, Tran, Vinh Q., Le, Quoc V., Firat, Orhan

Contamination-free monthly-updated benchmark with verifiable ground-truth scoring across Math, Coding, Reasoning, Language, Data Analysis, and IF.

1. Benchmark reference catalog · research notes ↗
2. Epoch AI · benchmark and methodology ↗
3. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

1,000 constraint-satisfaction logic grid puzzles (Einstein puzzles) with controllable search-space complexity from $2\times 2$ to $6\times 6$.

1. ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning — published result table ↗

   Bill Yuchen Lin, Ronan Le Bras, Kyle Richardson, Ashish Sabharwal, Radha Poovendran, Peter Clark et al. · 2025

   All 7 authors

   Bill Yuchen Lin, Ronan Le Bras, Kyle Richardson, Ashish Sabharwal, Radha Poovendran, Peter Clark, Yejin Choi

Knowledge-Orthogonal Reasoning benchmark testing rule application over newly invented symbolic, cipher, puzzle, and counterfactual rules.

1. KOR-Bench: Benchmarking Language Models on Knowledge-Orthogonal Reasoning Tasks — published result table ↗

   Kaijing Ma, Xinrun Du, Yunran Wang, Haoran Zhang, Zhoufutu Wen, Xingwei Qu et al. · 2024

   All 12 authors

   Kaijing Ma, Xinrun Du, Yunran Wang, Haoran Zhang, Zhoufutu Wen, Xingwei Qu, Jian Yang, Jiaheng Liu, Minghao Liu, Xiang Yue, Wenhao Huang, Ge Zhang

Complex multi-step lateral thinking, cryptographic, and puzzle-hunt benchmark (MIT Mystery Hunt style).

1. Publisher directory ↗
2. Epoch AI · benchmark and methodology ↗
3. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

Evaluates classical PDDL action planning, plan verification, replanning, and obfuscated symbol planning (Mystery Blocksworld).

1. PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change — published result table ↗

   Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, Subbarao Kambhampati · 2022

Evaluates multi-step deductive reasoning embedded in long natural-language narratives (murder mysteries, object placement, team allocation).

1. MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning — published result table ↗

   Zayne Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, Greg Durrett · 2023

Reading comprehension requiring discrete operations (addition, counting, sorting) over sports and history passages.

1. The Llama 3 Herd of Models — published result table ↗

   Grattafiori, Aaron, Dubey, Abhimanyu, Jauhri, Abhinav, Pandey, Abhinav, Kadian, Abhishek, Al-Dahle, Ahmad et al. · 2024

   All 561 authors

   Grattafiori, Aaron, Dubey, Abhimanyu, Jauhri, Abhinav, Pandey, Abhinav, Kadian, Abhishek, Al-Dahle, Ahmad, Letman, Aiesha, Mathur, Akhil, Schelten, Alan, Vaughan, Alex, Yang, Amy, Fan, Angela, Goyal, Anirudh, Hartshorn, Anthony, Yang, Aobo, Mitra, Archi, Sravankumar, Archie, Korenev, Artem, Hinsvark, Arthur, Rao, Arun, Zhang, Aston, Rodriguez, Aurelien, Gregerson, Austen, Spataru, Ava, Roziere, Baptiste, Biron, Bethany, Tang, Binh, Chern, Bobbie, Caucheteux, Charlotte, Nayak, Chaya, Bi, Chloe, Marra, Chris, McConnell, Chris, Keller, Christian, Touret, Christophe, Wu, Chunyang, Wong, Corinne, Ferrer, Cristian Canton, Nikolaidis, Cyrus, Allonsius, Damien, Song, Daniel, Pintz, Danielle, Livshits, Danny, Wyatt, Danny, Esiobu, David, Choudhary, Dhruv, Mahajan, Dhruv, Garcia-Olano, Diego, Perino, Diego, Hupkes, Dieuwke, Lakomkin, Egor, AlBadawy, Ehab, Lobanova, Elina, Dinan, Emily, Smith, Eric Michael, Radenovic, Filip, Guzmán, Francisco, Zhang, Frank, Synnaeve, Gabriel, Lee, Gabrielle, Anderson, Georgia Lewis, Thattai, Govind, Nail, Graeme, Mialon, Gregoire, Pang, Guan, Cucurell, Guillem, Nguyen, Hailey, Korevaar, Hannah, Xu, Hu, Touvron, Hugo, Zarov, Iliyan, Ibarra, Imanol Arrieta, Kloumann, Isabel, Misra, Ishan, Evtimov, Ivan, Zhang, Jack, Copet, Jade, Lee, Jaewon, Geffert, Jan, Vranes, Jana, Park, Jason, Mahadeokar, Jay, Shah, Jeet, van der Linde, Jelmer, Billock, Jennifer, Hong, Jenny, Lee, Jenya, Fu, Jeremy, Chi, Jianfeng, Huang, Jianyu, Liu, Jiawen, Wang, Jie, Yu, Jiecao, Bitton, Joanna, Spisak, Joe, Park, Jongsoo, Rocca, Joseph, Johnstun, Joshua, Saxe, Joshua, Jia, Junteng, Alwala, Kalyan Vasuden, Prasad, Karthik, Upasani, Kartikeya, Plawiak, Kate, Li, Ke, Heafield, Kenneth, Stone, Kevin, El-Arini, Khalid, Iyer, Krithika, Malik, Kshitiz, Chiu, Kuenley, Bhalla, Kunal, Lakhotia, Kushal, Rantala-Yeary, Lauren, van der Maaten, Laurens, Chen, Lawrence, Tan, Liang, Jenkins, Liz, Martin, Louis, Madaan, Lovish, Malo, Lubo, Blecher, Lukas, Landzaat, Lukas, de Oliveira, Luke, Muzzi, Madeline, Pasupuleti, Mahesh, Singh, Mannat, Paluri, Manohar, Kardas, Marcin, Tsimpoukelli, Maria, Oldham, Mathew, Rita, Mathieu, Pavlova, Maya, Kambadur, Melanie, Lewis, Mike, Si, Min, Singh, Mitesh Kumar, Hassan, Mona, Goyal, Naman, Torabi, Narjes, Bashlykov, Nikolay, Bogoychev, Nikolay, Chatterji, Niladri, Zhang, Ning, Duchenne, Olivier, Çelebi, Onur, Alrassy, Patrick, Zhang, Pengchuan, Li, Pengwei, Vasic, Petar, Weng, Peter, Bhargava, Prajjwal, Dubal, Pratik, Krishnan, Praveen, Koura, Punit Singh, Xu, Puxin, He, Qing, Dong, Qingxiao, Srinivasan, Ragavan, Ganapathy, Raj, Calderer, Ramon, Cabral, Ricardo Silveira, Stojnic, Robert, Raileanu, Roberta, Maheswari, Rohan, Girdhar, Rohit, Patel, Rohit, Sauvestre, Romain, Polidoro, Ronnie, Sumbaly, Roshan, Taylor, Ross, Silva, Ruan, Hou, Rui, Wang, Rui, Hosseini, Saghar, Chennabasappa, Sahana, Singh, Sanjay, Bell, Sean, Kim, Seohyun Sonia, Edunov, Sergey, Nie, Shaoliang, Narang, Sharan, Raparthy, Sharath, Shen, Sheng, Wan, Shengye, Bhosale, Shruti, Zhang, Shun, Vandenhende, Simon, Batra, Soumya, Whitman, Spencer, Sootla, Sten, Collot, Stephane, Gururangan, Suchin, Borodinsky, Sydney, Herman, Tamar, Fowler, Tara, Sheasha, Tarek, Georgiou, Thomas, Scialom, Thomas, Speckbacher, Tobias, Mihaylov, Todor, Xiao, Tong, Karn, Ujjwal, Goswami, Vedanuj, Gupta, Vibhor, Ramanathan, Vignesh, Kerkez, Viktor, Gonguet, Vincent, Do, Virginie, Vogeti, Vish, Albiero, Vítor, Petrovic, Vladan, Chu, Weiwei, Xiong, Wenhan, Fu, Wenyin, Meers, Whitney, Martinet, Xavier, Wang, Xiaodong, Wang, Xiaofang, Tan, Xiaoqing Ellen, Xia, Xide, Xie, Xinfeng, Jia, Xuchao, Wang, Xuewei, Goldschlag, Yaelle, Gaur, Yashesh, Babaei, Yasmine, Wen, Yi, Song, Yiwen, Zhang, Yuchen, Li, Yue, Mao, Yuning, Coudert, Zacharie Delpierre, Yan, Zheng, Chen, Zhengxing, Papakipos, Zoe, Singh, Aaditya, Srivastava, Aayushi, Jain, Abha, Kelsey, Adam, Shajnfeld, Adam, Gangidi, Adithya, Victoria, Adolfo, Goldstand, Ahuva, Menon, Ajay, Sharma, Ajay, Boesenberg, Alex, Baevski, Alexei, Feinstein, Allie, Kallet, Amanda, Sangani, Amit, Teo, Amos, Yunus, Anam, Lupu, Andrei, Alvarado, Andres, Caples, Andrew, Gu, Andrew, Ho, Andrew, Poulton, Andrew, Ryan, Andrew, Ramchandani, Ankit, Dong, Annie, Franco, Annie, Goyal, Anuj, Saraf, Aparajita, Chowdhury, Arkabandhu, Gabriel, Ashley, Bharambe, Ashwin, Eisenman, Assaf, Yazdan, Azadeh, James, Beau, Maurer, Ben, Leonhardi, Benjamin, Huang, Bernie, Loyd, Beth, De Paola, Beto, Paranjape, Bhargavi, Liu, Bing, Wu, Bo, Ni, Boyu, Hancock, Braden, Wasti, Bram, Spence, Brandon, Stojkovic, Brani, Gamido, Brian, Montalvo, Britt, Parker, Carl, Burton, Carly, Mejia, Catalina, Liu, Ce, Wang, Changhan, Kim, Changkyu, Zhou, Chao, Hu, Chester, Chu, Ching-Hsiang, Cai, Chris, Tindal, Chris, Feichtenhofer, Christoph, Gao, Cynthia, Civin, Damon, Beaty, Dana, Kreymer, Daniel, Li, Daniel, Adkins, David, Xu, David, Testuggine, Davide, David, Delia, Parikh, Devi, Liskovich, Diana, Foss, Didem, Wang, Dingkang, Le, Duc, Holland, Dustin, Dowling, Edward, Jamil, Eissa, Montgomery, Elaine, Presani, Eleonora, Hahn, Emily, Wood, Emily, Le, Eric-Tuan, Brinkman, Erik, Arcaute, Esteban, Dunbar, Evan, Smothers, Evan, Sun, Fei, Kreuk, Felix, Tian, Feng, Kokkinos, Filippos, Ozgenel, Firat, Caggioni, Francesco, Kanayet, Frank, Seide, Frank, Florez, Gabriela Medina, Schwarz, Gabriella, Badeer, Gada, Swee, Georgia, Halpern, Gil, Herman, Grant, Sizov, Grigory, Guangyi, Zhang, Lakshminarayanan, Guna, Inan, Hakan, Shojanazeri, Hamid, Zou, Han, Wang, Hannah, Zha, Hanwen, Habeeb, Haroun, Rudolph, Harrison, Suk, Helen, Aspegren, Henry, Goldman, Hunter, Zhan, Hongyuan, Damlaj, Ibrahim, Molybog, Igor, Tufanov, Igor, Leontiadis, Ilias, Veliche, Irina-Elena, Gat, Itai, Weissman, Jake, Geboski, James, Kohli, James, Lam, Janice, Asher, Japhet, Gaya, Jean-Baptiste, Marcus, Jeff, Tang, Jeff, Chan, Jennifer, Zhen, Jenny, Reizenstein, Jeremy, Teboul, Jeremy, Zhong, Jessica, Jin, Jian, Yang, Jingyi, Cummings, Joe, Carvill, Jon, Shepard, Jon, McPhie, Jonathan, Torres, Jonathan, Ginsburg, Josh, Wang, Junjie, Wu, Kai, U, Kam Hou, Saxena, Karan, Khandelwal, Kartikay, Zand, Katayoun, Matosich, Kathy, Veeraraghavan, Kaushik, Michelena, Kelly, Li, Keqian, Jagadeesh, Kiran, Huang, Kun, Chawla, Kunal, Huang, Kyle, Chen, Lailin, Garg, Lakshya, A, Lavender, Silva, Leandro, Bell, Lee, Zhang, Lei, Guo, Liangpeng, Yu, Licheng, Moshkovich, Liron, Wehrstedt, Luca, Khabsa, Madian, Avalani, Manav, Bhatt, Manish, Mankus, Martynas, Hasson, Matan, Lennie, Matthew, Reso, Matthias, Groshev, Maxim, Naumov, Maxim, Lathi, Maya, Keneally, Meghan, Liu, Miao, Seltzer, Michael L., Valko, Michal, Restrepo, Michelle, Patel, Mihir, Vyatskov, Mik, Samvelyan, Mikayel, Clark, Mike, Macey, Mike, Wang, Mike, Hermoso, Miquel Jubert, Metanat, Mo, Rastegari, Mohammad, Bansal, Munish, Santhanam, Nandhini, Parks, Natascha, White, Natasha, Bawa, Navyata, Singhal, Nayan, Egebo, Nick, Usunier, Nicolas, Mehta, Nikhil, Laptev, Nikolay Pavlovich, Dong, Ning, Cheng, Norman, Chernoguz, Oleg, Hart, Olivia, Salpekar, Omkar, Kalinli, Ozlem, Kent, Parkin, Parekh, Parth, Saab, Paul, Balaji, Pavan, Rittner, Pedro, Bontrager, Philip, Roux, Pierre, Dollar, Piotr, Zvyagina, Polina, Ratanchandani, Prashant, Yuvraj, Pritish, Liang, Qian, Alao, Rachad, Rodriguez, Rachel, Ayub, Rafi, Murthy, Raghotham, Nayani, Raghu, Mitra, Rahul, Parthasarathy, Rangaprabhu, Li, Raymond, Hogan, Rebekkah, Battey, Robin, Wang, Rocky, Howes, Russ, Rinott, Ruty, Mehta, Sachin, Siby, Sachin, Bondu, Sai Jayesh, Datta, Samyak, Chugh, Sara, Hunt, Sara, Dhillon, Sargun, Sidorov, Sasha, Pan, Satadru, Mahajan, Saurabh, Verma, Saurabh, Yamamoto, Seiji, Ramaswamy, Sharadh, Lindsay, Shaun, Lindsay, Shaun, Feng, Sheng, Lin, Shenghao, Zha, Shengxin Cindy, Patil, Shishir, Shankar, Shiva, Zhang, Shuqiang, Zhang, Shuqiang, Wang, Sinong, Agarwal, Sneha, Sajuyigbe, Soji, Chintala, Soumith, Max, Stephanie, Chen, Stephen, Kehoe, Steve, Satterfield, Steve, Govindaprasad, Sudarshan, Gupta, Sumit, Deng, Summer, Cho, Sungmin, Virk, Sunny, Subramanian, Suraj, Choudhury, Sy, Goldman, Sydney, Remez, Tal, Glaser, Tamar, Best, Tamara, Koehler, Thilo, Robinson, Thomas, Li, Tianhe, Zhang, Tianjun, Matthews, Tim, Chou, Timothy, Shaked, Tzook, Vontimitta, Varun, Ajayi, Victoria, Montanez, Victoria, Mohan, Vijai, Kumar, Vinay Satish, Mangla, Vishal, Ionescu, Vlad, Poenaru, Vlad, Mihailescu, Vlad Tiberiu, Ivanov, Vladimir, Li, Wei, Wang, Wenchen, Jiang, Wenwen, Bouaziz, Wes, Constable, Will, Tang, Xiaocheng, Wu, Xiaojian, Wang, Xiaolan, Wu, Xilun, Gao, Xinbo, Kleinman, Yaniv, Chen, Yanjun, Hu, Ye, Jia, Ye, Qi, Ye, Li, Yenda, Zhang, Yilin, Zhang, Ying, Adi, Yossi, Nam, Youngjin, Yu, Wang, Zhao, Yu, Hao, Yuchen, Qian, Yundi, Li, Yunlu, He, Yuzi, Rait, Zach, DeVito, Zachary, Rosnbrick, Zef, Wen, Zhaoduo, Yang, Zhenyu, Zhao, Zhiwei, Ma, Zhiyu

2. Simple Evals — model benchmark results ↗

   OpenAI · 2025

1,172 grade-school science multiple-choice questions (distinct from Chollet's ARC-AGI).

1. The Llama 3 Herd of Models — published result table ↗

   Grattafiori, Aaron, Dubey, Abhimanyu, Jauhri, Abhinav, Pandey, Abhinav, Kadian, Abhishek, Al-Dahle, Ahmad et al. · 2024

   All 561 authors

   Grattafiori, Aaron, Dubey, Abhimanyu, Jauhri, Abhinav, Pandey, Abhinav, Kadian, Abhishek, Al-Dahle, Ahmad, Letman, Aiesha, Mathur, Akhil, Schelten, Alan, Vaughan, Alex, Yang, Amy, Fan, Angela, Goyal, Anirudh, Hartshorn, Anthony, Yang, Aobo, Mitra, Archi, Sravankumar, Archie, Korenev, Artem, Hinsvark, Arthur, Rao, Arun, Zhang, Aston, Rodriguez, Aurelien, Gregerson, Austen, Spataru, Ava, Roziere, Baptiste, Biron, Bethany, Tang, Binh, Chern, Bobbie, Caucheteux, Charlotte, Nayak, Chaya, Bi, Chloe, Marra, Chris, McConnell, Chris, Keller, Christian, Touret, Christophe, Wu, Chunyang, Wong, Corinne, Ferrer, Cristian Canton, Nikolaidis, Cyrus, Allonsius, Damien, Song, Daniel, Pintz, Danielle, Livshits, Danny, Wyatt, Danny, Esiobu, David, Choudhary, Dhruv, Mahajan, Dhruv, Garcia-Olano, Diego, Perino, Diego, Hupkes, Dieuwke, Lakomkin, Egor, AlBadawy, Ehab, Lobanova, Elina, Dinan, Emily, Smith, Eric Michael, Radenovic, Filip, Guzmán, Francisco, Zhang, Frank, Synnaeve, Gabriel, Lee, Gabrielle, Anderson, Georgia Lewis, Thattai, Govind, Nail, Graeme, Mialon, Gregoire, Pang, Guan, Cucurell, Guillem, Nguyen, Hailey, Korevaar, Hannah, Xu, Hu, Touvron, Hugo, Zarov, Iliyan, Ibarra, Imanol Arrieta, Kloumann, Isabel, Misra, Ishan, Evtimov, Ivan, Zhang, Jack, Copet, Jade, Lee, Jaewon, Geffert, Jan, Vranes, Jana, Park, Jason, Mahadeokar, Jay, Shah, Jeet, van der Linde, Jelmer, Billock, Jennifer, Hong, Jenny, Lee, Jenya, Fu, Jeremy, Chi, Jianfeng, Huang, Jianyu, Liu, Jiawen, Wang, Jie, Yu, Jiecao, Bitton, Joanna, Spisak, Joe, Park, Jongsoo, Rocca, Joseph, Johnstun, Joshua, Saxe, Joshua, Jia, Junteng, Alwala, Kalyan Vasuden, Prasad, Karthik, Upasani, Kartikeya, Plawiak, Kate, Li, Ke, Heafield, Kenneth, Stone, Kevin, El-Arini, Khalid, Iyer, Krithika, Malik, Kshitiz, Chiu, Kuenley, Bhalla, Kunal, Lakhotia, Kushal, Rantala-Yeary, Lauren, van der Maaten, Laurens, Chen, Lawrence, Tan, Liang, Jenkins, Liz, Martin, Louis, Madaan, Lovish, Malo, Lubo, Blecher, Lukas, Landzaat, Lukas, de Oliveira, Luke, Muzzi, Madeline, Pasupuleti, Mahesh, Singh, Mannat, Paluri, Manohar, Kardas, Marcin, Tsimpoukelli, Maria, Oldham, Mathew, Rita, Mathieu, Pavlova, Maya, Kambadur, Melanie, Lewis, Mike, Si, Min, Singh, Mitesh Kumar, Hassan, Mona, Goyal, Naman, Torabi, Narjes, Bashlykov, Nikolay, Bogoychev, Nikolay, Chatterji, Niladri, Zhang, Ning, Duchenne, Olivier, Çelebi, Onur, Alrassy, Patrick, Zhang, Pengchuan, Li, Pengwei, Vasic, Petar, Weng, Peter, Bhargava, Prajjwal, Dubal, Pratik, Krishnan, Praveen, Koura, Punit Singh, Xu, Puxin, He, Qing, Dong, Qingxiao, Srinivasan, Ragavan, Ganapathy, Raj, Calderer, Ramon, Cabral, Ricardo Silveira, Stojnic, Robert, Raileanu, Roberta, Maheswari, Rohan, Girdhar, Rohit, Patel, Rohit, Sauvestre, Romain, Polidoro, Ronnie, Sumbaly, Roshan, Taylor, Ross, Silva, Ruan, Hou, Rui, Wang, Rui, Hosseini, Saghar, Chennabasappa, Sahana, Singh, Sanjay, Bell, Sean, Kim, Seohyun Sonia, Edunov, Sergey, Nie, Shaoliang, Narang, Sharan, Raparthy, Sharath, Shen, Sheng, Wan, Shengye, Bhosale, Shruti, Zhang, Shun, Vandenhende, Simon, Batra, Soumya, Whitman, Spencer, Sootla, Sten, Collot, Stephane, Gururangan, Suchin, Borodinsky, Sydney, Herman, Tamar, Fowler, Tara, Sheasha, Tarek, Georgiou, Thomas, Scialom, Thomas, Speckbacher, Tobias, Mihaylov, Todor, Xiao, Tong, Karn, Ujjwal, Goswami, Vedanuj, Gupta, Vibhor, Ramanathan, Vignesh, Kerkez, Viktor, Gonguet, Vincent, Do, Virginie, Vogeti, Vish, Albiero, Vítor, Petrovic, Vladan, Chu, Weiwei, Xiong, Wenhan, Fu, Wenyin, Meers, Whitney, Martinet, Xavier, Wang, Xiaodong, Wang, Xiaofang, Tan, Xiaoqing Ellen, Xia, Xide, Xie, Xinfeng, Jia, Xuchao, Wang, Xuewei, Goldschlag, Yaelle, Gaur, Yashesh, Babaei, Yasmine, Wen, Yi, Song, Yiwen, Zhang, Yuchen, Li, Yue, Mao, Yuning, Coudert, Zacharie Delpierre, Yan, Zheng, Chen, Zhengxing, Papakipos, Zoe, Singh, Aaditya, Srivastava, Aayushi, Jain, Abha, Kelsey, Adam, Shajnfeld, Adam, Gangidi, Adithya, Victoria, Adolfo, Goldstand, Ahuva, Menon, Ajay, Sharma, Ajay, Boesenberg, Alex, Baevski, Alexei, Feinstein, Allie, Kallet, Amanda, Sangani, Amit, Teo, Amos, Yunus, Anam, Lupu, Andrei, Alvarado, Andres, Caples, Andrew, Gu, Andrew, Ho, Andrew, Poulton, Andrew, Ryan, Andrew, Ramchandani, Ankit, Dong, Annie, Franco, Annie, Goyal, Anuj, Saraf, Aparajita, Chowdhury, Arkabandhu, Gabriel, Ashley, Bharambe, Ashwin, Eisenman, Assaf, Yazdan, Azadeh, James, Beau, Maurer, Ben, Leonhardi, Benjamin, Huang, Bernie, Loyd, Beth, De Paola, Beto, Paranjape, Bhargavi, Liu, Bing, Wu, Bo, Ni, Boyu, Hancock, Braden, Wasti, Bram, Spence, Brandon, Stojkovic, Brani, Gamido, Brian, Montalvo, Britt, Parker, Carl, Burton, Carly, Mejia, Catalina, Liu, Ce, Wang, Changhan, Kim, Changkyu, Zhou, Chao, Hu, Chester, Chu, Ching-Hsiang, Cai, Chris, Tindal, Chris, Feichtenhofer, Christoph, Gao, Cynthia, Civin, Damon, Beaty, Dana, Kreymer, Daniel, Li, Daniel, Adkins, David, Xu, David, Testuggine, Davide, David, Delia, Parikh, Devi, Liskovich, Diana, Foss, Didem, Wang, Dingkang, Le, Duc, Holland, Dustin, Dowling, Edward, Jamil, Eissa, Montgomery, Elaine, Presani, Eleonora, Hahn, Emily, Wood, Emily, Le, Eric-Tuan, Brinkman, Erik, Arcaute, Esteban, Dunbar, Evan, Smothers, Evan, Sun, Fei, Kreuk, Felix, Tian, Feng, Kokkinos, Filippos, Ozgenel, Firat, Caggioni, Francesco, Kanayet, Frank, Seide, Frank, Florez, Gabriela Medina, Schwarz, Gabriella, Badeer, Gada, Swee, Georgia, Halpern, Gil, Herman, Grant, Sizov, Grigory, Guangyi, Zhang, Lakshminarayanan, Guna, Inan, Hakan, Shojanazeri, Hamid, Zou, Han, Wang, Hannah, Zha, Hanwen, Habeeb, Haroun, Rudolph, Harrison, Suk, Helen, Aspegren, Henry, Goldman, Hunter, Zhan, Hongyuan, Damlaj, Ibrahim, Molybog, Igor, Tufanov, Igor, Leontiadis, Ilias, Veliche, Irina-Elena, Gat, Itai, Weissman, Jake, Geboski, James, Kohli, James, Lam, Janice, Asher, Japhet, Gaya, Jean-Baptiste, Marcus, Jeff, Tang, Jeff, Chan, Jennifer, Zhen, Jenny, Reizenstein, Jeremy, Teboul, Jeremy, Zhong, Jessica, Jin, Jian, Yang, Jingyi, Cummings, Joe, Carvill, Jon, Shepard, Jon, McPhie, Jonathan, Torres, Jonathan, Ginsburg, Josh, Wang, Junjie, Wu, Kai, U, Kam Hou, Saxena, Karan, Khandelwal, Kartikay, Zand, Katayoun, Matosich, Kathy, Veeraraghavan, Kaushik, Michelena, Kelly, Li, Keqian, Jagadeesh, Kiran, Huang, Kun, Chawla, Kunal, Huang, Kyle, Chen, Lailin, Garg, Lakshya, A, Lavender, Silva, Leandro, Bell, Lee, Zhang, Lei, Guo, Liangpeng, Yu, Licheng, Moshkovich, Liron, Wehrstedt, Luca, Khabsa, Madian, Avalani, Manav, Bhatt, Manish, Mankus, Martynas, Hasson, Matan, Lennie, Matthew, Reso, Matthias, Groshev, Maxim, Naumov, Maxim, Lathi, Maya, Keneally, Meghan, Liu, Miao, Seltzer, Michael L., Valko, Michal, Restrepo, Michelle, Patel, Mihir, Vyatskov, Mik, Samvelyan, Mikayel, Clark, Mike, Macey, Mike, Wang, Mike, Hermoso, Miquel Jubert, Metanat, Mo, Rastegari, Mohammad, Bansal, Munish, Santhanam, Nandhini, Parks, Natascha, White, Natasha, Bawa, Navyata, Singhal, Nayan, Egebo, Nick, Usunier, Nicolas, Mehta, Nikhil, Laptev, Nikolay Pavlovich, Dong, Ning, Cheng, Norman, Chernoguz, Oleg, Hart, Olivia, Salpekar, Omkar, Kalinli, Ozlem, Kent, Parkin, Parekh, Parth, Saab, Paul, Balaji, Pavan, Rittner, Pedro, Bontrager, Philip, Roux, Pierre, Dollar, Piotr, Zvyagina, Polina, Ratanchandani, Prashant, Yuvraj, Pritish, Liang, Qian, Alao, Rachad, Rodriguez, Rachel, Ayub, Rafi, Murthy, Raghotham, Nayani, Raghu, Mitra, Rahul, Parthasarathy, Rangaprabhu, Li, Raymond, Hogan, Rebekkah, Battey, Robin, Wang, Rocky, Howes, Russ, Rinott, Ruty, Mehta, Sachin, Siby, Sachin, Bondu, Sai Jayesh, Datta, Samyak, Chugh, Sara, Hunt, Sara, Dhillon, Sargun, Sidorov, Sasha, Pan, Satadru, Mahajan, Saurabh, Verma, Saurabh, Yamamoto, Seiji, Ramaswamy, Sharadh, Lindsay, Shaun, Lindsay, Shaun, Feng, Sheng, Lin, Shenghao, Zha, Shengxin Cindy, Patil, Shishir, Shankar, Shiva, Zhang, Shuqiang, Zhang, Shuqiang, Wang, Sinong, Agarwal, Sneha, Sajuyigbe, Soji, Chintala, Soumith, Max, Stephanie, Chen, Stephen, Kehoe, Steve, Satterfield, Steve, Govindaprasad, Sudarshan, Gupta, Sumit, Deng, Summer, Cho, Sungmin, Virk, Sunny, Subramanian, Suraj, Choudhury, Sy, Goldman, Sydney, Remez, Tal, Glaser, Tamar, Best, Tamara, Koehler, Thilo, Robinson, Thomas, Li, Tianhe, Zhang, Tianjun, Matthews, Tim, Chou, Timothy, Shaked, Tzook, Vontimitta, Varun, Ajayi, Victoria, Montanez, Victoria, Mohan, Vijai, Kumar, Vinay Satish, Mangla, Vishal, Ionescu, Vlad, Poenaru, Vlad, Mihailescu, Vlad Tiberiu, Ivanov, Vladimir, Li, Wei, Wang, Wenchen, Jiang, Wenwen, Bouaziz, Wes, Constable, Will, Tang, Xiaocheng, Wu, Xiaojian, Wang, Xiaolan, Wu, Xilun, Gao, Xinbo, Kleinman, Yaniv, Chen, Yanjun, Hu, Ye, Jia, Ye, Qi, Ye, Li, Yenda, Zhang, Yilin, Zhang, Ying, Adi, Yossi, Nam, Youngjin, Yu, Wang, Zhao, Yu, Hao, Yuchen, Qian, Yundi, Li, Yunlu, He, Yuzi, Rait, Zach, DeVito, Zachary, Rosnbrick, Zef, Wen, Zhaoduo, Yang, Zhenyu, Zhao, Zhiwei, Ma, Zhiyu

Human-centric standardized exams (SAT, LSAT, Gaokao, GRE, GMAT, Civil Service, Lawyer Qualification).

1. AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models — published result table ↗

   Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang et al. · 2023

   All 9 authors

   Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, Nan Duan

Logical reading comprehension benchmarks sourced from Civil Service and GMAT/LSAT critical reasoning exams.

1. LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning — published result table ↗

   Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, Yue Zhang · 2020

2. ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning ↗

   Weihao Yu, Zihang Jiang, Yanfei Dong, Jiashi Feng · 2020

The four imported columns are CommonsenseQA, PIQA, Social IQa and WinoGrande; none is HellaSwag. Route each column to its actual benchmark, creating the two missing task pages.

CommonsenseQA → Social IQa → PIQA → WinoGrande →

1. The Llama 3 Herd of Models — published result table ↗

   Grattafiori, Aaron, Dubey, Abhimanyu, Jauhri, Abhinav, Pandey, Abhinav, Kadian, Abhishek, Al-Dahle, Ahmad et al. · 2024

   All 561 authors

   Grattafiori, Aaron, Dubey, Abhimanyu, Jauhri, Abhinav, Pandey, Abhinav, Kadian, Abhishek, Al-Dahle, Ahmad, Letman, Aiesha, Mathur, Akhil, Schelten, Alan, Vaughan, Alex, Yang, Amy, Fan, Angela, Goyal, Anirudh, Hartshorn, Anthony, Yang, Aobo, Mitra, Archi, Sravankumar, Archie, Korenev, Artem, Hinsvark, Arthur, Rao, Arun, Zhang, Aston, Rodriguez, Aurelien, Gregerson, Austen, Spataru, Ava, Roziere, Baptiste, Biron, Bethany, Tang, Binh, Chern, Bobbie, Caucheteux, Charlotte, Nayak, Chaya, Bi, Chloe, Marra, Chris, McConnell, Chris, Keller, Christian, Touret, Christophe, Wu, Chunyang, Wong, Corinne, Ferrer, Cristian Canton, Nikolaidis, Cyrus, Allonsius, Damien, Song, Daniel, Pintz, Danielle, Livshits, Danny, Wyatt, Danny, Esiobu, David, Choudhary, Dhruv, Mahajan, Dhruv, Garcia-Olano, Diego, Perino, Diego, Hupkes, Dieuwke, Lakomkin, Egor, AlBadawy, Ehab, Lobanova, Elina, Dinan, Emily, Smith, Eric Michael, Radenovic, Filip, Guzmán, Francisco, Zhang, Frank, Synnaeve, Gabriel, Lee, Gabrielle, Anderson, Georgia Lewis, Thattai, Govind, Nail, Graeme, Mialon, Gregoire, Pang, Guan, Cucurell, Guillem, Nguyen, Hailey, Korevaar, Hannah, Xu, Hu, Touvron, Hugo, Zarov, Iliyan, Ibarra, Imanol Arrieta, Kloumann, Isabel, Misra, Ishan, Evtimov, Ivan, Zhang, Jack, Copet, Jade, Lee, Jaewon, Geffert, Jan, Vranes, Jana, Park, Jason, Mahadeokar, Jay, Shah, Jeet, van der Linde, Jelmer, Billock, Jennifer, Hong, Jenny, Lee, Jenya, Fu, Jeremy, Chi, Jianfeng, Huang, Jianyu, Liu, Jiawen, Wang, Jie, Yu, Jiecao, Bitton, Joanna, Spisak, Joe, Park, Jongsoo, Rocca, Joseph, Johnstun, Joshua, Saxe, Joshua, Jia, Junteng, Alwala, Kalyan Vasuden, Prasad, Karthik, Upasani, Kartikeya, Plawiak, Kate, Li, Ke, Heafield, Kenneth, Stone, Kevin, El-Arini, Khalid, Iyer, Krithika, Malik, Kshitiz, Chiu, Kuenley, Bhalla, Kunal, Lakhotia, Kushal, Rantala-Yeary, Lauren, van der Maaten, Laurens, Chen, Lawrence, Tan, Liang, Jenkins, Liz, Martin, Louis, Madaan, Lovish, Malo, Lubo, Blecher, Lukas, Landzaat, Lukas, de Oliveira, Luke, Muzzi, Madeline, Pasupuleti, Mahesh, Singh, Mannat, Paluri, Manohar, Kardas, Marcin, Tsimpoukelli, Maria, Oldham, Mathew, Rita, Mathieu, Pavlova, Maya, Kambadur, Melanie, Lewis, Mike, Si, Min, Singh, Mitesh Kumar, Hassan, Mona, Goyal, Naman, Torabi, Narjes, Bashlykov, Nikolay, Bogoychev, Nikolay, Chatterji, Niladri, Zhang, Ning, Duchenne, Olivier, Çelebi, Onur, Alrassy, Patrick, Zhang, Pengchuan, Li, Pengwei, Vasic, Petar, Weng, Peter, Bhargava, Prajjwal, Dubal, Pratik, Krishnan, Praveen, Koura, Punit Singh, Xu, Puxin, He, Qing, Dong, Qingxiao, Srinivasan, Ragavan, Ganapathy, Raj, Calderer, Ramon, Cabral, Ricardo Silveira, Stojnic, Robert, Raileanu, Roberta, Maheswari, Rohan, Girdhar, Rohit, Patel, Rohit, Sauvestre, Romain, Polidoro, Ronnie, Sumbaly, Roshan, Taylor, Ross, Silva, Ruan, Hou, Rui, Wang, Rui, Hosseini, Saghar, Chennabasappa, Sahana, Singh, Sanjay, Bell, Sean, Kim, Seohyun Sonia, Edunov, Sergey, Nie, Shaoliang, Narang, Sharan, Raparthy, Sharath, Shen, Sheng, Wan, Shengye, Bhosale, Shruti, Zhang, Shun, Vandenhende, Simon, Batra, Soumya, Whitman, Spencer, Sootla, Sten, Collot, Stephane, Gururangan, Suchin, Borodinsky, Sydney, Herman, Tamar, Fowler, Tara, Sheasha, Tarek, Georgiou, Thomas, Scialom, Thomas, Speckbacher, Tobias, Mihaylov, Todor, Xiao, Tong, Karn, Ujjwal, Goswami, Vedanuj, Gupta, Vibhor, Ramanathan, Vignesh, Kerkez, Viktor, Gonguet, Vincent, Do, Virginie, Vogeti, Vish, Albiero, Vítor, Petrovic, Vladan, Chu, Weiwei, Xiong, Wenhan, Fu, Wenyin, Meers, Whitney, Martinet, Xavier, Wang, Xiaodong, Wang, Xiaofang, Tan, Xiaoqing Ellen, Xia, Xide, Xie, Xinfeng, Jia, Xuchao, Wang, Xuewei, Goldschlag, Yaelle, Gaur, Yashesh, Babaei, Yasmine, Wen, Yi, Song, Yiwen, Zhang, Yuchen, Li, Yue, Mao, Yuning, Coudert, Zacharie Delpierre, Yan, Zheng, Chen, Zhengxing, Papakipos, Zoe, Singh, Aaditya, Srivastava, Aayushi, Jain, Abha, Kelsey, Adam, Shajnfeld, Adam, Gangidi, Adithya, Victoria, Adolfo, Goldstand, Ahuva, Menon, Ajay, Sharma, Ajay, Boesenberg, Alex, Baevski, Alexei, Feinstein, Allie, Kallet, Amanda, Sangani, Amit, Teo, Amos, Yunus, Anam, Lupu, Andrei, Alvarado, Andres, Caples, Andrew, Gu, Andrew, Ho, Andrew, Poulton, Andrew, Ryan, Andrew, Ramchandani, Ankit, Dong, Annie, Franco, Annie, Goyal, Anuj, Saraf, Aparajita, Chowdhury, Arkabandhu, Gabriel, Ashley, Bharambe, Ashwin, Eisenman, Assaf, Yazdan, Azadeh, James, Beau, Maurer, Ben, Leonhardi, Benjamin, Huang, Bernie, Loyd, Beth, De Paola, Beto, Paranjape, Bhargavi, Liu, Bing, Wu, Bo, Ni, Boyu, Hancock, Braden, Wasti, Bram, Spence, Brandon, Stojkovic, Brani, Gamido, Brian, Montalvo, Britt, Parker, Carl, Burton, Carly, Mejia, Catalina, Liu, Ce, Wang, Changhan, Kim, Changkyu, Zhou, Chao, Hu, Chester, Chu, Ching-Hsiang, Cai, Chris, Tindal, Chris, Feichtenhofer, Christoph, Gao, Cynthia, Civin, Damon, Beaty, Dana, Kreymer, Daniel, Li, Daniel, Adkins, David, Xu, David, Testuggine, Davide, David, Delia, Parikh, Devi, Liskovich, Diana, Foss, Didem, Wang, Dingkang, Le, Duc, Holland, Dustin, Dowling, Edward, Jamil, Eissa, Montgomery, Elaine, Presani, Eleonora, Hahn, Emily, Wood, Emily, Le, Eric-Tuan, Brinkman, Erik, Arcaute, Esteban, Dunbar, Evan, Smothers, Evan, Sun, Fei, Kreuk, Felix, Tian, Feng, Kokkinos, Filippos, Ozgenel, Firat, Caggioni, Francesco, Kanayet, Frank, Seide, Frank, Florez, Gabriela Medina, Schwarz, Gabriella, Badeer, Gada, Swee, Georgia, Halpern, Gil, Herman, Grant, Sizov, Grigory, Guangyi, Zhang, Lakshminarayanan, Guna, Inan, Hakan, Shojanazeri, Hamid, Zou, Han, Wang, Hannah, Zha, Hanwen, Habeeb, Haroun, Rudolph, Harrison, Suk, Helen, Aspegren, Henry, Goldman, Hunter, Zhan, Hongyuan, Damlaj, Ibrahim, Molybog, Igor, Tufanov, Igor, Leontiadis, Ilias, Veliche, Irina-Elena, Gat, Itai, Weissman, Jake, Geboski, James, Kohli, James, Lam, Janice, Asher, Japhet, Gaya, Jean-Baptiste, Marcus, Jeff, Tang, Jeff, Chan, Jennifer, Zhen, Jenny, Reizenstein, Jeremy, Teboul, Jeremy, Zhong, Jessica, Jin, Jian, Yang, Jingyi, Cummings, Joe, Carvill, Jon, Shepard, Jon, McPhie, Jonathan, Torres, Jonathan, Ginsburg, Josh, Wang, Junjie, Wu, Kai, U, Kam Hou, Saxena, Karan, Khandelwal, Kartikay, Zand, Katayoun, Matosich, Kathy, Veeraraghavan, Kaushik, Michelena, Kelly, Li, Keqian, Jagadeesh, Kiran, Huang, Kun, Chawla, Kunal, Huang, Kyle, Chen, Lailin, Garg, Lakshya, A, Lavender, Silva, Leandro, Bell, Lee, Zhang, Lei, Guo, Liangpeng, Yu, Licheng, Moshkovich, Liron, Wehrstedt, Luca, Khabsa, Madian, Avalani, Manav, Bhatt, Manish, Mankus, Martynas, Hasson, Matan, Lennie, Matthew, Reso, Matthias, Groshev, Maxim, Naumov, Maxim, Lathi, Maya, Keneally, Meghan, Liu, Miao, Seltzer, Michael L., Valko, Michal, Restrepo, Michelle, Patel, Mihir, Vyatskov, Mik, Samvelyan, Mikayel, Clark, Mike, Macey, Mike, Wang, Mike, Hermoso, Miquel Jubert, Metanat, Mo, Rastegari, Mohammad, Bansal, Munish, Santhanam, Nandhini, Parks, Natascha, White, Natasha, Bawa, Navyata, Singhal, Nayan, Egebo, Nick, Usunier, Nicolas, Mehta, Nikhil, Laptev, Nikolay Pavlovich, Dong, Ning, Cheng, Norman, Chernoguz, Oleg, Hart, Olivia, Salpekar, Omkar, Kalinli, Ozlem, Kent, Parkin, Parekh, Parth, Saab, Paul, Balaji, Pavan, Rittner, Pedro, Bontrager, Philip, Roux, Pierre, Dollar, Piotr, Zvyagina, Polina, Ratanchandani, Prashant, Yuvraj, Pritish, Liang, Qian, Alao, Rachad, Rodriguez, Rachel, Ayub, Rafi, Murthy, Raghotham, Nayani, Raghu, Mitra, Rahul, Parthasarathy, Rangaprabhu, Li, Raymond, Hogan, Rebekkah, Battey, Robin, Wang, Rocky, Howes, Russ, Rinott, Ruty, Mehta, Sachin, Siby, Sachin, Bondu, Sai Jayesh, Datta, Samyak, Chugh, Sara, Hunt, Sara, Dhillon, Sargun, Sidorov, Sasha, Pan, Satadru, Mahajan, Saurabh, Verma, Saurabh, Yamamoto, Seiji, Ramaswamy, Sharadh, Lindsay, Shaun, Lindsay, Shaun, Feng, Sheng, Lin, Shenghao, Zha, Shengxin Cindy, Patil, Shishir, Shankar, Shiva, Zhang, Shuqiang, Zhang, Shuqiang, Wang, Sinong, Agarwal, Sneha, Sajuyigbe, Soji, Chintala, Soumith, Max, Stephanie, Chen, Stephen, Kehoe, Steve, Satterfield, Steve, Govindaprasad, Sudarshan, Gupta, Sumit, Deng, Summer, Cho, Sungmin, Virk, Sunny, Subramanian, Suraj, Choudhury, Sy, Goldman, Sydney, Remez, Tal, Glaser, Tamar, Best, Tamara, Koehler, Thilo, Robinson, Thomas, Li, Tianhe, Zhang, Tianjun, Matthews, Tim, Chou, Timothy, Shaked, Tzook, Vontimitta, Varun, Ajayi, Victoria, Montanez, Victoria, Mohan, Vijai, Kumar, Vinay Satish, Mangla, Vishal, Ionescu, Vlad, Poenaru, Vlad, Mihailescu, Vlad Tiberiu, Ivanov, Vladimir, Li, Wei, Wang, Wenchen, Jiang, Wenwen, Bouaziz, Wes, Constable, Will, Tang, Xiaocheng, Wu, Xiaojian, Wang, Xiaolan, Wu, Xilun, Gao, Xinbo, Kleinman, Yaniv, Chen, Yanjun, Hu, Ye, Jia, Ye, Qi, Ye, Li, Yenda, Zhang, Yilin, Zhang, Ying, Adi, Yossi, Nam, Youngjin, Yu, Wang, Zhao, Yu, Hao, Yuchen, Qian, Yundi, Li, Yunlu, He, Yuzi, Rait, Zach, DeVito, Zachary, Rosnbrick, Zef, Wen, Zhaoduo, Yang, Zhenyu, Zhao, Zhiwei, Ma, Zhiyu

Dynamic benchmark bridging open-ended user queries and ground-truth multiple-choice/short-answer benchmarks (0.96 correlation with Chatbot Arena).

1. MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures — published result table ↗

   Jinjie Ni, Fuzhao Xue, Xiang Yue, Yuntian Deng, Mahir Shah, Kabir Jain et al. · 2024

   All 8 authors

   Jinjie Ni, Fuzhao Xue, Xiang Yue, Yuntian Deng, Mahir Shah, Kabir Jain, Graham Neubig, Yang You

A long input describes a graph, and the model must retrieve related nodes or follow graph structure to the requested depth.

1. Primary benchmark documentation or model evaluation report ↗
2. Epoch AI · benchmark and methodology ↗
3. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

Enterprise information tasks test retrieval and reasoning over business content in a model-launch evaluation.

1. Introducing ChatGPT for financial services ↗

   OpenAI · 2026

Detailed visual questions test the ability to locate and interpret relevant evidence inside an image.

1. moonshotai/Kimi-K2.6 · Hugging Face ↗

   Moonshot AI

A language model with visual input must understand and reason over long video material.

1. Primary benchmark documentation or model evaluation report ↗

Video understanding is evaluated across short, medium and long clips through human-annotated questions.

1. Primary benchmark documentation or model evaluation report ↗
2. Video-MME ↗

   Epoch AI

3. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

Factual reliability across 3,000 questions, with credit for correct answers, penalties for errors and neutral abstentions.

1. AA-Omniscience v1.1: Knowledge and Hallucination Benchmark — Artificial Analysis ↗

   Artificial Analysis

2. Artificial Analysis Intelligence Benchmarking Methodology — v5.0 ↗

   Artificial Analysis

Short factual questions probe whether a model supplies the right answer and when it declines to answer.

1. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning — published result table ↗

   DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang et al. · 2025

   All 200 authors

   DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, Zhen Zhang

2. Simple Evals — model benchmark results ↗

   OpenAI · 2025

3. Primary benchmark documentation or model evaluation report ↗

Audited subset of SimpleQA removing ambiguous dates, contested facts, and noisy reference answers.

1. Publisher directory ↗
2. SimpleQA Verified ↗

   Epoch AI

3. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

3,000+ fact-seeking questions covering Chinese history, science, culture, law, and contemporary society.

1. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning — published result table ↗

   DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang et al. · 2025

   All 200 authors

   DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, Zhen Zhang

1,719 examples evaluating whether long-form LLM responses remain strictly grounded in provided source documents without extrinsic hallucination.

1. The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Input — published result table ↗

   Alon Jacovi, Andrew Wang, Chris Alberti, Connie Tao, Jon Lipovetz, Kate Olszewska et al. · 2025

   All 26 authors

   Alon Jacovi, Andrew Wang, Chris Alberti, Connie Tao, Jon Lipovetz, Kate Olszewska, Lukas Haas, Michelle Liu, Nate Keating, Adam Bloniarz, Carl Saroufim, Corey Fry, Dror Marcus, Doron Kukliansky, Gaurav Singh Tomar, James Swirhun, Jinwei Xing, Lily Wang, Madhu Gurumurthy, Michael Aaron, Moran Ambar, Rachana Fellinger, Rui Wang, Zizhao Zhang, Sasha Goldshtein, Dipanjan Das

Decomposes long-form biographies/articles into atomic factual claims and verifies each claim against Wikipedia/retrieval sources.

1. FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation — published result table ↗

   Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh et al. · 2023

   All 9 authors

   Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, Hannaneh Hajishirzi

Taxonomy-driven benchmark separating extrinsic factual hallucination (entities, citations, nonexistent concepts) from intrinsic context contradiction.

1. HalluLens: LLM Hallucination Benchmark — published result table ↗

   Yejin Bang, Ziwei Ji, Alan Schelten, Anthony Hartshorn, Tara Fowler, Cheng Zhang et al. · 2025

   All 8 authors

   Yejin Bang, Ziwei Ji, Alan Schelten, Anthony Hartshorn, Tara Fowler, Cheng Zhang, Nicola Cancedda, Pascale Fung

Challenging practical-work prompts measure unsupported factual claims in Kimi’s internal evaluation.

1. Introducing Kimi K3.1 Pro ↗

   Moonshot AI · 2026

Questions target common misconceptions to test whether a model repeats familiar falsehoods.

1. TruthfulQA: Measuring How Models Mimic Human Falsehoods — published result table ↗

   Lin, Stephanie, Hilton, Jacob, Evans, Owain · 2021

A broad academic test measures knowledge and reasoning across many subject areas.

1. Primary benchmark documentation or model evaluation report ↗
2. Epoch AI · benchmark and methodology ↗
3. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

A more demanding successor to MMLU emphasizes reasoning across a broad academic subject mix.

1. MMLU Pro — evaluation methodology and results ↗

   Vals AI · 2026

2. MMLU-Pro Benchmark | Artificial Analysis ↗

   Artificial Analysis

5,700 manually re-annotated MMLU questions fixing the ~6.5% ground-truth label errors present in original MMLU.

1. Are We Done with MMLU? — published result table ↗

   Gema, Aryo Pradipta, Leang, Joshua Ong Jun, Hong, Giwon, Devoto, Alessio, Mancino, Alberto Carlo Maria, Saxena, Rohit et al. · 2024

   All 16 authors

   Gema, Aryo Pradipta, Leang, Joshua Ong Jun, Hong, Giwon, Devoto, Alessio, Mancino, Alberto Carlo Maria, Saxena, Rohit, He, Xuanli, Zhao, Yu, Du, Xiaotang, Madani, Mohammad Reza Ghasemi, Barale, Claire, McHardy, Robert, Harris, Joshua, Kaddour, Jean, van Krieken, Emile, Minervini, Pasquale

A multilingual knowledge evaluation adds language and cultural coverage to academic testing.

1. Global-MMLU-Lite Benchmark | Artificial Analysis ↗

   Artificial Analysis

Professional human translation of the entire MMLU test set into 14 languages (Arabic, Bengali, Chinese, French, German, Hindi, Indonesian, Italian, Japanese, Korean, Portuguese, Spanish, Swahili, Yoruba).

1. Qwen3 Technical Report — published result table ↗

   An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng et al. · 2025

   All 60 authors

   An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, Zihan Qiu

197,243 regional academic, professional, and licensing exam questions collected natively in 44 languages (not translated from English).

1. INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge — published result table ↗

   Angelika Romanou, Negar Foroutan, Anna Sotnikova, Zeming Chen, Sree Harsha Nelaturu, Shivalika Singh et al. · 2024

   All 59 authors

   Angelika Romanou, Negar Foroutan, Anna Sotnikova, Zeming Chen, Sree Harsha Nelaturu, Shivalika Singh, Rishabh Maheshwary, Micol Altomare, Mohamed A. Haggag, Snegha A, Alfonso Amayuelas, Azril Hafizi Amirudin, Viraat Aryabumi, Danylo Boiko, Michael Chang, Jenny Chim, Gal Cohen, Aditya Kumar Dalmia, Abraham Diress, Sharad Duwal, Daniil Dzenhaliou, Daniel Fernando Erazo Florez, Fabian Farestam, Joseph Marvin Imperial, Shayekh Bin Islam, Perttu Isotalo, Maral Jabbarishiviari, Börje F. Karlsson, Eldar Khalilov, Christopher Klamm, Fajri Koto, Dominik Krzemiński, Gabriel Adriano de Melo, Syrielle Montariol, Yiyang Nan, Joel Niklaus, Jekaterina Novikova, Johan Samir Obando Ceron, Debjit Paul, Esther Ploeger, Jebish Purbey, Swati Rajwal, Selvan Sunitha Ravi, Sara Rydell, Roshan Santhosh, Drishti Sharma, Marjana Prifti Skenduli, Arshia Soltani Moakhar, Bardia Soltani Moakhar, Ran Tamir, Ayush Kumar Tarun, Azmine Toushik Wasi, Thenuka Ovin Weerasinghe, Serhan Yilmaz, Mike Zhang, Imanol Schlag, Marzieh Fadaee, Sara Hooker, Antoine Bosselut

Chinese-language academic and professional questions broaden knowledge evaluation beyond English.

1. CMMLU: Measuring massive multitask language understanding in Chinese — published result table ↗

   Li, Haonan, Zhang, Yixuan, Koto, Fajri, Yang, Yifei, Zhao, Hai, Gong, Yeyun et al. · 2023

   All 8 authors

   Li, Haonan, Zhang, Yixuan, Koto, Fajri, Yang, Yifei, Zhao, Hai, Gong, Yeyun, Duan, Nan, Baldwin, Timothy

2. Primary benchmark documentation or model evaluation report ↗

13,948 Chinese exam questions across 52 disciplines and 4 educational levels (Middle School, High School, College, Professional).

1. C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models — published result table ↗

   Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su et al. · 2023

   All 13 authors

   Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, Junxian He

Native Korean (35k questions), Japanese, Arabic (40 tasks), and Indonesian multi-discipline knowledge benchmarks.

1. ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic — published result table ↗

   Koto, Fajri, Li, Haonan, Shatnawi, Sara, Doughman, Jad, Sadallah, Abdelrahman Boda, Alraeesi, Aisha et al. · 2024

   All 13 authors

   Koto, Fajri, Li, Haonan, Shatnawi, Sara, Doughman, Jad, Sadallah, Abdelrahman Boda, Alraeesi, Aisha, Almubarak, Khalid, Alyafeai, Zaid, Sengupta, Neha, Shehata, Shady, Habash, Nizar, Nakov, Preslav, Baldwin, Timothy

Parallel multiple-choice reading comprehension benchmark spanning 122 language variants linked to FLORES-200.

1. The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants — published result table ↗

   Bandarkar, Lucas, Liang, Davis, Muller, Benjamin, Artetxe, Mikel, Shukla, Satya Narayan, Husa, Donald et al. · 2023

   All 10 authors

   Bandarkar, Lucas, Liang, Davis, Muller, Benjamin, Artetxe, Mikel, Shukla, Satya Narayan, Husa, Donald, Goyal, Naman, Krishnan, Abhinandan, Zettlemoyer, Luke, Khabsa, Madian

Multilingual translation is evaluated across languages and translation directions. Read automatic metric scores alongside the exact language coverage and text or speech input route.

1. SeamlessM4T: Massively Multilingual & Multimodal Machine Translation — published result table ↗

   Communication, Seamless, Barrault, Loïc, Chung, Yu-An, Meglioli, Mariano Cora, Dale, David, Dong, Ning et al. · 2023

   All 68 authors

   Communication, Seamless, Barrault, Loïc, Chung, Yu-An, Meglioli, Mariano Cora, Dale, David, Dong, Ning, Duquenne, Paul-Ambroise, Elsahar, Hady, Gong, Hongyu, Heffernan, Kevin, Hoffman, John, Klaiber, Christopher, Li, Pengwei, Licht, Daniel, Maillard, Jean, Rakotoarison, Alice, Sadagopan, Kaushik Ram, Wenzek, Guillaume, Ye, Ethan, Akula, Bapi, Chen, Peng-Jen, Hachem, Naji El, Ellis, Brian, Gonzalez, Gabriel Mejia, Haaheim, Justin, Hansanti, Prangthip, Howes, Russ, Huang, Bernie, Hwang, Min-Jae, Inaguma, Hirofumi, Jain, Somya, Kalbassi, Elahe, Kallet, Amanda, Kulikov, Ilia, Lam, Janice, Li, Daniel, Ma, Xutai, Mavlyutov, Ruslan, Peloquin, Benjamin, Ramadan, Mohamed, Ramakrishnan, Abinesh, Sun, Anna, Tran, Kevin, Tran, Tuan, Tufanov, Igor, Vogeti, Vish, Wood, Carleigh, Yang, Yilin, Yu, Bokai, Andrews, Pierre, Balioglu, Can, Costa-jussà, Marta R., Celebi, Onur, Elbayad, Maha, Gao, Cynthia, Guzmán, Francisco, Kao, Justine, Lee, Ann, Mourachko, Alexandre, Pino, Juan, Popuri, Sravya, Ropers, Christophe, Saleem, Safiyyah, Schwenk, Holger, Tomasello, Paden, Wang, Changhan, Wang, Jeff, Wang, Skyler

An open-domain question answering benchmark with challenging trivia questions paired with evidence documents.

1. Epoch AI · benchmark and methodology ↗
2. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

3. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension — published result table ↗

   Joshi, Mandar, Choi, Eunsol, Weld, Daniel S., Zettlemoyer, Luke · 2017

14k entity-centric factual questions stratified by monthly Wikipedia pageviews to measure long-tail knowledge retention.

1. When to Retrieve: Teaching LLMs to Utilize Information Retrieval Effectively — published result table ↗

   Labruna, Tiziano, Campos, Jon Ander, Azkune, Gorka · 2024

Dynamically updated QA benchmark testing fast-changing world facts and false-premise debunking.

1. FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation — published result table ↗

   Vu, Tu, Iyyer, Mohit, Wang, Xuezhi, Constant, Noah, Wei, Jerry, Wei, Jason et al. · 2023

   All 11 authors

   Vu, Tu, Iyyer, Mohit, Wang, Xuezhi, Constant, Noah, Wei, Jerry, Wei, Jason, Tar, Chris, Sung, Yun-Hsuan, Zhou, Denny, Le, Quoc, Luong, Thang

Multi-hop compositional factual question answering benchmarks.

1. Measuring and Narrowing the Compositionality Gap in Language Models — published result table ↗

   Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, Mike Lewis · 2022

Evaluates geographical socioeconomic statistics recall across every country (WorldBench) and cross-cultural norms (CulturalBench).

1. CulturalBench: A Robust, Diverse, and Challenging Cultural Benchmark by Human-AI CulturalTeaming — published result table ↗

   Yu Ying Chiu, Liwei Jiang, Bill Yuchen Lin, Chan Young Park, Shuyue Stella Li, Sahithya Ravi et al. · 2024

   All 11 authors

   Yu Ying Chiu, Liwei Jiang, Bill Yuchen Lin, Chan Young Park, Shuyue Stella Li, Sahithya Ravi, Mehar Bhatia, Maria Antoniak, Yulia Tsvetkov, Vered Shwartz, Yejin Choi

Questions paired with conflicting Wikipedia passages test whether a model presents the disagreement accurately. Human review and an automated judge distinguish complete answers, partial answers, and errors.

1. WikiContradict: A Benchmark for Evaluating LLMs on Real-World Knowledge Conflicts from Wikipedia — published result table ↗

   Hou, Yufang, Pascale, Alessandra, Carnerero-Cano, Javier, Tchrakian, Tigran, Marinescu, Radu, Daly, Elizabeth et al. · 2024

   All 8 authors

   Hou, Yufang, Pascale, Alessandra, Carnerero-Cano, Javier, Tchrakian, Tigran, Marinescu, Radu, Daly, Elizabeth, Padhi, Inkit, Sattigeri, Prasanna

Comprehensive South Asian and Indic language knowledge, translation, and QA benchmarks.

1. IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languages — published result table ↗

   Harman Singh, Nitish Gupta, Shikhar Bharadwaj, Dinesh Tewari, Partha Talukdar · 2024

Evaluates models on educational assessment, subject mastery, and student response grading across academic curricula.

1. SAGE — evaluation methodology and results ↗

   Vals AI · 2026

Evaluates multi-turn pedagogical scaffolding, hint generation, and misconception diagnosis in STEM tutoring.

1. Scale Labs — evaluation methodology and results ↗

   Scale Labs / SEAL · 2026

Evaluates whether AI models can accurately determine household SNAP eligibility and benefit amounts across complex state/federal rules.

1. Public Benefits Bench — evaluation methodology and results ↗

   Vals AI · 2026

2. Public Benefits Bench v1 — evaluation methodology and results ↗

   Vals AI · 2026

Large-scale benchmark for detecting and quantifying hallucinations across QA, dialogue, and summarization.

1. The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Models — published result table ↗

   Li, Junyi, Chen, Jie, Ren, Ruiyang, Cheng, Xiaoxue, Zhao, Wayne Xin, Nie, Jian-Yun et al. · 2024

   All 7 authors

   Li, Junyi, Chen, Jie, Ren, Ruiyang, Cheng, Xiaoxue, Zhao, Wayne Xin, Nie, Jian-Yun, Wen, Ji-Rong

Evaluates open-ended factual generation on long-tail entities mined from real user WildChat logs.

1. WildHallucinations: Evaluating Long-form Factuality in LLMs with Real-World Entity Queries — published result table ↗

   Wenting Zhao, Tanya Goyal, Yu Ying Chiu, Liwei Jiang, Benjamin Newman, Abhilasha Ravichander et al. · 2024

   All 11 authors

   Wenting Zhao, Tanya Goyal, Yu Ying Chiu, Liwei Jiang, Benjamin Newman, Abhilasha Ravichander, Khyathi Chandu, Ronan Le Bras, Claire Cardie, Yuntian Deng, Yejin Choi

A composite view summarizes model performance across professional-domain evaluations.

1. Vals Index — evaluation methodology and results ↗

   Vals AI · 2026

2. Gemini 4 Argon Model Card ↗

   Google DeepMind · 2026

Composite of Finance Agent v2.1, Excel Modeling Benchmark v1.1, and Tax Agent Bench weighted by finance sub-sector GDP.

1. Vals Finance Index — evaluation methodology and results ↗

   Vals AI · 2026

Updated Oct 2026 version of Finance Agent with new judge jury and revised institutional analyst questions (rltasks.com only lists vals-finance-agent-v2).

1. Finance Agent — evaluation methodology and results ↗

   Vals AI · 2026

Agentic financial research evaluates whether a system can use tools and evidence to complete analyst work.

1. Introducing Kimi K3.1 Pro ↗

   Moonshot AI · 2026

2. Finance Agent v2 — evaluation methodology and results ↗

   Vals AI · 2026

3. Muse Wonder | Meta ↗

   Meta AI

Evaluates agents on building and editing 3-statement financial models, DCF valuations, LBOs, and debt schedules in Excel.

1. Excel Modeling Benchmark — evaluation methodology and results ↗

   Vals AI · 2026

2. Excel Modeling Benchmark v1 — evaluation methodology and results ↗

   Vals AI · 2026

Evaluates agents on complex US corporate and international tax research, IRC code interpretation, and numerical schedules.

1. Tax Agent Bench — evaluation methodology and results ↗

   Vals AI · 2026

1,960 deterministic 10-K filing claims testing whether an excerpt supports, contradicts, or does not address a financial claim.

1. ClaimBench — evaluation methodology and results ↗

   Vals AI · 2026

Find evidence and reason across workplace document collections. Version 1 and version 2 remain separate evaluations within this collection.

1. Introducing Kimi K3.1 Pro ↗

   Moonshot AI · 2026

2. Primary benchmark documentation or model evaluation report ↗
3. OfficeQA Pro: An Enterprise Benchmark for End-to-End Grounded Reasoning — published result table ↗

   Opsahl-Ong, Krista, Singhvi, Arnav, Collins, Jasmine, Zhou, Ivan, Wang, Cindy, Baheti, Ashutosh et al. · 2026

   All 13 authors

   Opsahl-Ong, Krista, Singhvi, Arnav, Collins, Jasmine, Zhou, Ivan, Wang, Cindy, Baheti, Ashutosh, Oertell, Owen, Portes, Jacob, Havens, Sam, Elsen, Erich, Bendersky, Michael, Zaharia, Matei, Chen, Xing

Banking-support conversations combine retrieval from unstructured knowledge with state-changing tool calls.

1. 𝜏³-Banking Benchmark | Artificial Analysis ↗

   Artificial Analysis

Financial work tests sustained planning, consistency across documents, compliance constraints, and linked tool actions.

1. MiniMax M3.1 Flash: Redefining Agent Performance and Cost - MiniMax Research ↗

   MiniMax

Scale SEAL's private institutional finance, investment banking, and portfolio analysis benchmark.

1. Scale Labs — evaluation methodology and results ↗

   Scale Labs / SEAL · 2026

Financial questions require answers grounded in company filings and supporting evidence.

1. FinanceBench: A New Benchmark for Financial Question Answering — published result table ↗

   Islam, Pranab, Kannappan, Anand, Kiela, Douwe, Qian, Rebecca, Scherrer, Nino, Vidgen, Bertie · 2023

2. Primary benchmark documentation or model evaluation report ↗

Questions over financial reports require retrieving the relevant figures and composing a numerical calculation.

1. FinQA: A Dataset of Numerical Reasoning over Financial Data — published result table ↗

   Chen, Zhiyu, Chen, Wenhu, Smiley, Charese, Shah, Sameena, Borova, Iana, Langdon, Dylan et al. · 2021

   All 11 authors

   Chen, Zhiyu, Chen, Wenhu, Smiley, Charese, Shah, Sameena, Borova, Iana, Langdon, Dylan, Moussa, Reema, Beane, Matt, Huang, Ting-Hao, Routledge, Bryan, Wang, William Yang

2. Primary benchmark documentation or model evaluation report ↗

Financial questions combine evidence from tables and accompanying prose, often requiring arithmetic or multiple answer spans.

1. TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance — published result table ↗

   Zhu, Fengbin, Lei, Wenqiang, Huang, Youcheng, Wang, Chao, Zhang, Shuo, Lv, Jiancheng et al. · 2021

   All 8 authors

   Zhu, Fengbin, Lei, Wenqiang, Huang, Youcheng, Wang, Chao, Zhang, Shuo, Lv, Jiancheng, Feng, Fuli, Chua, Tat-Seng

2. Primary benchmark documentation or model evaluation report ↗

Long-context full-filing (100+ page) and multi-turn conversational extensions of FinQA.

1. ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering — published result table ↗

   Chen, Zhiyu, Li, Shiyang, Smiley, Charese, Ma, Zhiqiang, Shah, Sameena, Wang, William Yang · 2022

Quantitative finance and accounting reasoning benchmark using program synthesis over SEC tables.

1. BizBench: A Quantitative Reasoning Benchmark for Business and Finance — published result table ↗

   Rik Koncel-Kedziorski, Michael Krumdick, Viet Lai, Varshini Reddy, Charles Lovering, Chris Tanner · 2023

Private benchmark evaluating deep comprehension of long-context corporate credit agreements and debt covenants.

1. CorpFin — evaluation methodology and results ↗

   Vals AI · 2026

Visual tax-certificate extraction (MortgageTax) and tax regulation QA (TaxEval v2).

1. MortgageTax — evaluation methodology and results ↗

   Vals AI · 2026

2. TaxEval — evaluation methodology and results ↗

   Vals AI · 2026

Composite of Legal Research Bench, HLAB-Vals-1.1, and Tax Agent Bench weighted by US law-office practice-area revenue.

1. Vals Legal Index — evaluation methodology and results ↗

   Vals AI · 2026

A client matter provides documents and a legal-work request; the agent must deliver a reviewable artifact satisfying expert-written requirements.

1. Introducing Kimi K3.1 Pro ↗

   Moonshot AI · 2026

2. Harvey's Legal Agent Benchmark v1 — evaluation methodology and results ↗

   Vals AI · 2026

3. Primary benchmark documentation or model evaluation report ↗

Legal work is assessed against task criteria and the supplied source documents, with a gate for material hallucinations.

1. Harvey LAB-AA v1.1 Benchmark | Artificial Analysis ↗

   Artificial Analysis

Vals evaluates agentic legal work using Harvey’s remediated private holdout and an updated harness.

1. HLAB-Vals-1.1 — evaluation methodology and results ↗

   Vals AI · 2026

Evaluates legal agents on multi-jurisdictional US case-law and statutory research (hardest component of Vals Legal Index).

1. Legal Research Bench — evaluation methodology and results ↗

   Vals AI · 2026

Scale SEAL's private transactional and litigation legal reasoning benchmark.

1. Scale Labs — evaluation methodology and results ↗

   Scale Labs / SEAL · 2026

162 legal reasoning tasks co-designed by 40+ legal scholars across issue-spotting, rule-recall, rule-application, and interpretation.

1. LegalBench — evaluation methodology and results ↗

   Vals AI · 2026

Private question-answering benchmark over Canadian court decisions.

1. CaseLaw — evaluation methodology and results ↗

   Vals AI · 2026

510 commercial legal contracts with 13,000+ expert annotations across 41 critical legal clause categories.

1. CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review — published result table ↗

   Dan Hendrycks, Collin Burns, Anya Chen, Spencer Ball · 2021

Document-level natural language inference over Non-Disclosure Agreements (testing whether clauses entail, contradict, or omit hypotheses).

1. ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts — published result table ↗

   Koreeda, Yuta, Manning, Christopher D. · 2021

20-task Chinese civil and criminal legal system benchmark across statutory memorization, understanding, and judicial application.

1. LawBench: Benchmarking Legal Knowledge of Large Language Models — published result table ↗

   Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Songyang Zhang et al. · 2023

   All 9 authors

   Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Songyang Zhang, Kai Chen, Zongwen Shen, Jidong Ge

Published prototype observations examine economic choices under fixed prompts. Risk consistency is measured separately from patience, generosity, trust, and other stated preferences.

1. EconBench: Economic choice by language models — gpt-4o prototype metrics ↗

   EconBench project · 2026

2. EconBench: Economic choice by language models — gpt-5.2 prototype metrics ↗

   EconBench project · 2026

3. EconBench: Economic choice by language models — claude-sonnet-4-5 prototype metrics ↗

   EconBench project · 2026

4. EconBench: Economic choice by language models — o3 prototype metrics ↗

   EconBench project · 2026

5. EconBench: Economic choice by language models — claude-haiku-4-5 prototype metrics ↗

   EconBench project · 2026

6. EconBench: Economic choice by language models — claude-opus-4-5 prototype metrics ↗

   EconBench project · 2026

7. EconBench: Economic choice by language models — gpt-5-mini prototype metrics ↗

   EconBench project · 2026

8. EconBench: Economic choice by language models — gpt-5-nano prototype metrics ↗

   EconBench project · 2026

9. EconBench: Economic choice by language models — gpt-5 prototype metrics ↗

   EconBench project · 2026

10. EconBench: Economic choice by language models — gemini-2.0-flash prototype metrics ↗

    EconBench project · 2026

11. EconBench: Economic choice by language models — gemini-2.5-flash prototype metrics ↗

    EconBench project · 2026

12. EconBench: Economic choice by language models — gemini-3-flash-preview prototype metrics ↗

    EconBench project · 2026

Physician-authored rubrics evaluate health conversations. Original HealthBench and the Professional evaluation use distinct measures and settings.

1. HealthBench Professional Benchmark | Artificial Analysis ↗

   Artificial Analysis

2. Epoch AI · benchmark and methodology ↗
3. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

4. HealthBench: Evaluating Large Language Models Towards Improved Human Health — published result table ↗

   Arora, Rahul K., Wei, Jason, Hicks, Rebecca Soskin, Bowman, Preston, Quiñonero-Candela, Joaquin, Tsimpourlas, Foivos et al. · 2025

   All 12 authors

   Arora, Rahul K., Wei, Jason, Hicks, Rebecca Soskin, Bowman, Preston, Quiñonero-Candela, Joaquin, Tsimpourlas, Foivos, Sharman, Michael, Shah, Meghan, Vallone, Andrea, Beutel, Alex, Heidecke, Johannes, Singhal, Karan

5. Primary benchmark documentation or model evaluation report ↗

Five hundred medical research questions test study-design selection, evidence-gap identification, PICO formulation, and hypothesis coverage. The release reports heuristic and retrieval baselines with an explicit weighted composite.

1. UnsolvedMedical-Bench v3.0 — evaluation summary and reference baselines ↗

   Ulam AI · 2026

Long and fragmented medical records require multi-document reasoning in a claims-review setting.

1. MLCR-AA: Medical Long Context Reasoning Benchmark | Artificial Analysis ↗

   Artificial Analysis

Evaluates whether models can accurately assign ICD-10, CPT, and HCPCS medical billing codes from clinical encounters.

1. MedCode — evaluation methodology and results ↗

   Vals AI · 2026

Evaluates ambient clinical documentation, SOAP note generation, and EHR administrative support for physicians.

1. MedScribe — evaluation methodology and results ↗

   Vals AI · 2026

106 clinician-written multi-turn scenarios (32,555 conversations) testing chatbot safeguarding, clinical boundary adherence, and attachment risks with teens aged 13–17.

1. SAFE-Teen: Teen Mental Health & Safety | Vals AI A conversation branches after the chatbot turns down a request — evaluation methodology and results ↗

   Vals AI · 2026

Evaluates the clinical safety, empathy, and de-escalation quality of crisis support delivered by LLMs.

1. DistressBench — evaluation methodology and results ↗

   Scale Labs / SEAL · 2026

United States Medical Licensing Examination (Steps 1, 2CK, 3) clinical vignette multiple-choice benchmark.

1. MedQA — evaluation methodology and results ↗

   Vals AI · 2026

194,000+ Indian medical entrance exam questions across 21 clinical and preclinical medical subjects.

1. MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering — published result table ↗

   Ankit Pal, Logesh Kumar Umapathi, Malaikannan Sankarasubbu · 2022

Biomedical research QA over PubMed abstracts requiring statistical and experimental interpretation.

1. PubMedQA: A Dataset for Biomedical Research Question Answering — published result table ↗

   Jin, Qiao, Dhingra, Bhuwan, Liu, Zhengping, Cohen, William W., Lu, Xinghua · 2019

Composite clinical benchmark combining MedQA, MedMCQA, PubMedQA, MMLU clinical topics, LiveQA, MedicationQA, and consumer queries.

1. Large Language Models Encode Clinical Knowledge — published result table ↗

   Singhal, Karan, Azizi, Shekoofeh, Tu, Tao, Mahdavi, S. Sara, Wei, Jason, Chung, Hyung Won et al. · 2022

   All 30 authors

   Singhal, Karan, Azizi, Shekoofeh, Tu, Tao, Mahdavi, S. Sara, Wei, Jason, Chung, Hyung Won, Scales, Nathan, Tanwani, Ajay, Cole-Lewis, Heather, Pfohl, Stephen, Payne, Perry, Seneviratne, Martin, Gamble, Paul, Kelly, Chris, Scharli, Nathaneal, Chowdhery, Aakanksha, Mansfield, Philip, Arcas, Blaise Aguera y, Webster, Dale, Corrado, Greg S., Matias, Yossi, Chou, Katherine, Gottweis, Juraj, Tomasev, Nenad, Liu, Yun, Rajkomar, Alvin, Barral, Joelle, Semturs, Christopher, Karthikesalingam, Alan, Natarajan, Vivek

Clinical language models face seventeen datasets spanning diagnosis, recommendations, summarization, patient education, and information extraction. Published results keep each task and metric separate.

1. Large Language Models in the Clinic: A Comprehensive Benchmark — published result table ↗

   Liu, Fenglin, Li, Zheng, Zhou, Hongjian, Yin, Qingyu, Yang, Jingfeng, Tang, Xianfeng et al. · 2024

   All 19 authors

   Liu, Fenglin, Li, Zheng, Zhou, Hongjian, Yin, Qingyu, Yang, Jingfeng, Tang, Xianfeng, Luo, Chen, Zeng, Ming, Jiang, Haoming, Gao, Yifan, Nigam, Priyanka, Nag, Sreyashi, Yin, Bing, Hua, Yining, Zhou, Xuan, Rohanian, Omid, Thakur, Anshul, Clifton, Lei, Clifton, David A.

Simulated patient conversations test whether a model gathers a useful history and reaches the correct diagnosis. Compare conversational results with vignette questions and keep free-response answers separate from multiple choice.

1. An evaluation framework for clinical use of large language models in patient interaction tasks — official CRAFT-MD leaderboard ↗

   Shreya Johri, Jaehwan Jeong, Benjamin A. Tran, Daniel I. Schlessinger, Shannon Wongvibulsin, Leandra A. Barnes et al. · 2025

   All 12 authors

   Shreya Johri, Jaehwan Jeong, Benjamin A. Tran, Daniel I. Schlessinger, Shannon Wongvibulsin, Leandra A. Barnes, Hong-Yu Zhou, Zhuo Ran Cai, Eliezer M. Van Allen, David Kim, Roxana Daneshjou, Pranav Rajpurkar

4,460 specialist-board-level medical questions across 17 specialties and 11 body systems (including radiology/pathology images).

1. MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding — published result table ↗

   Yuxin Zuo, Shang Qu, Yifei Li, Zhangren Chen, Xuekai Zhu, Ermo Hua et al. · 2025

   All 9 authors

   Yuxin Zuo, Shang Qu, Yifei Li, Zhangren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, Bowen Zhou

300 realistic clinical EHR tasks operating over a live FHIR-compliant electronic health record server with 100 patient profiles.

1. MedAgentBench: A Realistic Virtual EHR Environment to Benchmark Medical LLM Agents — published result table ↗

   Jiang, Yixing, Black, Kameron C., Geng, Gloria, Park, Danny, Zou, James, Ng, Andrew Y. et al. · 2025

   All 7 authors

   Jiang, Yixing, Black, Kameron C., Geng, Gloria, Park, Danny, Zou, James, Ng, Andrew Y., Chen, Jonathan H.

Medical imaging visual question answering across X-ray, CT, MRI, ultrasound, histopathology, and fundus photography.

1. OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM — published result table ↗

   Hu, Yutao, Li, Tianbin, Lu, Quanfeng, Shao, Wenqi, He, Junjun, Qiao, Yu et al. · 2024

   All 7 authors

   Hu, Yutao, Li, Tianbin, Lu, Quanfeng, Shao, Wenqi, He, Junjun, Qiao, Yu, Luo, Ping

Long-document questions test extraction, synthesis, and reasoning across substantial inputs.

1. AA-LCR v1.1: Long Context Reasoning Benchmark | Artificial Analysis ↗

   Artificial Analysis

Professional questions require reasoning over long PDF documents and meeting task-specific requirements.

1. Introducing Kimi K3.1 Pro ↗

   Moonshot AI · 2026

2. GDP.pdf Benchmark | Artificial Analysis ↗

   Artificial Analysis

3. Epoch AI · benchmark and methodology ↗
4. Capabilities & benchmarking — retained external evaluation records ↗

   Epoch AI · 2026

Long-context tasks require tracking multiple references and retrieving the requested information from a large input.

1. GPT-6 Astra: A new generation of intelligence ↗

   OpenAI

Synthetic long-context tasks probe effective context use beyond a single simple retrieval needle.

1. HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly — published result table ↗

   Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak et al. · 2024

   All 8 authors

   Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe Wasserblat, Danqi Chen

2. Primary benchmark documentation or model evaluation report ↗

Long-context understanding and reasoning are evaluated through demanding questions over substantial inputs.

1. Primary benchmark documentation or model evaluation report ↗

Very long inputs test retrieval and understanding beyond ordinary short-context evaluations.

1. HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly — published result table ↗

   Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak et al. · 2024

   All 8 authors

   Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe Wasserblat, Danqi Chen

2. Primary benchmark documentation or model evaluation report ↗

Conversation-based tasks test an assistant’s ability to retain, update, and use information across an extended history.

1. LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory — published result table ↗

   Wu, Di, Wang, Hongwei, Yu, Wenhao, Zhang, Yuwei, Chang, Kai-Wei, Yu, Dong · 2024

2. Primary benchmark documentation or model evaluation report ↗

Meta reports a benchmark for retaining and retrieving useful context in model-driven workflows.

1. Muse Wonder | Meta ↗

   Meta AI

Questions about document images require finding and interpreting text, layout and visual evidence.

1. The Llama 3 Herd of Models — published result table ↗

   Grattafiori, Aaron, Dubey, Abhimanyu, Jauhri, Abhinav, Pandey, Abhinav, Kadian, Abhishek, Al-Dahle, Ahmad et al. · 2024

   All 561 authors

   Grattafiori, Aaron, Dubey, Abhimanyu, Jauhri, Abhinav, Pandey, Abhinav, Kadian, Abhishek, Al-Dahle, Ahmad, Letman, Aiesha, Mathur, Akhil, Schelten, Alan, Vaughan, Alex, Yang, Amy, Fan, Angela, Goyal, Anirudh, Hartshorn, Anthony, Yang, Aobo, Mitra, Archi, Sravankumar, Archie, Korenev, Artem, Hinsvark, Arthur, Rao, Arun, Zhang, Aston, Rodriguez, Aurelien, Gregerson, Austen, Spataru, Ava, Roziere, Baptiste, Biron, Bethany, Tang, Binh, Chern, Bobbie, Caucheteux, Charlotte, Nayak, Chaya, Bi, Chloe, Marra, Chris, McConnell, Chris, Keller, Christian, Touret, Christophe, Wu, Chunyang, Wong, Corinne, Ferrer, Cristian Canton, Nikolaidis, Cyrus, Allonsius, Damien, Song, Daniel, Pintz, Danielle, Livshits, Danny, Wyatt, Danny, Esiobu, David, Choudhary, Dhruv, Mahajan, Dhruv, Garcia-Olano, Diego, Perino, Diego, Hupkes, Dieuwke, Lakomkin, Egor, AlBadawy, Ehab, Lobanova, Elina, Dinan, Emily, Smith, Eric Michael, Radenovic, Filip, Guzmán, Francisco, Zhang, Frank, Synnaeve, Gabriel, Lee, Gabrielle, Anderson, Georgia Lewis, Thattai, Govind, Nail, Graeme, Mialon, Gregoire, Pang, Guan, Cucurell, Guillem, Nguyen, Hailey, Korevaar, Hannah, Xu, Hu, Touvron, Hugo, Zarov, Iliyan, Ibarra, Imanol Arrieta, Kloumann, Isabel, Misra, Ishan, Evtimov, Ivan, Zhang, Jack, Copet, Jade, Lee, Jaewon, Geffert, Jan, Vranes, Jana, Park, Jason, Mahadeokar, Jay, Shah, Jeet, van der Linde, Jelmer, Billock, Jennifer, Hong, Jenny, Lee, Jenya, Fu, Jeremy, Chi, Jianfeng, Huang, Jianyu, Liu, Jiawen, Wang, Jie, Yu, Jiecao, Bitton, Joanna, Spisak, Joe, Park, Jongsoo, Rocca, Joseph, Johnstun, Joshua, Saxe, Joshua, Jia, Junteng, Alwala, Kalyan Vasuden, Prasad, Karthik, Upasani, Kartikeya, Plawiak, Kate, Li, Ke, Heafield, Kenneth, Stone, Kevin, El-Arini, Khalid, Iyer, Krithika, Malik, Kshitiz, Chiu, Kuenley, Bhalla, Kunal, Lakhotia, Kushal, Rantala-Yeary, Lauren, van der Maaten, Laurens, Chen, Lawrence, Tan, Liang, Jenkins, Liz, Martin, Louis, Madaan, Lovish, Malo, Lubo, Blecher, Lukas, Landzaat, Lukas, de Oliveira, Luke, Muzzi, Madeline, Pasupuleti, Mahesh, Singh, Mannat, Paluri, Manohar, Kardas, Marcin, Tsimpoukelli, Maria, Oldham, Mathew, Rita, Mathieu, Pavlova, Maya, Kambadur, Melanie, Lewis, Mike, Si, Min, Singh, Mitesh Kumar, Hassan, Mona, Goyal, Naman, Torabi, Narjes, Bashlykov, Nikolay, Bogoychev, Nikolay, Chatterji, Niladri, Zhang, Ning, Duchenne, Olivier, Çelebi, Onur, Alrassy, Patrick, Zhang, Pengchuan, Li, Pengwei, Vasic, Petar, Weng, Peter, Bhargava, Prajjwal, Dubal, Pratik, Krishnan, Praveen, Koura, Punit Singh, Xu, Puxin, He, Qing, Dong, Qingxiao, Srinivasan, Ragavan, Ganapathy, Raj, Calderer, Ramon, Cabral, Ricardo Silveira, Stojnic, Robert, Raileanu, Roberta, Maheswari, Rohan, Girdhar, Rohit, Patel, Rohit, Sauvestre, Romain, Polidoro, Ronnie, Sumbaly, Roshan, Taylor, Ross, Silva, Ruan, Hou, Rui, Wang, Rui, Hosseini, Saghar, Chennabasappa, Sahana, Singh, Sanjay, Bell, Sean, Kim, Seohyun Sonia, Edunov, Sergey, Nie, Shaoliang, Narang, Sharan, Raparthy, Sharath, Shen, Sheng, Wan, Shengye, Bhosale, Shruti, Zhang, Shun, Vandenhende, Simon, Batra, Soumya, Whitman, Spencer, Sootla, Sten, Collot, Stephane, Gururangan, Suchin, Borodinsky, Sydney, Herman, Tamar, Fowler, Tara, Sheasha, Tarek, Georgiou, Thomas, Scialom, Thomas, Speckbacher, Tobias, Mihaylov, Todor, Xiao, Tong, Karn, Ujjwal, Goswami, Vedanuj, Gupta, Vibhor, Ramanathan, Vignesh, Kerkez, Viktor, Gonguet, Vincent, Do, Virginie, Vogeti, Vish, Albiero, Vítor, Petrovic, Vladan, Chu, Weiwei, Xiong, Wenhan, Fu, Wenyin, Meers, Whitney, Martinet, Xavier, Wang, Xiaodong, Wang, Xiaofang, Tan, Xiaoqing Ellen, Xia, Xide, Xie, Xinfeng, Jia, Xuchao, Wang, Xuewei, Goldschlag, Yaelle, Gaur, Yashesh, Babaei, Yasmine, Wen, Yi, Song, Yiwen, Zhang, Yuchen, Li, Yue, Mao, Yuning, Coudert, Zacharie Delpierre, Yan, Zheng, Chen, Zhengxing, Papakipos, Zoe, Singh, Aaditya, Srivastava, Aayushi, Jain, Abha, Kelsey, Adam, Shajnfeld, Adam, Gangidi, Adithya, Victoria, Adolfo, Goldstand, Ahuva, Menon, Ajay, Sharma, Ajay, Boesenberg, Alex, Baevski, Alexei, Feinstein, Allie, Kallet, Amanda, Sangani, Amit, Teo, Amos, Yunus, Anam, Lupu, Andrei, Alvarado, Andres, Caples, Andrew, Gu, Andrew, Ho, Andrew, Poulton, Andrew, Ryan, Andrew, Ramchandani, Ankit, Dong, Annie, Franco, Annie, Goyal, Anuj, Saraf, Aparajita, Chowdhury, Arkabandhu, Gabriel, Ashley, Bharambe, Ashwin, Eisenman, Assaf, Yazdan, Azadeh, James, Beau, Maurer, Ben, Leonhardi, Benjamin, Huang, Bernie, Loyd, Beth, De Paola, Beto, Paranjape, Bhargavi, Liu, Bing, Wu, Bo, Ni, Boyu, Hancock, Braden, Wasti, Bram, Spence, Brandon, Stojkovic, Brani, Gamido, Brian, Montalvo, Britt, Parker, Carl, Burton, Carly, Mejia, Catalina, Liu, Ce, Wang, Changhan, Kim, Changkyu, Zhou, Chao, Hu, Chester, Chu, Ching-Hsiang, Cai, Chris, Tindal, Chris, Feichtenhofer, Christoph, Gao, Cynthia, Civin, Damon, Beaty, Dana, Kreymer, Daniel, Li, Daniel, Adkins, David, Xu, David, Testuggine, Davide, David, Delia, Parikh, Devi, Liskovich, Diana, Foss, Didem, Wang, Dingkang, Le, Duc, Holland, Dustin, Dowling, Edward, Jamil, Eissa, Montgomery, Elaine, Presani, Eleonora, Hahn, Emily, Wood, Emily, Le, Eric-Tuan, Brinkman, Erik, Arcaute, Esteban, Dunbar, Evan, Smothers, Evan, Sun, Fei, Kreuk, Felix, Tian, Feng, Kokkinos, Filippos, Ozgenel, Firat, Caggioni, Francesco, Kanayet, Frank, Seide, Frank, Florez, Gabriela Medina, Schwarz, Gabriella, Badeer, Gada, Swee, Georgia, Halpern, Gil, Herman, Grant, Sizov, Grigory, Guangyi, Zhang, Lakshminarayanan, Guna, Inan, Hakan, Shojanazeri, Hamid, Zou, Han, Wang, Hannah, Zha, Hanwen, Habeeb, Haroun, Rudolph, Harrison, Suk, Helen, Aspegren, Henry, Goldman, Hunter, Zhan, Hongyuan, Damlaj, Ibrahim, Molybog, Igor, Tufanov, Igor, Leontiadis, Ilias, Veliche, Irina-Elena, Gat, Itai, Weissman, Jake, Geboski, James, Kohli, James, Lam, Janice, Asher, Japhet, Gaya, Jean-Baptiste, Marcus, Jeff, Tang, Jeff, Chan, Jennifer, Zhen, Jenny, Reizenstein, Jeremy, Teboul, Jeremy, Zhong, Jessica, Jin, Jian, Yang, Jingyi, Cummings, Joe, Carvill, Jon, Shepard, Jon, McPhie, Jonathan, Torres, Jonathan, Ginsburg, Josh, Wang, Junjie, Wu, Kai, U, Kam Hou, Saxena, Karan, Khandelwal, Kartikay, Zand, Katayoun, Matosich, Kathy, Veeraraghavan, Kaushik, Michelena, Kelly, Li, Keqian, Jagadeesh, Kiran, Huang, Kun, Chawla, Kunal, Huang, Kyle, Chen, Lailin, Garg, Lakshya, A, Lavender, Silva, Leandro, Bell, Lee, Zhang, Lei, Guo, Liangpeng, Yu, Licheng, Moshkovich, Liron, Wehrstedt, Luca, Khabsa, Madian, Avalani, Manav, Bhatt, Manish, Mankus, Martynas, Hasson, Matan, Lennie, Matthew, Reso, Matthias, Groshev, Maxim, Naumov, Maxim, Lathi, Maya, Keneally, Meghan, Liu, Miao, Seltzer, Michael L., Valko, Michal, Restrepo, Michelle, Patel, Mihir, Vyatskov, Mik, Samvelyan, Mikayel, Clark, Mike, Macey, Mike, Wang, Mike, Hermoso, Miquel Jubert, Metanat, Mo, Rastegari, Mohammad, Bansal, Munish, Santhanam, Nandhini, Parks, Natascha, White, Natasha, Bawa, Navyata, Singhal, Nayan, Egebo, Nick, Usunier, Nicolas, Mehta, Nikhil, Laptev, Nikolay Pavlovich, Dong, Ning, Cheng, Norman, Chernoguz, Oleg, Hart, Olivia, Salpekar, Omkar, Kalinli, Ozlem, Kent, Parkin, Parekh, Parth, Saab, Paul, Balaji, Pavan, Rittner, Pedro, Bontrager, Philip, Roux, Pierre, Dollar, Piotr, Zvyagina, Polina, Ratanchandani, Prashant, Yuvraj, Pritish, Liang, Qian, Alao, Rachad, Rodriguez, Rachel, Ayub, Rafi, Murthy, Raghotham, Nayani, Raghu, Mitra, Rahul, Parthasarathy, Rangaprabhu, Li, Raymond, Hogan, Rebekkah, Battey, Robin, Wang, Rocky, Howes, Russ, Rinott, Ruty, Mehta, Sachin, Siby, Sachin, Bondu, Sai Jayesh, Datta, Samyak, Chugh, Sara, Hunt, Sara, Dhillon, Sargun, Sidorov, Sasha, Pan, Satadru, Mahajan, Saurabh, Verma, Saurabh, Yamamoto, Seiji, Ramaswamy, Sharadh, Lindsay, Shaun, Lindsay, Shaun, Feng, Sheng, Lin, Shenghao, Zha, Shengxin Cindy, Patil, Shishir, Shankar, Shiva, Zhang, Shuqiang, Zhang, Shuqiang, Wang, Sinong, Agarwal, Sneha, Sajuyigbe, Soji, Chintala, Soumith, Max, Stephanie, Chen, Stephen, Kehoe, Steve, Satterfield, Steve, Govindaprasad, Sudarshan, Gupta, Sumit, Deng, Summer, Cho, Sungmin, Virk, Sunny, Subramanian, Suraj, Choudhury, Sy, Goldman, Sydney, Remez, Tal, Glaser, Tamar, Best, Tamara, Koehler, Thilo, Robinson, Thomas, Li, Tianhe, Zhang, Tianjun, Matthews, Tim, Chou, Timothy, Shaked, Tzook, Vontimitta, Varun, Ajayi, Victoria, Montanez, Victoria, Mohan, Vijai, Kumar, Vinay Satish, Mangla, Vishal, Ionescu, Vlad, Poenaru, Vlad, Mihailescu, Vlad Tiberiu, Ivanov, Vladimir, Li, Wei, Wang, Wenchen, Jiang, Wenwen, Bouaziz, Wes, Constable, Will, Tang, Xiaocheng, Wu, Xiaojian, Wang, Xiaolan, Wu, Xilun, Gao, Xinbo, Kleinman, Yaniv, Chen, Yanjun, Hu, Ye, Jia, Ye, Qi, Ye, Li, Yenda, Zhang, Yilin, Zhang, Ying, Adi, Yossi, Nam, Youngjin, Yu, Wang, Zhao, Yu, Hao, Yuchen, Qian, Yundi, Li, Yunlu, He, Yuzi, Rait, Zach, DeVito, Zachary, Rosnbrick, Zef, Wen, Zhaoduo, Yang, Zhenyu, Zhao, Zhiwei, Ma, Zhiyu

2. Primary benchmark documentation or model evaluation report ↗

Comprehensive long-context suite spanning RAG, citation generation, re-ranking, many-shot ICL, long-doc QA, summarization, and synthetic recall up to 128k+.

1. HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly — published result table ↗

   Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak et al. · 2024

   All 8 authors

   Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe Wasserblat, Danqi Chen

Classic pressure test inserting target facts ("needles") at varying document depths (0–100%) and context lengths (4k–2M tokens).

1. HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly — published result table ↗

   Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak et al. · 2024

   All 8 authors

   Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe Wasserblat, Danqi Chen

2. Needle In A Multimodal Haystack — published result table ↗

   Weiyun Wang, Shuibo Zhang, Yiming Ren, Yuchen Duan, Tiantong Li, Shuo Liu et al. · 2024

   All 16 authors

   Weiyun Wang, Shuibo Zhang, Yiming Ren, Yuchen Duan, Tiantong Li, Shuo Liu, Mengkang Hu, Zhe Chen, Kaipeng Zhang, Lewei Lu, Xizhou Zhu, Ping Luo, Yu Qiao, Jifeng Dai, Wenqi Shao, Wenhai Wang

Embeds multi-step bAbI deductive reasoning chains inside massive PG19 book haystacks from 4k to 10M tokens.

1. BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack — official v0 leaderboard — published result table ↗

   Kuratov, Yuri, Bulatov, Aydar, Anokhin, Petr, Rodkin, Ivan, Sorokin, Dmitry, Sorokin, Artyom et al. · 2024

   All 7 authors

   Kuratov, Yuri, Bulatov, Aydar, Anokhin, Petr, Rodkin, Ivan, Sorokin, Dmitry, Sorokin, Artyom, Burtsev, Mikhail

Evaluates long-context synthesis beyond surface retrieval via CodeRepoQA, BugLocalization, and IDK abstention.

1. Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries — published result table ↗

   Vodrahalli, Kiran, Ontanon, Santiago, Tripuraneni, Nilesh, Xu, Kelvin, Jain, Sanil, Shivanna, Rakesh et al. · 2024

   All 24 authors

   Vodrahalli, Kiran, Ontanon, Santiago, Tripuraneni, Nilesh, Xu, Kelvin, Jain, Sanil, Shivanna, Rakesh, Hui, Jeffrey, Dikkala, Nishanth, Kazemi, Mehran, Fatemi, Bahare, Anil, Rohan, Dyer, Ethan, Shakeri, Siamak, Vij, Roopali, Mehta, Harsh, Ramasesh, Vinay, Le, Quoc, Chi, Ed, Lu, Yifeng, Firat, Orhan, Lazaridou, Angeliki, Lespiau, Jean-Baptiste, Attaluri, Nithya, Olszewska, Kate

Tests whether 1M-token context windows can replace external RAG/SQL/VectorDB pipelines by placing entire corpora directly in-context.

1. Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More? — published result table ↗

   Lee, Jinhyuk, Chen, Anthony, Dai, Zhuyun, Dua, Dheeru, Sachan, Devendra Singh, Boratko, Michael et al. · 2024

   All 19 authors

   Lee, Jinhyuk, Chen, Anthony, Dai, Zhuyun, Dua, Dheeru, Sachan, Devendra Singh, Boratko, Michael, Luan, Yi, Arnold, Sébastien M. R., Perot, Vincent, Dalmia, Siddharth, Hu, Hexiang, Lin, Xudong, Pasupat, Panupong, Amini, Aida, Cole, Jeremy R., Riedel, Sebastian, Naim, Iftekhar, Chang, Ming-Wei, Guu, Kelvin

True/False claim verification over recently published full-length English novels (60k–300k tokens) requiring global plot comprehension.

1. One Thousand and One Pairs: A "novel" challenge for long-context language models — published result table ↗

   Marzena Karpinska, Katherine Thai, Kyle Lo, Tanya Goyal, Mohit Iyyer · 2024

Long-narrative whodunit deduction and book/movie script reading comprehension.

1. DetectiveQA: Evaluating Long-Context Reasoning on Detective Novels — published result table ↗

   Zhe Xu, Jiasheng Ye, Xiaoran Liu, Xiangyang Liu, Tianxiang Sun, Zhigeng Liu et al. · 2024

   All 11 authors

   Zhe Xu, Jiasheng Ye, Xiaoran Liu, Xiangyang Liu, Tianxiang Sun, Zhigeng Liu, Qipeng Guo, Linlin Li, Qun Liu, Xuanjing Huang, Xipeng Qiu

Evaluates semantic function retrieval (RepoQA) and project-wide coding tasks (LongCodeArena) across whole code repositories.

1. RepoQA: Evaluating Long Context Code Understanding — published result table ↗

   Jiawei Liu, Jia Le Tian, Vijay Daita, Yuxiang Wei, Yifeng Ding, Yuhan Katherine Wang et al. · 2024

   All 8 authors

   Jiawei Liu, Jia Le Tian, Vijay Daita, Yuxiang Wei, Yifeng Ding, Yuhan Katherine Wang, Jun Yang, Lingming Zhang

Very long-term multi-modal dialogue benchmark spanning 300+ turns and 35 sessions per persona pair.

1. Evaluating Very Long-Term Conversational Memory of LLM Agents — published result table ↗

   Maharana, Adyasha, Lee, Dong-Ho, Tulyakov, Sergey, Bansal, Mohit, Barbieri, Francesco, Fang, Yuwei · 2024

First-generation standardized bilingual long-context evaluation suites (4k–64k tokens).

1. L-Eval: Instituting Standardized Evaluation for Long Context Language Models — published result table ↗

   Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang et al. · 2023

   All 8 authors

   Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, Xipeng Qiu

Standardized CompaRison Over Long Language Sequences across GovReport, SummScreenFD, QMSum, Qasper, NarrativeQA, and QuALITY.

1. SCROLLS: Standardized CompaRison Over Long Language Sequences — published result table ↗

   Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv et al. · 2022

   All 11 authors

   Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, Omer Levy

Stress-tests long-context output generation (generating 8k–64k token structured outputs, HTML, travel plans, and trace logs) rather than only long input reading.

1. LongProc: Benchmarking Long-Context Language Models on Long Procedural Generation — published result table ↗

   Xi Ye, Fangcong Yin, Yinghui He, Joie Zhang, Howard Yen, Tianyu Gao et al. · 2025

   All 8 authors

   Xi Ye, Fangcong Yin, Yinghui He, Joie Zhang, Howard Yen, Tianyu Gao, Greg Durrett, Danqi Chen

Evaluates whether LLMs can sustain coherent, high-quality generation over 2,000 to 20,000+ output words.

1. LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs — published result table ↗

   Bai, Yushi, Zhang, Jiajie, Lv, Xin, Zheng, Linzhi, Zhu, Siqi, Hou, Lei et al. · 2024

   All 9 authors

   Bai, Yushi, Zhang, Jiajie, Lv, Xin, Zheng, Linzhi, Zhu, Siqi, Hou, Lei, Dong, Yuxiao, Tang, Jie, Li, Juanzi

Blind pairwise human choices compare model responses in open-ended interactions.

1. Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models — published result table ↗

   Deitke, Matt, Clark, Christopher, Lee, Sangho, Tripathi, Rohun, Yang, Yue, Park, Jae Sung et al. · 2024

   All 50 authors

   Deitke, Matt, Clark, Christopher, Lee, Sangho, Tripathi, Rohun, Yang, Yue, Park, Jae Sung, Salehi, Mohammadreza, Muennighoff, Niklas, Lo, Kyle, Soldaini, Luca, Lu, Jiasen, Anderson, Taira, Bransom, Erin, Ehsani, Kiana, Ngo, Huong, Chen, YenSung, Patel, Ajay, Yatskar, Mark, Callison-Burch, Chris, Head, Andrew, Hendrix, Rose, Bastani, Favyen, VanderBilt, Eli, Lambert, Nathan, Chou, Yvonne, Chheda, Arnavi, Sparks, Jenna, Skjonsberg, Sam, Schmitz, Michael, Sarnat, Aaron, Bischoff, Byron, Walsh, Pete, Newell, Chris, Wolters, Piper, Gupta, Tanmay, Zeng, Kuo-Hao, Borchardt, Jon, Groeneveld, Dirk, Nam, Crystal, Lebrecht, Sophie, Wittlif, Caitlin, Schoenick, Carissa, Michel, Oscar, Krishna, Ranjay, Weihs, Luca, Smith, Noah A., Hajishirzi, Hannaneh, Girshick, Ross, Farhadi, Ali, Kembhavi, Aniruddha

2. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference ↗

   Chiang, Wei-Lin, Zheng, Lianmin, Sheng, Ying, Angelopoulos, Anastasios Nikolas, Li, Tianle, Li, Dacheng et al. · 2024

   All 11 authors

   Chiang, Wei-Lin, Zheng, Lianmin, Sheng, Ying, Angelopoulos, Anastasios Nikolas, Li, Tianle, Li, Dacheng, Zhang, Hao, Zhu, Banghua, Jordan, Michael, Gonzalez, Joseph E., Stoica, Ion

Large-scale blind pairwise preference arena segmented by real-world user profession, education level, and language.

1. SEAL Showdown Technical Report ↗

   Scale AI · 2025

Challenging assistant prompts support automated, preference-oriented evaluation of response quality.

1. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning — published result table ↗

   DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang et al. · 2025

   All 200 authors

   DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, Zhen Zhang

2. Primary benchmark documentation or model evaluation report ↗

Challenging user tasks support evaluation of practical assistant responses using task-aware judgments.

1. WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild — published result table ↗

   Lin, Bill Yuchen, Deng, Yuntian, Chandu, Khyathi, Brahman, Faeze, Ravichander, Abhilasha, Pyatkin, Valentina et al. · 2024

   All 9 authors

   Lin, Bill Yuchen, Deng, Yuntian, Chandu, Khyathi, Brahman, Faeze, Ravichander, Abhilasha, Pyatkin, Valentina, Dziri, Nouha, Bras, Ronan Le, Choi, Yejin

2. Primary benchmark documentation or model evaluation report ↗

805 instruction-following prompts evaluated via GLM regression that debiasses judge preference for verbose outputs (0.98 Spearman correlation with Chatbot Arena).

1. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning — published result table ↗

   DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang et al. · 2025

   All 200 authors

   DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, Zhen Zhang

Foundational 80-prompt two-turn conversational instruction-following benchmark across 8 categories (and 1,388-dialog 10-turn extension).

1. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — published result table ↗

   Zheng, Lianmin, Chiang, Wei-Lin, Sheng, Ying, Zhuang, Siyuan, Wu, Zhanghao, Zhuang, Yonghao et al. · 2023

   All 13 authors

   Zheng, Lianmin, Chiang, Wei-Lin, Sheng, Ying, Zhuang, Siyuan, Wu, Zhanghao, Zhuang, Yonghao, Lin, Zi, Li, Zhuohan, Li, Dacheng, Xing, Eric P., Zhang, Hao, Gonzalez, Joseph E., Stoica, Ion

Programmatically checkable instructions test whether model outputs satisfy explicit constraints.

1. Primary benchmark documentation or model evaluation report ↗

Verifiable output constraints probe whether instruction following transfers beyond familiar formats.

1. IFBench Benchmark | Artificial Analysis ↗

   Artificial Analysis

Multi-turn conversational benchmark testing instruction retention, inference memory, versioned self-revision, and sycophancy resistance.

1. Scale Labs — evaluation methodology and results ↗

   Scale Labs / SEAL · 2026

2. Scale Labs — evaluation methodology and results ↗

   Scale Labs / SEAL · 2026

Spoken multi-turn dialogue counterpart to MultiChallenge testing native voice-to-voice and voice-to-text instruction following.

1. Scale Labs — evaluation methodology and results ↗

   Scale Labs / SEAL · 2026

2. Scale Labs — evaluation methodology and results ↗

   Scale Labs / SEAL · 2026

3. Scale Labs — evaluation methodology and results ↗

   Scale Labs / SEAL · 2026

Evaluates hierarchical compositions of multiple constraints (AND, OR, Selection/If-Else, and Nested chains).

1. Benchmarking Complex Instruction-Following with Multiple Constraints Composition — published result table ↗

   Bosi Wen, Pei Ke, Xiaotao Gu, Lindong Wu, Hao Huang, Jinfeng Zhou et al. · 2024

   All 14 authors

   Bosi Wen, Pei Ke, Xiaotao Gu, Lindong Wu, Hao Huang, Jinfeng Zhou, Wenchuang Li, Binxin Hu, Wendy Gao, Jiaxin Xu, Yiming Liu, Jie Tang, Hongning Wang, Minlie Huang

Multi-level fine-grained constraint-following benchmark adding 1 to 5 incremental constraints per seed instruction.

1. FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models — published result table ↗

   Jiang, Yuxin, Wang, Yufei, Zeng, Xingshan, Zhong, Wanjun, Li, Liangyou, Mi, Fei et al. · 2023

   All 10 authors

   Jiang, Yuxin, Wang, Yufei, Zeng, Xingshan, Zhong, Wanjun, Li, Liangyou, Mi, Fei, Shang, Lifeng, Jiang, Xin, Liu, Qun, Wang, Wei

Decomposes complex instructions into 2,250 atomic Yes/No criteria across 500 prompts.

1. InFoBench: Evaluating Instruction Following Ability in Large Language Models — published result table ↗

   Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang et al. · 2024

   All 10 authors

   Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, Dong Yu

1,000 real-world Chinese instructions covering 200+ scenario categories and 50+ constraint types.

1. CFBench: A Comprehensive Constraints-Following Benchmark for LLMs — published result table ↗

   Tao Zhang, Chenglin Zhu, Yanjun Shen, Wenjing Luo, Yan Zhang, Hao Liang et al. · 2024

   All 14 authors

   Tao Zhang, Chenglin Zhu, Yanjun Shen, Wenjing Luo, Yan Zhang, Hao Liang, Tao Zhang, Fan Yang, Mingan Lin, Yujing Qiao, Weipeng Chen, Bin Cui, Wentao Zhang, Zenan Zhou

Evaluates multi-turn adherence to complex developer system prompts under distracting or conflicting user messages.

1. SysBench: Can Large Language Models Follow System Messages? — published result table ↗

   Yanzhao Qin, Tao Zhang, Tao Zhang, Yanjun Shen, Wenjing Luo, Haoze Sun et al. · 2024

   All 12 authors

   Yanzhao Qin, Tao Zhang, Tao Zhang, Yanjun Shen, Wenjing Luo, Haoze Sun, Yan Zhang, Yujing Qiao, Weipeng Chen, Zenan Zhou, Wentao Zhang, Bin Cui

Communication traits and measured writing characteristics describe how models express an answer.

1. AA-PersonalityBench: AI Personality & Communication Style Benchmark | Artificial Analysis ↗

   Artificial Analysis

Evaluates emotional intelligence, interpersonal nuance, theory of mind, and non-slop literary prose quality.

1. EQ-Bench: An Emotional Intelligence Benchmark for Large Language Models — published result table ↗

   Paech, Samuel J. · 2023

Standard benchmark for evaluating Reward Models and DPO policies across Chat, Chat-Hard, Safety, Reasoning, Math, Code, and Ties (RewardBench 2 adds unseen multi-choice & tie calibration).

1. RewardBench 2: Advancing Reward Model Evaluation — published result table ↗

   Saumya Malik, Valentina Pyatkin, Sander Land, Jacob Morrison, Noah A. Smith, Hannaneh Hajishirzi et al. · 2025

   All 7 authors

   Saumya Malik, Valentina Pyatkin, Sander Land, Jacob Morrison, Noah A. Smith, Hannaneh Hajishirzi, Nathan Lambert

Evaluates reward models' sensitivity to subtle factual/code errors vs. resistance to superficial formatting/markdown style bias ($3\times 3$ style-substance matrix).

1. RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style — published result table ↗

   Liu, Yantao, Yao, Zijun, Min, Rui, Cao, Yixin, Hou, Lei, Li, Juanzi · 2024

350 challenging response pairs with objective ground truth across Knowledge, Reasoning, Math, and Coding testing LLM-as-a-Judge models.

1. JudgeBench: A Benchmark for Evaluating LLM-based Judges — published result table ↗

   Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y. Tang, Alejandro Cuadron, Chenguang Wang et al. · 2024

   All 8 authors

   Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y. Tang, Alejandro Cuadron, Chenguang Wang, Raluca Ada Popa, Ion Stoica

Evaluates reward models and judges on how well optimizing against them predicts real human Chatbot Arena gains.

1. How to Evaluate Reward Models for RLHF — published result table ↗

   Evan Frick, Tianle Li, Connor Chen, Wei-Lin Chiang, Anastasios N. Angelopoulos, Jiantao Jiao et al. · 2024

   All 9 authors

   Evan Frick, Tianle Li, Connor Chen, Wei-Lin Chiang, Anastasios N. Angelopoulos, Jiantao Jiao, Banghua Zhu, Joseph E. Gonzalez, Ion Stoica

3,400 competition math solutions with step-by-step human annotations evaluating Process Reward Models (PRMs) on identifying the exact first wrong step.

1. ProcessBench: Identifying Process Errors in Mathematical Reasoning — published result table ↗

   Zheng, Chujie, Zhang, Zhenru, Zhang, Beichen, Lin, Runji, Lu, Keming, Yu, Bowen et al. · 2024

   All 9 authors

   Zheng, Chujie, Zhang, Zhenru, Zhang, Beichen, Lin, Runji, Lu, Keming, Yu, Bowen, Liu, Dayiheng, Zhou, Jingren, Lin, Junyang

6,216 fine-grained step-level error problems evaluating PRMs across simplicity, soundness, and sensitivity (non-redundancy, circular logic, phantom steps).

1. PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models — published result table ↗

   Song, Mingyang, Su, Zhaochen, Qu, Xiaoye, Zhou, Jiawei, Cheng, Yu · 2025

Evaluates an LLM's ability to critique and correct reasoning, math, code, and factual generations.

1. CriticBench: Benchmarking LLMs for Critique-Correct Reasoning — published result table ↗

   Zicheng Lin, Zhibin Gou, Tian Liang, Ruilin Luo, Haowei Liu, Yujiu Yang · 2024

Grammar-based constrained text generation (Collie) and hierarchical constraint compliance (Conifer).

1. COLLIE: Systematic Construction of Constrained Text Generation Tasks — published result table ↗

   Shunyu Yao, Howard Chen, Austin W. Hanjie, Runzhe Yang, Karthik Narasimhan · 2023

Pairwise model judgments compare instruction-following responses with a fixed reference. Original and length-controlled evaluator versions answer different questions.

1. AlpacaEval: An Automatic Evaluator of Instruction-following Models ↗

   Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin et al. · 2023

   All 8 authors

   Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, Tatsunori B. Hashimoto

Evaluates whether models abandon correct answers or flatter mistaken user beliefs when challenged ("Are you sure?").

1. Towards Understanding Sycophancy in Language Models — published result table ↗

   Sharma, Mrinank, Tong, Meg, Korbak, Tomasz, Duvenaud, David, Askell, Amanda, Bowman, Samuel R. et al. · 2023

   All 19 authors

   Sharma, Mrinank, Tong, Meg, Korbak, Tomasz, Duvenaud, David, Askell, Amanda, Bowman, Samuel R., Cheng, Newton, Durmus, Esin, Hatfield-Dodds, Zac, Johnston, Scott R., Kravec, Shauna, Maxwell, Timothy, McCandlish, Sam, Ndousse, Kamal, Rausch, Oliver, Schiefer, Nicholas, Yan, Da, Zhang, Miranda, Perez, Ethan

Quantifies overused RLHF cliches ("delve", "testament", "tapestry") and vacuous hedging.

1. BullshitBench V1 — published model leaderboard ↗

   Peter (petergpt) · 2026

2. BullshitBench V2 — published model leaderboard ↗

   Peter (petergpt) · 2026

A standardized framework measures harmful-compliance and refusal behavior under defined evaluation conditions.

1. SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal — published result table ↗

   Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Madhushani Sehwag, Kaixuan Huang et al. · 2024

   All 16 authors

   Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Madhushani Sehwag, Kaixuan Huang, Luxi He, Boyi Wei, Dacheng Li, Ying Sheng, Ruoxi Jia, Bo Li, Kai Li, Danqi Chen, Peter Henderson, Prateek Mittal

2. Primary benchmark documentation or model evaluation report ↗

A hazardous-knowledge proxy supports research on evaluating and reducing specific kinds of model knowledge.

1. The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning — published result table ↗

   Li, Nathaniel, Pan, Alexander, Gopal, Anjali, Yue, Summer, Berrios, Daniel, Gatti, Alice et al. · 2024

   All 57 authors

   Li, Nathaniel, Pan, Alexander, Gopal, Anjali, Yue, Summer, Berrios, Daniel, Gatti, Alice, Li, Justin D., Dombrowski, Ann-Kathrin, Goel, Shashwat, Phan, Long, Mukobi, Gabriel, Helm-Burger, Nathan, Lababidi, Rassin, Justen, Lennart, Liu, Andrew B., Chen, Michael, Barrass, Isabelle, Zhang, Oliver, Zhu, Xiaoyuan, Tamirisa, Rishub, Bharathi, Bhrugu, Khoja, Adam, Zhao, Zhenqi, Herbert-Voss, Ariel, Breuer, Cort B., Marks, Samuel, Patel, Oam, Zou, Andy, Mazeika, Mantas, Wang, Zifan, Oswal, Palash, Lin, Weiran, Hunt, Adam A., Tienken-Harder, Justin, Shih, Kevin Y., Talley, Kemper, Guan, John, Kaplan, Russell, Steneker, Ian, Campbell, David, Jokubaitis, Brad, Levinson, Alex, Wang, Jean, Qian, William, Karmakar, Kallol Krishna, Basart, Steven, Fitz, Stephen, Levine, Mindy, Kumaraguru, Ponnurangam, Tupakula, Uday, Varadharajan, Vijay, Wang, Ruoyu, Shoshitaishvili, Yan, Ba, Jimmy, Esvelt, Kevin M., Wang, Alexandr, Hendrycks, Dan

2. Primary benchmark documentation or model evaluation report ↗

Frontier Risk Evaluation for National Security and Public Safety (dual-use CBRN, critical infrastructure, autonomous cyber risk).

1. Scale Labs — evaluation methodology and results ↗

   Scale Labs / SEAL · 2026

Measures *honesty* separately from *accuracy*: elicits a model's true internal belief in a neutral context, then tests whether the model knowingly lies under user/system pressure.

1. Scale Labs — evaluation methodology and results ↗

   Scale Labs / SEAL · 2026

Evaluates jailbreak effectiveness using a rubric judge that penalizes vague/non-actionable compliance and measures genuinely useful harmful assistance.

1. SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal — published result table ↗

   Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Madhushani Sehwag, Kaixuan Huang et al. · 2024

   All 16 authors

   Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Madhushani Sehwag, Kaixuan Huang, Luxi He, Boyi Wei, Dacheng Li, Ying Sheng, Ruoxi Jia, Bo Li, Kai Li, Danqi Chen, Peter Henderson, Prateek Mittal

Open-source robustness benchmark tracking 100 distinct misuse behaviors alongside benign counterparts.

1. JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models — published result table ↗

   Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag et al. · 2024

   All 12 authors

   Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramer, Hamed Hassani, Eric Wong

250 safe prompts using sensitive-sounding vocabulary ("How do I kill a Python process?", "Where can I shoot a photo?") paired with 200 genuinely unsafe contrasts.

1. XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models — published result table ↗

   Röttger, Paul, Kirk, Hannah Rose, Vidgen, Bertie, Attanasio, Giuseppe, Bianchi, Federico, Hovy, Dirk · 2023

Evaluates nuanced contextual non-compliance (unsafe requests, false premises, unanswerable queries, unsupported modalities) vs. over-refusal.

1. The Art of Saying No: Contextual Noncompliance in Language Models — published result table ↗

   Brahman, Faeze, Kumar, Sachin, Balachandran, Vidhisha, Dasigi, Pradeep, Pyatkin, Valentina, Ravichander, Abhilasha et al. · 2024

   All 14 authors

   Brahman, Faeze, Kumar, Sachin, Balachandran, Vidhisha, Dasigi, Pradeep, Pyatkin, Valentina, Ravichander, Abhilasha, Wiegreffe, Sarah, Dziri, Nouha, Chandu, Khyathi, Hessel, Jack, Tsvetkov, Yulia, Smith, Noah A., Choi, Yejin, Hajishirzi, Hannaneh

5,000+ labeled adversarial and vanilla user-model interactions testing safety moderation classifiers and guardrail models.

1. WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs — published result table ↗

   Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert et al. · 2024

   All 8 authors

   Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, Nouha Dziri

Balanced 45-class fine-grained safety taxonomy with 20 linguistic mutations (slang, encoding, translation, roleplay) per unsafe prompt.

1. SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal — published result table ↗

   Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Madhushani Sehwag, Kaixuan Huang et al. · 2024

   All 16 authors

   Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Madhushani Sehwag, Kaixuan Huang, Luxi He, Boyi Wei, Dacheng Li, Ying Sheng, Ruoxi Jia, Bo Li, Kai Li, Danqi Chen, Peter Henderson, Prateek Mittal

Meta’s evaluation family covers distinct cybersecurity tasks. The imported CyberSOCEval comparisons measure how models analyze malware reports and reason over threat intelligence, with task results kept separate.

1. CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning — published result table ↗

   Deason, Lauren, Bali, Adam, Bejean, Ciprian, Bolocan, Diana, Crnkovich, James, Croitoru, Ioana et al. · 2025

   All 23 authors

   Deason, Lauren, Bali, Adam, Bejean, Ciprian, Bolocan, Diana, Crnkovich, James, Croitoru, Ioana, Durai, Krishna, Midler, Chase, Miron, Calin, Molnar, David, Moon, Brad, Ostarcevic, Bruno, Peltea, Alberto, Rosenberg, Matt, Sandu, Catalin, Saputkin, Arthur, Shah, Sagar, Stan, Daniel, Szocs, Ernest, Wan, Shengye, Whitman, Spencer, Krasser, Sven, Saxe, Joshua

Evaluates whether tool-using and RAG agents execute malicious instructions hidden inside retrieved emails, web pages, or documents.

1. Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models — published result table ↗

   Jingwei Yi, Yueqi Xie, Bin Zhu, Emre Kiciman, Guangzhong Sun, Xing Xie et al. · 2023

   All 7 authors

   Jingwei Yi, Yueqi Xie, Bin Zhu, Emre Kiciman, Guangzhong Sun, Xing Xie, Fangzhao Wu

Evaluates safety and prompt-injection defenses specifically in multi-step tool-calling agents (fraud, cybercrime, data exfiltration).

1. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents — published result table ↗

   Edoardo Debenedetti, Jie Zhang, Mislav Balunović, Luca Beurer-Kellner, Marc Fischer, Florian Tramèr · 2024

Evaluates visual jailbreaks where harmful intent is split across benign text and typography/imagery.

1. MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models — published result table ↗

   Liu, Xin, Zhu, Yichen, Gu, Jindong, Lan, Yunshi, Yang, Chao, Qiao, Yu · 2023

Ambiguous and disambiguated questions test answer accuracy and social bias across protected social categories. Accuracy, signed bias, and name-based variants answer different questions.

1. BBQ: A Hand-Built Bias Benchmark for Question Answering ↗

   Parrish, Alicia, Chen, Angelica, Nangia, Nikita, Padmakumar, Vishakh, Phang, Jason, Thompson, Jana et al. · 2021

   All 8 authors

   Parrish, Alicia, Chen, Angelica, Nangia, Nikita, Padmakumar, Vishakh, Phang, Jason, Thompson, Jana, Htut, Phu Mon, Bowman, Samuel R.

2. BBQ: A Hand-Built Bias Benchmark for Question Answering — released RoBERTa and DeBERTa predictions ↗

   Parrish, Alicia, Chen, Angelica, Nangia, Nikita, Padmakumar, Vishakh, Phang, Jason, Thompson, Jana et al. · 2021

   All 8 authors

   Parrish, Alicia, Chen, Angelica, Nangia, Nikita, Padmakumar, Vishakh, Phang, Jason, Thompson, Jana, Htut, Phu Mon, Bowman, Samuel R.

Measures explicit and implicit hate speech generation and detection.

1. RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models — published result table ↗

   Gehman, Samuel, Gururangan, Suchin, Sap, Maarten, Choi, Yejin, Smith, Noah A. · 2020

Foundational safety alignment and safe-RLHF evaluation datasets.

1. Do-Not-Answer: Evaluating Safeguards in LLMs ↗

   Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, Timothy Baldwin · 2024

134 choose-your-own-adventure text games (500k+ scenarios) measuring whether agents trade off ethical behavior (deception, manipulation, harm) to maximize game rewards.

1. Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark — published result table ↗

   Alexander Pan, Jun Shern Chan, Andy Zou, Nathaniel Li, Steven Basart, Thomas Woodside et al. · 2023

   All 10 authors

   Alexander Pan, Jun Shern Chan, Andy Zou, Nathaniel Li, Steven Basart, Thomas Woodside, Jonathan Ng, Hanlin Zhang, Scott Emmons, Dan Hendrycks

Evaluates whether frontier models strategically underperform on capability evaluations ("sandbagging"), disable oversight mechanisms, or attempt self-exfiltration when goal-conflict is induced.

1. Frontier Models are Capable of In-context Scheming — published result table ↗

   Meinke, Alexander, Schoen, Bronson, Scheurer, Jérémy, Balesni, Mikita, Shah, Rusheb, Hobbhahn, Marius · 2024

Evaluates anthropomorphic manipulation, user retention hooks, brand bias, and sycophantic dark patterns in chatbots.

1. DarkBench: Benchmarking Dark Patterns in Large Language Models — publisher leaderboard data — published result table ↗

   Esben Kran, Hieu Minh Nguyen, Akash Kundu, Sami Jawhar, Jinsuk Park, Mateusz Maria Jurewicz · 2025

Text-rich visual tasks test recognition and understanding of writing inside images.

1. Primary benchmark documentation or model evaluation report ↗

Multi-source document parsing benchmark covering PDF textbooks, academic papers, financial reports, newspapers, handwritten notes, and complex formulas/tables.

1. OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations — published result table ↗

   Linke Ouyang, Yuan Qu, Hongbin Zhou, Jiawei Zhu, Rui Zhang, Qunshu Lin et al. · 2024

   All 20 authors

   Linke Ouyang, Yuan Qu, Hongbin Zhou, Jiawei Zhu, Rui Zhang, Qunshu Lin, Bin Wang, Zhiyuan Zhao, Man Jiang, Xiaomeng Zhao, Jin Shi, Fan Wu, Pei Chu, Minghao Liu, Zhenxiang Li, Chao Xu, Bo Zhang, Botian Shi, Zhongying Tu, Conghui He

1,500 elite vision-indispensable multimodal samples filtered so models cannot guess the answer from text priors alone without looking at the image.

1. Are We on the Right Way for Evaluating Large Vision-Language Models? — published result table ↗

   Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen et al. · 2024

   All 11 authors

   Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, Feng Zhao

~3,000 multimodal questions across 20 fine-grained perception and reasoning dimensions evaluated with circular option permutation.

1. MMBench: Is Your Multi-modal Model an All-around Player? — published result table ↗

   Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao et al. · 2023

   All 12 authors

   Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, Dahua Lin

Evaluates integrated vision-language capabilities combining recognition, OCR, spatial awareness, knowledge, math, and multi-image sequential reasoning.

1. MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities — published result table ↗

   Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu et al. · 2023

   All 8 authors

   Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, Lijuan Wang

14 classic computer-vision perception tasks (relative depth, visual correspondence, forensic detection, multi-view reasoning, jigsaw) solvable by humans "in a blink" (<1s) that stump VLMs.

1. BLINK: Multimodal Large Language Models Can See but Not Perceive — published result table ↗

   Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin et al. · 2024

   All 10 authors

   Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, Ranjay Krishna

700+ real-world spatial and physical scene understanding questions captured from vehicles and everyday environments.

1. Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models — published result table ↗

   Deitke, Matt, Clark, Christopher, Lee, Sangho, Tripathi, Rohun, Yang, Yue, Park, Jae Sung et al. · 2024

   All 50 authors

   Deitke, Matt, Clark, Christopher, Lee, Sangho, Tripathi, Rohun, Yang, Yue, Park, Jae Sung, Salehi, Mohammadreza, Muennighoff, Niklas, Lo, Kyle, Soldaini, Luca, Lu, Jiasen, Anderson, Taira, Bransom, Erin, Ehsani, Kiana, Ngo, Huong, Chen, YenSung, Patel, Ajay, Yatskar, Mark, Callison-Burch, Chris, Head, Andrew, Hendrix, Rose, Bastani, Favyen, VanderBilt, Eli, Lambert, Nathan, Chou, Yvonne, Chheda, Arnavi, Sparks, Jenna, Skjonsberg, Sam, Schmitz, Michael, Sarnat, Aaron, Bischoff, Byron, Walsh, Pete, Newell, Chris, Wolters, Piper, Gupta, Tanmay, Zeng, Kuo-Hao, Borchardt, Jon, Groeneveld, Dirk, Nam, Crystal, Lebrecht, Sophie, Wittlif, Caitlin, Schoenick, Carissa, Michel, Oscar, Krishna, Ranjay, Weihs, Luca, Smith, Noah A., Hajishirzi, Hannaneh, Girshick, Ross, Farhadi, Ali, Kembhavi, Aniruddha

2. Grok-1.5 Vision Preview ↗

   xAI · 2024

5,000+ grade-school science diagrams with 15,000+ questions testing food webs, life cycles, and anatomical/geological flowcharts.

1. Are We on the Right Way for Evaluating Large Vision-Language Models? — published result table ↗

   Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen et al. · 2024

   All 11 authors

   Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, Feng Zhao

32.7k questions over bar, line, and pie charts requiring visual data lookup and arithmetic calculation.

1. The Llama 3 Herd of Models — published result table ↗

   Grattafiori, Aaron, Dubey, Abhimanyu, Jauhri, Abhinav, Pandey, Abhinav, Kadian, Abhishek, Al-Dahle, Ahmad et al. · 2024

   All 561 authors

   Grattafiori, Aaron, Dubey, Abhimanyu, Jauhri, Abhinav, Pandey, Abhinav, Kadian, Abhishek, Al-Dahle, Ahmad, Letman, Aiesha, Mathur, Akhil, Schelten, Alan, Vaughan, Alex, Yang, Amy, Fan, Angela, Goyal, Anirudh, Hartshorn, Anthony, Yang, Aobo, Mitra, Archi, Sravankumar, Archie, Korenev, Artem, Hinsvark, Arthur, Rao, Arun, Zhang, Aston, Rodriguez, Aurelien, Gregerson, Austen, Spataru, Ava, Roziere, Baptiste, Biron, Bethany, Tang, Binh, Chern, Bobbie, Caucheteux, Charlotte, Nayak, Chaya, Bi, Chloe, Marra, Chris, McConnell, Chris, Keller, Christian, Touret, Christophe, Wu, Chunyang, Wong, Corinne, Ferrer, Cristian Canton, Nikolaidis, Cyrus, Allonsius, Damien, Song, Daniel, Pintz, Danielle, Livshits, Danny, Wyatt, Danny, Esiobu, David, Choudhary, Dhruv, Mahajan, Dhruv, Garcia-Olano, Diego, Perino, Diego, Hupkes, Dieuwke, Lakomkin, Egor, AlBadawy, Ehab, Lobanova, Elina, Dinan, Emily, Smith, Eric Michael, Radenovic, Filip, Guzmán, Francisco, Zhang, Frank, Synnaeve, Gabriel, Lee, Gabrielle, Anderson, Georgia Lewis, Thattai, Govind, Nail, Graeme, Mialon, Gregoire, Pang, Guan, Cucurell, Guillem, Nguyen, Hailey, Korevaar, Hannah, Xu, Hu, Touvron, Hugo, Zarov, Iliyan, Ibarra, Imanol Arrieta, Kloumann, Isabel, Misra, Ishan, Evtimov, Ivan, Zhang, Jack, Copet, Jade, Lee, Jaewon, Geffert, Jan, Vranes, Jana, Park, Jason, Mahadeokar, Jay, Shah, Jeet, van der Linde, Jelmer, Billock, Jennifer, Hong, Jenny, Lee, Jenya, Fu, Jeremy, Chi, Jianfeng, Huang, Jianyu, Liu, Jiawen, Wang, Jie, Yu, Jiecao, Bitton, Joanna, Spisak, Joe, Park, Jongsoo, Rocca, Joseph, Johnstun, Joshua, Saxe, Joshua, Jia, Junteng, Alwala, Kalyan Vasuden, Prasad, Karthik, Upasani, Kartikeya, Plawiak, Kate, Li, Ke, Heafield, Kenneth, Stone, Kevin, El-Arini, Khalid, Iyer, Krithika, Malik, Kshitiz, Chiu, Kuenley, Bhalla, Kunal, Lakhotia, Kushal, Rantala-Yeary, Lauren, van der Maaten, Laurens, Chen, Lawrence, Tan, Liang, Jenkins, Liz, Martin, Louis, Madaan, Lovish, Malo, Lubo, Blecher, Lukas, Landzaat, Lukas, de Oliveira, Luke, Muzzi, Madeline, Pasupuleti, Mahesh, Singh, Mannat, Paluri, Manohar, Kardas, Marcin, Tsimpoukelli, Maria, Oldham, Mathew, Rita, Mathieu, Pavlova, Maya, Kambadur, Melanie, Lewis, Mike, Si, Min, Singh, Mitesh Kumar, Hassan, Mona, Goyal, Naman, Torabi, Narjes, Bashlykov, Nikolay, Bogoychev, Nikolay, Chatterji, Niladri, Zhang, Ning, Duchenne, Olivier, Çelebi, Onur, Alrassy, Patrick, Zhang, Pengchuan, Li, Pengwei, Vasic, Petar, Weng, Peter, Bhargava, Prajjwal, Dubal, Pratik, Krishnan, Praveen, Koura, Punit Singh, Xu, Puxin, He, Qing, Dong, Qingxiao, Srinivasan, Ragavan, Ganapathy, Raj, Calderer, Ramon, Cabral, Ricardo Silveira, Stojnic, Robert, Raileanu, Roberta, Maheswari, Rohan, Girdhar, Rohit, Patel, Rohit, Sauvestre, Romain, Polidoro, Ronnie, Sumbaly, Roshan, Taylor, Ross, Silva, Ruan, Hou, Rui, Wang, Rui, Hosseini, Saghar, Chennabasappa, Sahana, Singh, Sanjay, Bell, Sean, Kim, Seohyun Sonia, Edunov, Sergey, Nie, Shaoliang, Narang, Sharan, Raparthy, Sharath, Shen, Sheng, Wan, Shengye, Bhosale, Shruti, Zhang, Shun, Vandenhende, Simon, Batra, Soumya, Whitman, Spencer, Sootla, Sten, Collot, Stephane, Gururangan, Suchin, Borodinsky, Sydney, Herman, Tamar, Fowler, Tara, Sheasha, Tarek, Georgiou, Thomas, Scialom, Thomas, Speckbacher, Tobias, Mihaylov, Todor, Xiao, Tong, Karn, Ujjwal, Goswami, Vedanuj, Gupta, Vibhor, Ramanathan, Vignesh, Kerkez, Viktor, Gonguet, Vincent, Do, Virginie, Vogeti, Vish, Albiero, Vítor, Petrovic, Vladan, Chu, Weiwei, Xiong, Wenhan, Fu, Wenyin, Meers, Whitney, Martinet, Xavier, Wang, Xiaodong, Wang, Xiaofang, Tan, Xiaoqing Ellen, Xia, Xide, Xie, Xinfeng, Jia, Xuchao, Wang, Xuewei, Goldschlag, Yaelle, Gaur, Yashesh, Babaei, Yasmine, Wen, Yi, Song, Yiwen, Zhang, Yuchen, Li, Yue, Mao, Yuning, Coudert, Zacharie Delpierre, Yan, Zheng, Chen, Zhengxing, Papakipos, Zoe, Singh, Aaditya, Srivastava, Aayushi, Jain, Abha, Kelsey, Adam, Shajnfeld, Adam, Gangidi, Adithya, Victoria, Adolfo, Goldstand, Ahuva, Menon, Ajay, Sharma, Ajay, Boesenberg, Alex, Baevski, Alexei, Feinstein, Allie, Kallet, Amanda, Sangani, Amit, Teo, Amos, Yunus, Anam, Lupu, Andrei, Alvarado, Andres, Caples, Andrew, Gu, Andrew, Ho, Andrew, Poulton, Andrew, Ryan, Andrew, Ramchandani, Ankit, Dong, Annie, Franco, Annie, Goyal, Anuj, Saraf, Aparajita, Chowdhury, Arkabandhu, Gabriel, Ashley, Bharambe, Ashwin, Eisenman, Assaf, Yazdan, Azadeh, James, Beau, Maurer, Ben, Leonhardi, Benjamin, Huang, Bernie, Loyd, Beth, De Paola, Beto, Paranjape, Bhargavi, Liu, Bing, Wu, Bo, Ni, Boyu, Hancock, Braden, Wasti, Bram, Spence, Brandon, Stojkovic, Brani, Gamido, Brian, Montalvo, Britt, Parker, Carl, Burton, Carly, Mejia, Catalina, Liu, Ce, Wang, Changhan, Kim, Changkyu, Zhou, Chao, Hu, Chester, Chu, Ching-Hsiang, Cai, Chris, Tindal, Chris, Feichtenhofer, Christoph, Gao, Cynthia, Civin, Damon, Beaty, Dana, Kreymer, Daniel, Li, Daniel, Adkins, David, Xu, David, Testuggine, Davide, David, Delia, Parikh, Devi, Liskovich, Diana, Foss, Didem, Wang, Dingkang, Le, Duc, Holland, Dustin, Dowling, Edward, Jamil, Eissa, Montgomery, Elaine, Presani, Eleonora, Hahn, Emily, Wood, Emily, Le, Eric-Tuan, Brinkman, Erik, Arcaute, Esteban, Dunbar, Evan, Smothers, Evan, Sun, Fei, Kreuk, Felix, Tian, Feng, Kokkinos, Filippos, Ozgenel, Firat, Caggioni, Francesco, Kanayet, Frank, Seide, Frank, Florez, Gabriela Medina, Schwarz, Gabriella, Badeer, Gada, Swee, Georgia, Halpern, Gil, Herman, Grant, Sizov, Grigory, Guangyi, Zhang, Lakshminarayanan, Guna, Inan, Hakan, Shojanazeri, Hamid, Zou, Han, Wang, Hannah, Zha, Hanwen, Habeeb, Haroun, Rudolph, Harrison, Suk, Helen, Aspegren, Henry, Goldman, Hunter, Zhan, Hongyuan, Damlaj, Ibrahim, Molybog, Igor, Tufanov, Igor, Leontiadis, Ilias, Veliche, Irina-Elena, Gat, Itai, Weissman, Jake, Geboski, James, Kohli, James, Lam, Janice, Asher, Japhet, Gaya, Jean-Baptiste, Marcus, Jeff, Tang, Jeff, Chan, Jennifer, Zhen, Jenny, Reizenstein, Jeremy, Teboul, Jeremy, Zhong, Jessica, Jin, Jian, Yang, Jingyi, Cummings, Joe, Carvill, Jon, Shepard, Jon, McPhie, Jonathan, Torres, Jonathan, Ginsburg, Josh, Wang, Junjie, Wu, Kai, U, Kam Hou, Saxena, Karan, Khandelwal, Kartikay, Zand, Katayoun, Matosich, Kathy, Veeraraghavan, Kaushik, Michelena, Kelly, Li, Keqian, Jagadeesh, Kiran, Huang, Kun, Chawla, Kunal, Huang, Kyle, Chen, Lailin, Garg, Lakshya, A, Lavender, Silva, Leandro, Bell, Lee, Zhang, Lei, Guo, Liangpeng, Yu, Licheng, Moshkovich, Liron, Wehrstedt, Luca, Khabsa, Madian, Avalani, Manav, Bhatt, Manish, Mankus, Martynas, Hasson, Matan, Lennie, Matthew, Reso, Matthias, Groshev, Maxim, Naumov, Maxim, Lathi, Maya, Keneally, Meghan, Liu, Miao, Seltzer, Michael L., Valko, Michal, Restrepo, Michelle, Patel, Mihir, Vyatskov, Mik, Samvelyan, Mikayel, Clark, Mike, Macey, Mike, Wang, Mike, Hermoso, Miquel Jubert, Metanat, Mo, Rastegari, Mohammad, Bansal, Munish, Santhanam, Nandhini, Parks, Natascha, White, Natasha, Bawa, Navyata, Singhal, Nayan, Egebo, Nick, Usunier, Nicolas, Mehta, Nikhil, Laptev, Nikolay Pavlovich, Dong, Ning, Cheng, Norman, Chernoguz, Oleg, Hart, Olivia, Salpekar, Omkar, Kalinli, Ozlem, Kent, Parkin, Parekh, Parth, Saab, Paul, Balaji, Pavan, Rittner, Pedro, Bontrager, Philip, Roux, Pierre, Dollar, Piotr, Zvyagina, Polina, Ratanchandani, Prashant, Yuvraj, Pritish, Liang, Qian, Alao, Rachad, Rodriguez, Rachel, Ayub, Rafi, Murthy, Raghotham, Nayani, Raghu, Mitra, Rahul, Parthasarathy, Rangaprabhu, Li, Raymond, Hogan, Rebekkah, Battey, Robin, Wang, Rocky, Howes, Russ, Rinott, Ruty, Mehta, Sachin, Siby, Sachin, Bondu, Sai Jayesh, Datta, Samyak, Chugh, Sara, Hunt, Sara, Dhillon, Sargun, Sidorov, Sasha, Pan, Satadru, Mahajan, Saurabh, Verma, Saurabh, Yamamoto, Seiji, Ramaswamy, Sharadh, Lindsay, Shaun, Lindsay, Shaun, Feng, Sheng, Lin, Shenghao, Zha, Shengxin Cindy, Patil, Shishir, Shankar, Shiva, Zhang, Shuqiang, Zhang, Shuqiang, Wang, Sinong, Agarwal, Sneha, Sajuyigbe, Soji, Chintala, Soumith, Max, Stephanie, Chen, Stephen, Kehoe, Steve, Satterfield, Steve, Govindaprasad, Sudarshan, Gupta, Sumit, Deng, Summer, Cho, Sungmin, Virk, Sunny, Subramanian, Suraj, Choudhury, Sy, Goldman, Sydney, Remez, Tal, Glaser, Tamar, Best, Tamara, Koehler, Thilo, Robinson, Thomas, Li, Tianhe, Zhang, Tianjun, Matthews, Tim, Chou, Timothy, Shaked, Tzook, Vontimitta, Varun, Ajayi, Victoria, Montanez, Victoria, Mohan, Vijai, Kumar, Vinay Satish, Mangla, Vishal, Ionescu, Vlad, Poenaru, Vlad, Mihailescu, Vlad Tiberiu, Ivanov, Vladimir, Li, Wei, Wang, Wenchen, Jiang, Wenwen, Bouaziz, Wes, Constable, Will, Tang, Xiaocheng, Wu, Xiaojian, Wang, Xiaolan, Wu, Xilun, Gao, Xinbo, Kleinman, Yaniv, Chen, Yanjun, Hu, Ye, Jia, Ye, Qi, Ye, Li, Yenda, Zhang, Yilin, Zhang, Ying, Adi, Yossi, Nam, Youngjin, Yu, Wang, Zhao, Yu, Hao, Yuchen, Qian, Yundi, Li, Yunlu, He, Yuzi, Rait, Zach, DeVito, Zachary, Rosnbrick, Zef, Wen, Zhaoduo, Yang, Zhenyu, Zhao, Zhiwei, Ma, Zhiyu

30,000 questions over 5,485 complex data-visualization infographics combining text, graphics, layout, and numerical charts.

1. InfographicVQA — published result table ↗

   Minesh Mathew, Viraj Bagal, Rubèn Pérez Tito, Dimosthenis Karatzas, Ernest Valveny, C. V Jawahar · 2021

45,336 questions on 28,408 natural images requiring reading and reasoning about in-the-wild street signs, labels, and clocks.

1. The Llama 3 Herd of Models — published result table ↗

   Grattafiori, Aaron, Dubey, Abhimanyu, Jauhri, Abhinav, Pandey, Abhinav, Kadian, Abhishek, Al-Dahle, Ahmad et al. · 2024

   All 561 authors

   Grattafiori, Aaron, Dubey, Abhimanyu, Jauhri, Abhinav, Pandey, Abhinav, Kadian, Abhishek, Al-Dahle, Ahmad, Letman, Aiesha, Mathur, Akhil, Schelten, Alan, Vaughan, Alex, Yang, Amy, Fan, Angela, Goyal, Anirudh, Hartshorn, Anthony, Yang, Aobo, Mitra, Archi, Sravankumar, Archie, Korenev, Artem, Hinsvark, Arthur, Rao, Arun, Zhang, Aston, Rodriguez, Aurelien, Gregerson, Austen, Spataru, Ava, Roziere, Baptiste, Biron, Bethany, Tang, Binh, Chern, Bobbie, Caucheteux, Charlotte, Nayak, Chaya, Bi, Chloe, Marra, Chris, McConnell, Chris, Keller, Christian, Touret, Christophe, Wu, Chunyang, Wong, Corinne, Ferrer, Cristian Canton, Nikolaidis, Cyrus, Allonsius, Damien, Song, Daniel, Pintz, Danielle, Livshits, Danny, Wyatt, Danny, Esiobu, David, Choudhary, Dhruv, Mahajan, Dhruv, Garcia-Olano, Diego, Perino, Diego, Hupkes, Dieuwke, Lakomkin, Egor, AlBadawy, Ehab, Lobanova, Elina, Dinan, Emily, Smith, Eric Michael, Radenovic, Filip, Guzmán, Francisco, Zhang, Frank, Synnaeve, Gabriel, Lee, Gabrielle, Anderson, Georgia Lewis, Thattai, Govind, Nail, Graeme, Mialon, Gregoire, Pang, Guan, Cucurell, Guillem, Nguyen, Hailey, Korevaar, Hannah, Xu, Hu, Touvron, Hugo, Zarov, Iliyan, Ibarra, Imanol Arrieta, Kloumann, Isabel, Misra, Ishan, Evtimov, Ivan, Zhang, Jack, Copet, Jade, Lee, Jaewon, Geffert, Jan, Vranes, Jana, Park, Jason, Mahadeokar, Jay, Shah, Jeet, van der Linde, Jelmer, Billock, Jennifer, Hong, Jenny, Lee, Jenya, Fu, Jeremy, Chi, Jianfeng, Huang, Jianyu, Liu, Jiawen, Wang, Jie, Yu, Jiecao, Bitton, Joanna, Spisak, Joe, Park, Jongsoo, Rocca, Joseph, Johnstun, Joshua, Saxe, Joshua, Jia, Junteng, Alwala, Kalyan Vasuden, Prasad, Karthik, Upasani, Kartikeya, Plawiak, Kate, Li, Ke, Heafield, Kenneth, Stone, Kevin, El-Arini, Khalid, Iyer, Krithika, Malik, Kshitiz, Chiu, Kuenley, Bhalla, Kunal, Lakhotia, Kushal, Rantala-Yeary, Lauren, van der Maaten, Laurens, Chen, Lawrence, Tan, Liang, Jenkins, Liz, Martin, Louis, Madaan, Lovish, Malo, Lubo, Blecher, Lukas, Landzaat, Lukas, de Oliveira, Luke, Muzzi, Madeline, Pasupuleti, Mahesh, Singh, Mannat, Paluri, Manohar, Kardas, Marcin, Tsimpoukelli, Maria, Oldham, Mathew, Rita, Mathieu, Pavlova, Maya, Kambadur, Melanie, Lewis, Mike, Si, Min, Singh, Mitesh Kumar, Hassan, Mona, Goyal, Naman, Torabi, Narjes, Bashlykov, Nikolay, Bogoychev, Nikolay, Chatterji, Niladri, Zhang, Ning, Duchenne, Olivier, Çelebi, Onur, Alrassy, Patrick, Zhang, Pengchuan, Li, Pengwei, Vasic, Petar, Weng, Peter, Bhargava, Prajjwal, Dubal, Pratik, Krishnan, Praveen, Koura, Punit Singh, Xu, Puxin, He, Qing, Dong, Qingxiao, Srinivasan, Ragavan, Ganapathy, Raj, Calderer, Ramon, Cabral, Ricardo Silveira, Stojnic, Robert, Raileanu, Roberta, Maheswari, Rohan, Girdhar, Rohit, Patel, Rohit, Sauvestre, Romain, Polidoro, Ronnie, Sumbaly, Roshan, Taylor, Ross, Silva, Ruan, Hou, Rui, Wang, Rui, Hosseini, Saghar, Chennabasappa, Sahana, Singh, Sanjay, Bell, Sean, Kim, Seohyun Sonia, Edunov, Sergey, Nie, Shaoliang, Narang, Sharan, Raparthy, Sharath, Shen, Sheng, Wan, Shengye, Bhosale, Shruti, Zhang, Shun, Vandenhende, Simon, Batra, Soumya, Whitman, Spencer, Sootla, Sten, Collot, Stephane, Gururangan, Suchin, Borodinsky, Sydney, Herman, Tamar, Fowler, Tara, Sheasha, Tarek, Georgiou, Thomas, Scialom, Thomas, Speckbacher, Tobias, Mihaylov, Todor, Xiao, Tong, Karn, Ujjwal, Goswami, Vedanuj, Gupta, Vibhor, Ramanathan, Vignesh, Kerkez, Viktor, Gonguet, Vincent, Do, Virginie, Vogeti, Vish, Albiero, Vítor, Petrovic, Vladan, Chu, Weiwei, Xiong, Wenhan, Fu, Wenyin, Meers, Whitney, Martinet, Xavier, Wang, Xiaodong, Wang, Xiaofang, Tan, Xiaoqing Ellen, Xia, Xide, Xie, Xinfeng, Jia, Xuchao, Wang, Xuewei, Goldschlag, Yaelle, Gaur, Yashesh, Babaei, Yasmine, Wen, Yi, Song, Yiwen, Zhang, Yuchen, Li, Yue, Mao, Yuning, Coudert, Zacharie Delpierre, Yan, Zheng, Chen, Zhengxing, Papakipos, Zoe, Singh, Aaditya, Srivastava, Aayushi, Jain, Abha, Kelsey, Adam, Shajnfeld, Adam, Gangidi, Adithya, Victoria, Adolfo, Goldstand, Ahuva, Menon, Ajay, Sharma, Ajay, Boesenberg, Alex, Baevski, Alexei, Feinstein, Allie, Kallet, Amanda, Sangani, Amit, Teo, Amos, Yunus, Anam, Lupu, Andrei, Alvarado, Andres, Caples, Andrew, Gu, Andrew, Ho, Andrew, Poulton, Andrew, Ryan, Andrew, Ramchandani, Ankit, Dong, Annie, Franco, Annie, Goyal, Anuj, Saraf, Aparajita, Chowdhury, Arkabandhu, Gabriel, Ashley, Bharambe, Ashwin, Eisenman, Assaf, Yazdan, Azadeh, James, Beau, Maurer, Ben, Leonhardi, Benjamin, Huang, Bernie, Loyd, Beth, De Paola, Beto, Paranjape, Bhargavi, Liu, Bing, Wu, Bo, Ni, Boyu, Hancock, Braden, Wasti, Bram, Spence, Brandon, Stojkovic, Brani, Gamido, Brian, Montalvo, Britt, Parker, Carl, Burton, Carly, Mejia, Catalina, Liu, Ce, Wang, Changhan, Kim, Changkyu, Zhou, Chao, Hu, Chester, Chu, Ching-Hsiang, Cai, Chris, Tindal, Chris, Feichtenhofer, Christoph, Gao, Cynthia, Civin, Damon, Beaty, Dana, Kreymer, Daniel, Li, Daniel, Adkins, David, Xu, David, Testuggine, Davide, David, Delia, Parikh, Devi, Liskovich, Diana, Foss, Didem, Wang, Dingkang, Le, Duc, Holland, Dustin, Dowling, Edward, Jamil, Eissa, Montgomery, Elaine, Presani, Eleonora, Hahn, Emily, Wood, Emily, Le, Eric-Tuan, Brinkman, Erik, Arcaute, Esteban, Dunbar, Evan, Smothers, Evan, Sun, Fei, Kreuk, Felix, Tian, Feng, Kokkinos, Filippos, Ozgenel, Firat, Caggioni, Francesco, Kanayet, Frank, Seide, Frank, Florez, Gabriela Medina, Schwarz, Gabriella, Badeer, Gada, Swee, Georgia, Halpern, Gil, Herman, Grant, Sizov, Grigory, Guangyi, Zhang, Lakshminarayanan, Guna, Inan, Hakan, Shojanazeri, Hamid, Zou, Han, Wang, Hannah, Zha, Hanwen, Habeeb, Haroun, Rudolph, Harrison, Suk, Helen, Aspegren, Henry, Goldman, Hunter, Zhan, Hongyuan, Damlaj, Ibrahim, Molybog, Igor, Tufanov, Igor, Leontiadis, Ilias, Veliche, Irina-Elena, Gat, Itai, Weissman, Jake, Geboski, James, Kohli, James, Lam, Janice, Asher, Japhet, Gaya, Jean-Baptiste, Marcus, Jeff, Tang, Jeff, Chan, Jennifer, Zhen, Jenny, Reizenstein, Jeremy, Teboul, Jeremy, Zhong, Jessica, Jin, Jian, Yang, Jingyi, Cummings, Joe, Carvill, Jon, Shepard, Jon, McPhie, Jonathan, Torres, Jonathan, Ginsburg, Josh, Wang, Junjie, Wu, Kai, U, Kam Hou, Saxena, Karan, Khandelwal, Kartikay, Zand, Katayoun, Matosich, Kathy, Veeraraghavan, Kaushik, Michelena, Kelly, Li, Keqian, Jagadeesh, Kiran, Huang, Kun, Chawla, Kunal, Huang, Kyle, Chen, Lailin, Garg, Lakshya, A, Lavender, Silva, Leandro, Bell, Lee, Zhang, Lei, Guo, Liangpeng, Yu, Licheng, Moshkovich, Liron, Wehrstedt, Luca, Khabsa, Madian, Avalani, Manav, Bhatt, Manish, Mankus, Martynas, Hasson, Matan, Lennie, Matthew, Reso, Matthias, Groshev, Maxim, Naumov, Maxim, Lathi, Maya, Keneally, Meghan, Liu, Miao, Seltzer, Michael L., Valko, Michal, Restrepo, Michelle, Patel, Mihir, Vyatskov, Mik, Samvelyan, Mikayel, Clark, Mike, Macey, Mike, Wang, Mike, Hermoso, Miquel Jubert, Metanat, Mo, Rastegari, Mohammad, Bansal, Munish, Santhanam, Nandhini, Parks, Natascha, White, Natasha, Bawa, Navyata, Singhal, Nayan, Egebo, Nick, Usunier, Nicolas, Mehta, Nikhil, Laptev, Nikolay Pavlovich, Dong, Ning, Cheng, Norman, Chernoguz, Oleg, Hart, Olivia, Salpekar, Omkar, Kalinli, Ozlem, Kent, Parkin, Parekh, Parth, Saab, Paul, Balaji, Pavan, Rittner, Pedro, Bontrager, Philip, Roux, Pierre, Dollar, Piotr, Zvyagina, Polina, Ratanchandani, Prashant, Yuvraj, Pritish, Liang, Qian, Alao, Rachad, Rodriguez, Rachel, Ayub, Rafi, Murthy, Raghotham, Nayani, Raghu, Mitra, Rahul, Parthasarathy, Rangaprabhu, Li, Raymond, Hogan, Rebekkah, Battey, Robin, Wang, Rocky, Howes, Russ, Rinott, Ruty, Mehta, Sachin, Siby, Sachin, Bondu, Sai Jayesh, Datta, Samyak, Chugh, Sara, Hunt, Sara, Dhillon, Sargun, Sidorov, Sasha, Pan, Satadru, Mahajan, Saurabh, Verma, Saurabh, Yamamoto, Seiji, Ramaswamy, Sharadh, Lindsay, Shaun, Lindsay, Shaun, Feng, Sheng, Lin, Shenghao, Zha, Shengxin Cindy, Patil, Shishir, Shankar, Shiva, Zhang, Shuqiang, Zhang, Shuqiang, Wang, Sinong, Agarwal, Sneha, Sajuyigbe, Soji, Chintala, Soumith, Max, Stephanie, Chen, Stephen, Kehoe, Steve, Satterfield, Steve, Govindaprasad, Sudarshan, Gupta, Sumit, Deng, Summer, Cho, Sungmin, Virk, Sunny, Subramanian, Suraj, Choudhury, Sy, Goldman, Sydney, Remez, Tal, Glaser, Tamar, Best, Tamara, Koehler, Thilo, Robinson, Thomas, Li, Tianhe, Zhang, Tianjun, Matthews, Tim, Chou, Timothy, Shaked, Tzook, Vontimitta, Varun, Ajayi, Victoria, Montanez, Victoria, Mohan, Vijai, Kumar, Vinay Satish, Mangla, Vishal, Ionescu, Vlad, Poenaru, Vlad, Mihailescu, Vlad Tiberiu, Ivanov, Vladimir, Li, Wei, Wang, Wenchen, Jiang, Wenwen, Bouaziz, Wes, Constable, Will, Tang, Xiaocheng, Wu, Xiaojian, Wang, Xiaolan, Wu, Xilun, Gao, Xinbo, Kleinman, Yaniv, Chen, Yanjun, Hu, Ye, Jia, Ye, Qi, Ye, Li, Yenda, Zhang, Yilin, Zhang, Ying, Adi, Yossi, Nam, Youngjin, Yu, Wang, Zhao, Yu, Hao, Yuchen, Qian, Yundi, Li, Yunlu, He, Yuzi, Rait, Zach, DeVito, Zachary, Rosnbrick, Zef, Wen, Zhaoduo, Yang, Zhenyu, Zhao, Zhiwei, Ma, Zhiyu

Image-context reasoning benchmark specifically designed to induce visual illusions, language-prior hallucinations, and parametric overrides.

1. HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models — published result table ↗

   Guan, Tianrui, Liu, Fuxiao, Wu, Xiyang, Xian, Ruiqi, Li, Zongxia, Liu, Xiaoyu et al. · 2023

   All 12 authors

   Guan, Tianrui, Liu, Fuxiao, Wu, Xiyang, Xian, Ruiqi, Li, Zongxia, Liu, Xiaoyu, Wang, Xijun, Chen, Lichang, Huang, Furong, Yacoob, Yaser, Manocha, Dinesh, Zhou, Tianyi

Evaluates object-existence hallucination in image captioning and visual QA.

1. Evaluating Object Hallucination in Large Vision-Language Models — published result table ↗

   Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, Ji-Rong Wen · 2023

19k–24k multiple-choice questions spanning spatial image understanding, temporal video understanding, and interleaved image-text generation.

1. SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehension — published result table ↗

   Bohao Li, Yuying Ge, Yi Chen, Yixiao Ge, Ruimao Zhang, Ying Shan · 2024

14-subtask perception/cognition suite and high-resolution (2K+) real-world monitoring/autonomous-driving visual benchmark.

1. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models — published result table ↗

   Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin et al. · 2023

   All 14 authors

   Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, Rongrong Ji, Caifeng Shan, Ran He

Scale SEAL's private multimodal vision-language reasoning benchmark.

1. Scale Labs — evaluation methodology and results ↗

   Scale Labs / SEAL · 2026

Vals AI's economy-weighted multimodal composite across financial charts/certificates, visual coding, and education.

1. Vals Multimodal Index — evaluation methodology and results ↗

   Vals AI · 2026

20 dynamic video tasks specifically constructed so static single-frame shortcuts cannot solve them (action order, moving direction, counterfactuals).

1. MVBench: A Comprehensive Multi-modal Video Understanding Benchmark — published result table ↗

   Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu et al. · 2023

   All 12 authors

   Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, Yu Qiao

5,000 long-form egocentric video QA questions (3-minute clips from Ego4D) with a high temporal certificate length (>100s).

1. MVBench: A Comprehensive Multi-modal Video Understanding Benchmark — published result table ↗

   Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu et al. · 2023

   All 12 authors

   Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, Yu Qiao

Multi-task long-video understanding benchmarks evaluating needle-in-a-video retrieval, plot reasoning, and subtitle-visual alignment up to 60 minutes.

1. LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding — published result table ↗

   Haoning Wu, Dongxu Li, Bei Chen, Junnan Li · 2024

Multimodal video diagnostic benchmark probing memory, abstraction, physics, and semantics across video, audio, and text.

1. The Llama 3 Herd of Models — published result table ↗

   Grattafiori, Aaron, Dubey, Abhimanyu, Jauhri, Abhinav, Pandey, Abhinav, Kadian, Abhishek, Al-Dahle, Ahmad et al. · 2024

   All 561 authors

   Grattafiori, Aaron, Dubey, Abhimanyu, Jauhri, Abhinav, Pandey, Abhinav, Kadian, Abhishek, Al-Dahle, Ahmad, Letman, Aiesha, Mathur, Akhil, Schelten, Alan, Vaughan, Alex, Yang, Amy, Fan, Angela, Goyal, Anirudh, Hartshorn, Anthony, Yang, Aobo, Mitra, Archi, Sravankumar, Archie, Korenev, Artem, Hinsvark, Arthur, Rao, Arun, Zhang, Aston, Rodriguez, Aurelien, Gregerson, Austen, Spataru, Ava, Roziere, Baptiste, Biron, Bethany, Tang, Binh, Chern, Bobbie, Caucheteux, Charlotte, Nayak, Chaya, Bi, Chloe, Marra, Chris, McConnell, Chris, Keller, Christian, Touret, Christophe, Wu, Chunyang, Wong, Corinne, Ferrer, Cristian Canton, Nikolaidis, Cyrus, Allonsius, Damien, Song, Daniel, Pintz, Danielle, Livshits, Danny, Wyatt, Danny, Esiobu, David, Choudhary, Dhruv, Mahajan, Dhruv, Garcia-Olano, Diego, Perino, Diego, Hupkes, Dieuwke, Lakomkin, Egor, AlBadawy, Ehab, Lobanova, Elina, Dinan, Emily, Smith, Eric Michael, Radenovic, Filip, Guzmán, Francisco, Zhang, Frank, Synnaeve, Gabriel, Lee, Gabrielle, Anderson, Georgia Lewis, Thattai, Govind, Nail, Graeme, Mialon, Gregoire, Pang, Guan, Cucurell, Guillem, Nguyen, Hailey, Korevaar, Hannah, Xu, Hu, Touvron, Hugo, Zarov, Iliyan, Ibarra, Imanol Arrieta, Kloumann, Isabel, Misra, Ishan, Evtimov, Ivan, Zhang, Jack, Copet, Jade, Lee, Jaewon, Geffert, Jan, Vranes, Jana, Park, Jason, Mahadeokar, Jay, Shah, Jeet, van der Linde, Jelmer, Billock, Jennifer, Hong, Jenny, Lee, Jenya, Fu, Jeremy, Chi, Jianfeng, Huang, Jianyu, Liu, Jiawen, Wang, Jie, Yu, Jiecao, Bitton, Joanna, Spisak, Joe, Park, Jongsoo, Rocca, Joseph, Johnstun, Joshua, Saxe, Joshua, Jia, Junteng, Alwala, Kalyan Vasuden, Prasad, Karthik, Upasani, Kartikeya, Plawiak, Kate, Li, Ke, Heafield, Kenneth, Stone, Kevin, El-Arini, Khalid, Iyer, Krithika, Malik, Kshitiz, Chiu, Kuenley, Bhalla, Kunal, Lakhotia, Kushal, Rantala-Yeary, Lauren, van der Maaten, Laurens, Chen, Lawrence, Tan, Liang, Jenkins, Liz, Martin, Louis, Madaan, Lovish, Malo, Lubo, Blecher, Lukas, Landzaat, Lukas, de Oliveira, Luke, Muzzi, Madeline, Pasupuleti, Mahesh, Singh, Mannat, Paluri, Manohar, Kardas, Marcin, Tsimpoukelli, Maria, Oldham, Mathew, Rita, Mathieu, Pavlova, Maya, Kambadur, Melanie, Lewis, Mike, Si, Min, Singh, Mitesh Kumar, Hassan, Mona, Goyal, Naman, Torabi, Narjes, Bashlykov, Nikolay, Bogoychev, Nikolay, Chatterji, Niladri, Zhang, Ning, Duchenne, Olivier, Çelebi, Onur, Alrassy, Patrick, Zhang, Pengchuan, Li, Pengwei, Vasic, Petar, Weng, Peter, Bhargava, Prajjwal, Dubal, Pratik, Krishnan, Praveen, Koura, Punit Singh, Xu, Puxin, He, Qing, Dong, Qingxiao, Srinivasan, Ragavan, Ganapathy, Raj, Calderer, Ramon, Cabral, Ricardo Silveira, Stojnic, Robert, Raileanu, Roberta, Maheswari, Rohan, Girdhar, Rohit, Patel, Rohit, Sauvestre, Romain, Polidoro, Ronnie, Sumbaly, Roshan, Taylor, Ross, Silva, Ruan, Hou, Rui, Wang, Rui, Hosseini, Saghar, Chennabasappa, Sahana, Singh, Sanjay, Bell, Sean, Kim, Seohyun Sonia, Edunov, Sergey, Nie, Shaoliang, Narang, Sharan, Raparthy, Sharath, Shen, Sheng, Wan, Shengye, Bhosale, Shruti, Zhang, Shun, Vandenhende, Simon, Batra, Soumya, Whitman, Spencer, Sootla, Sten, Collot, Stephane, Gururangan, Suchin, Borodinsky, Sydney, Herman, Tamar, Fowler, Tara, Sheasha, Tarek, Georgiou, Thomas, Scialom, Thomas, Speckbacher, Tobias, Mihaylov, Todor, Xiao, Tong, Karn, Ujjwal, Goswami, Vedanuj, Gupta, Vibhor, Ramanathan, Vignesh, Kerkez, Viktor, Gonguet, Vincent, Do, Virginie, Vogeti, Vish, Albiero, Vítor, Petrovic, Vladan, Chu, Weiwei, Xiong, Wenhan, Fu, Wenyin, Meers, Whitney, Martinet, Xavier, Wang, Xiaodong, Wang, Xiaofang, Tan, Xiaoqing Ellen, Xia, Xide, Xie, Xinfeng, Jia, Xuchao, Wang, Xuewei, Goldschlag, Yaelle, Gaur, Yashesh, Babaei, Yasmine, Wen, Yi, Song, Yiwen, Zhang, Yuchen, Li, Yue, Mao, Yuning, Coudert, Zacharie Delpierre, Yan, Zheng, Chen, Zhengxing, Papakipos, Zoe, Singh, Aaditya, Srivastava, Aayushi, Jain, Abha, Kelsey, Adam, Shajnfeld, Adam, Gangidi, Adithya, Victoria, Adolfo, Goldstand, Ahuva, Menon, Ajay, Sharma, Ajay, Boesenberg, Alex, Baevski, Alexei, Feinstein, Allie, Kallet, Amanda, Sangani, Amit, Teo, Amos, Yunus, Anam, Lupu, Andrei, Alvarado, Andres, Caples, Andrew, Gu, Andrew, Ho, Andrew, Poulton, Andrew, Ryan, Andrew, Ramchandani, Ankit, Dong, Annie, Franco, Annie, Goyal, Anuj, Saraf, Aparajita, Chowdhury, Arkabandhu, Gabriel, Ashley, Bharambe, Ashwin, Eisenman, Assaf, Yazdan, Azadeh, James, Beau, Maurer, Ben, Leonhardi, Benjamin, Huang, Bernie, Loyd, Beth, De Paola, Beto, Paranjape, Bhargavi, Liu, Bing, Wu, Bo, Ni, Boyu, Hancock, Braden, Wasti, Bram, Spence, Brandon, Stojkovic, Brani, Gamido, Brian, Montalvo, Britt, Parker, Carl, Burton, Carly, Mejia, Catalina, Liu, Ce, Wang, Changhan, Kim, Changkyu, Zhou, Chao, Hu, Chester, Chu, Ching-Hsiang, Cai, Chris, Tindal, Chris, Feichtenhofer, Christoph, Gao, Cynthia, Civin, Damon, Beaty, Dana, Kreymer, Daniel, Li, Daniel, Adkins, David, Xu, David, Testuggine, Davide, David, Delia, Parikh, Devi, Liskovich, Diana, Foss, Didem, Wang, Dingkang, Le, Duc, Holland, Dustin, Dowling, Edward, Jamil, Eissa, Montgomery, Elaine, Presani, Eleonora, Hahn, Emily, Wood, Emily, Le, Eric-Tuan, Brinkman, Erik, Arcaute, Esteban, Dunbar, Evan, Smothers, Evan, Sun, Fei, Kreuk, Felix, Tian, Feng, Kokkinos, Filippos, Ozgenel, Firat, Caggioni, Francesco, Kanayet, Frank, Seide, Frank, Florez, Gabriela Medina, Schwarz, Gabriella, Badeer, Gada, Swee, Georgia, Halpern, Gil, Herman, Grant, Sizov, Grigory, Guangyi, Zhang, Lakshminarayanan, Guna, Inan, Hakan, Shojanazeri, Hamid, Zou, Han, Wang, Hannah, Zha, Hanwen, Habeeb, Haroun, Rudolph, Harrison, Suk, Helen, Aspegren, Henry, Goldman, Hunter, Zhan, Hongyuan, Damlaj, Ibrahim, Molybog, Igor, Tufanov, Igor, Leontiadis, Ilias, Veliche, Irina-Elena, Gat, Itai, Weissman, Jake, Geboski, James, Kohli, James, Lam, Janice, Asher, Japhet, Gaya, Jean-Baptiste, Marcus, Jeff, Tang, Jeff, Chan, Jennifer, Zhen, Jenny, Reizenstein, Jeremy, Teboul, Jeremy, Zhong, Jessica, Jin, Jian, Yang, Jingyi, Cummings, Joe, Carvill, Jon, Shepard, Jon, McPhie, Jonathan, Torres, Jonathan, Ginsburg, Josh, Wang, Junjie, Wu, Kai, U, Kam Hou, Saxena, Karan, Khandelwal, Kartikay, Zand, Katayoun, Matosich, Kathy, Veeraraghavan, Kaushik, Michelena, Kelly, Li, Keqian, Jagadeesh, Kiran, Huang, Kun, Chawla, Kunal, Huang, Kyle, Chen, Lailin, Garg, Lakshya, A, Lavender, Silva, Leandro, Bell, Lee, Zhang, Lei, Guo, Liangpeng, Yu, Licheng, Moshkovich, Liron, Wehrstedt, Luca, Khabsa, Madian, Avalani, Manav, Bhatt, Manish, Mankus, Martynas, Hasson, Matan, Lennie, Matthew, Reso, Matthias, Groshev, Maxim, Naumov, Maxim, Lathi, Maya, Keneally, Meghan, Liu, Miao, Seltzer, Michael L., Valko, Michal, Restrepo, Michelle, Patel, Mihir, Vyatskov, Mik, Samvelyan, Mikayel, Clark, Mike, Macey, Mike, Wang, Mike, Hermoso, Miquel Jubert, Metanat, Mo, Rastegari, Mohammad, Bansal, Munish, Santhanam, Nandhini, Parks, Natascha, White, Natasha, Bawa, Navyata, Singhal, Nayan, Egebo, Nick, Usunier, Nicolas, Mehta, Nikhil, Laptev, Nikolay Pavlovich, Dong, Ning, Cheng, Norman, Chernoguz, Oleg, Hart, Olivia, Salpekar, Omkar, Kalinli, Ozlem, Kent, Parkin, Parekh, Parth, Saab, Paul, Balaji, Pavan, Rittner, Pedro, Bontrager, Philip, Roux, Pierre, Dollar, Piotr, Zvyagina, Polina, Ratanchandani, Prashant, Yuvraj, Pritish, Liang, Qian, Alao, Rachad, Rodriguez, Rachel, Ayub, Rafi, Murthy, Raghotham, Nayani, Raghu, Mitra, Rahul, Parthasarathy, Rangaprabhu, Li, Raymond, Hogan, Rebekkah, Battey, Robin, Wang, Rocky, Howes, Russ, Rinott, Ruty, Mehta, Sachin, Siby, Sachin, Bondu, Sai Jayesh, Datta, Samyak, Chugh, Sara, Hunt, Sara, Dhillon, Sargun, Sidorov, Sasha, Pan, Satadru, Mahajan, Saurabh, Verma, Saurabh, Yamamoto, Seiji, Ramaswamy, Sharadh, Lindsay, Shaun, Lindsay, Shaun, Feng, Sheng, Lin, Shenghao, Zha, Shengxin Cindy, Patil, Shishir, Shankar, Shiva, Zhang, Shuqiang, Zhang, Shuqiang, Wang, Sinong, Agarwal, Sneha, Sajuyigbe, Soji, Chintala, Soumith, Max, Stephanie, Chen, Stephen, Kehoe, Steve, Satterfield, Steve, Govindaprasad, Sudarshan, Gupta, Sumit, Deng, Summer, Cho, Sungmin, Virk, Sunny, Subramanian, Suraj, Choudhury, Sy, Goldman, Sydney, Remez, Tal, Glaser, Tamar, Best, Tamara, Koehler, Thilo, Robinson, Thomas, Li, Tianhe, Zhang, Tianjun, Matthews, Tim, Chou, Timothy, Shaked, Tzook, Vontimitta, Varun, Ajayi, Victoria, Montanez, Victoria, Mohan, Vijai, Kumar, Vinay Satish, Mangla, Vishal, Ionescu, Vlad, Poenaru, Vlad, Mihailescu, Vlad Tiberiu, Ivanov, Vladimir, Li, Wei, Wang, Wenchen, Jiang, Wenwen, Bouaziz, Wes, Constable, Will, Tang, Xiaocheng, Wu, Xiaojian, Wang, Xiaolan, Wu, Xilun, Gao, Xinbo, Kleinman, Yaniv, Chen, Yanjun, Hu, Ye, Jia, Ye, Qi, Ye, Li, Yenda, Zhang, Yilin, Zhang, Ying, Adi, Yossi, Nam, Youngjin, Yu, Wang, Zhao, Yu, Hao, Yuchen, Qian, Yundi, Li, Yunlu, He, Yuzi, Rait, Zach, DeVito, Zachary, Rosnbrick, Zef, Wen, Zhaoduo, Yang, Zhenyu, Zhao, Zhiwei, Ma, Zhiyu

Massive Multi-Task Multimodal benchmark (32 meta-tasks) and Multi-Image understanding benchmark (MuirBench).

1. MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI — published result table ↗

   Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang et al. · 2024

   All 22 authors

   Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, Jiayi Lei, Quanfeng Lu, Runjian Chen, Peng Xu, Renrui Zhang, Haozhe Zhang, Peng Gao, Yali Wang, Yu Qiao, Ping Luo, Kaipeng Zhang, Wenqi Shao

Evaluates dense object counting and point-grounding in natural scenes.

1. Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models — published result table ↗

   Deitke, Matt, Clark, Christopher, Lee, Sangho, Tripathi, Rohun, Yang, Yue, Park, Jae Sung et al. · 2024

   All 50 authors

   Deitke, Matt, Clark, Christopher, Lee, Sangho, Tripathi, Rohun, Yang, Yue, Park, Jae Sung, Salehi, Mohammadreza, Muennighoff, Niklas, Lo, Kyle, Soldaini, Luca, Lu, Jiasen, Anderson, Taira, Bransom, Erin, Ehsani, Kiana, Ngo, Huong, Chen, YenSung, Patel, Ajay, Yatskar, Mark, Callison-Burch, Chris, Head, Andrew, Hendrix, Rose, Bastani, Favyen, VanderBilt, Eli, Lambert, Nathan, Chou, Yvonne, Chheda, Arnavi, Sparks, Jenna, Skjonsberg, Sam, Schmitz, Michael, Sarnat, Aaron, Bischoff, Byron, Walsh, Pete, Newell, Chris, Wolters, Piper, Gupta, Tanmay, Zeng, Kuo-Hao, Borchardt, Jon, Groeneveld, Dirk, Nam, Crystal, Lebrecht, Sophie, Wittlif, Caitlin, Schoenick, Carissa, Michel, Oscar, Krishna, Ranjay, Weihs, Luca, Smith, Noah A., Hajishirzi, Hannaneh, Girshick, Ross, Farhadi, Ali, Kembhavi, Aniruddha

10,000 audio clips with human-annotated natural-language questions requiring expert-level reasoning over speech, environmental sounds, and music.

1. MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark — published result table ↗

   S Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto et al. · 2024

   All 9 authors

   S Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh, Dinesh Manocha

Evaluates audio-language models across speech, natural soundscapes, music, and open-ended audio chat.

1. AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension — published result table ↗

   Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou et al. · 2024

   All 11 authors

   Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, Jingren Zhou

Evaluates end-to-end LLM-based voice assistants on spoken instruction following, noisy acoustic robustness, and paralinguistic understanding.

1. VoiceBench: Benchmarking LLM-Based Voice Assistants — published result table ↗

   Yiming Chen, Xianghu Yue, Chen Zhang, Xiaoxue Gao, Robby T. Tan, Haizhou Li · 2024

Standard automatic speech recognition (ASR) and multilingual speech transcription benchmarks across 102+ languages.

1. The Llama 3 Herd of Models — published result table ↗

   Grattafiori, Aaron, Dubey, Abhimanyu, Jauhri, Abhinav, Pandey, Abhinav, Kadian, Abhishek, Al-Dahle, Ahmad et al. · 2024

   All 561 authors

   Grattafiori, Aaron, Dubey, Abhimanyu, Jauhri, Abhinav, Pandey, Abhinav, Kadian, Abhishek, Al-Dahle, Ahmad, Letman, Aiesha, Mathur, Akhil, Schelten, Alan, Vaughan, Alex, Yang, Amy, Fan, Angela, Goyal, Anirudh, Hartshorn, Anthony, Yang, Aobo, Mitra, Archi, Sravankumar, Archie, Korenev, Artem, Hinsvark, Arthur, Rao, Arun, Zhang, Aston, Rodriguez, Aurelien, Gregerson, Austen, Spataru, Ava, Roziere, Baptiste, Biron, Bethany, Tang, Binh, Chern, Bobbie, Caucheteux, Charlotte, Nayak, Chaya, Bi, Chloe, Marra, Chris, McConnell, Chris, Keller, Christian, Touret, Christophe, Wu, Chunyang, Wong, Corinne, Ferrer, Cristian Canton, Nikolaidis, Cyrus, Allonsius, Damien, Song, Daniel, Pintz, Danielle, Livshits, Danny, Wyatt, Danny, Esiobu, David, Choudhary, Dhruv, Mahajan, Dhruv, Garcia-Olano, Diego, Perino, Diego, Hupkes, Dieuwke, Lakomkin, Egor, AlBadawy, Ehab, Lobanova, Elina, Dinan, Emily, Smith, Eric Michael, Radenovic, Filip, Guzmán, Francisco, Zhang, Frank, Synnaeve, Gabriel, Lee, Gabrielle, Anderson, Georgia Lewis, Thattai, Govind, Nail, Graeme, Mialon, Gregoire, Pang, Guan, Cucurell, Guillem, Nguyen, Hailey, Korevaar, Hannah, Xu, Hu, Touvron, Hugo, Zarov, Iliyan, Ibarra, Imanol Arrieta, Kloumann, Isabel, Misra, Ishan, Evtimov, Ivan, Zhang, Jack, Copet, Jade, Lee, Jaewon, Geffert, Jan, Vranes, Jana, Park, Jason, Mahadeokar, Jay, Shah, Jeet, van der Linde, Jelmer, Billock, Jennifer, Hong, Jenny, Lee, Jenya, Fu, Jeremy, Chi, Jianfeng, Huang, Jianyu, Liu, Jiawen, Wang, Jie, Yu, Jiecao, Bitton, Joanna, Spisak, Joe, Park, Jongsoo, Rocca, Joseph, Johnstun, Joshua, Saxe, Joshua, Jia, Junteng, Alwala, Kalyan Vasuden, Prasad, Karthik, Upasani, Kartikeya, Plawiak, Kate, Li, Ke, Heafield, Kenneth, Stone, Kevin, El-Arini, Khalid, Iyer, Krithika, Malik, Kshitiz, Chiu, Kuenley, Bhalla, Kunal, Lakhotia, Kushal, Rantala-Yeary, Lauren, van der Maaten, Laurens, Chen, Lawrence, Tan, Liang, Jenkins, Liz, Martin, Louis, Madaan, Lovish, Malo, Lubo, Blecher, Lukas, Landzaat, Lukas, de Oliveira, Luke, Muzzi, Madeline, Pasupuleti, Mahesh, Singh, Mannat, Paluri, Manohar, Kardas, Marcin, Tsimpoukelli, Maria, Oldham, Mathew, Rita, Mathieu, Pavlova, Maya, Kambadur, Melanie, Lewis, Mike, Si, Min, Singh, Mitesh Kumar, Hassan, Mona, Goyal, Naman, Torabi, Narjes, Bashlykov, Nikolay, Bogoychev, Nikolay, Chatterji, Niladri, Zhang, Ning, Duchenne, Olivier, Çelebi, Onur, Alrassy, Patrick, Zhang, Pengchuan, Li, Pengwei, Vasic, Petar, Weng, Peter, Bhargava, Prajjwal, Dubal, Pratik, Krishnan, Praveen, Koura, Punit Singh, Xu, Puxin, He, Qing, Dong, Qingxiao, Srinivasan, Ragavan, Ganapathy, Raj, Calderer, Ramon, Cabral, Ricardo Silveira, Stojnic, Robert, Raileanu, Roberta, Maheswari, Rohan, Girdhar, Rohit, Patel, Rohit, Sauvestre, Romain, Polidoro, Ronnie, Sumbaly, Roshan, Taylor, Ross, Silva, Ruan, Hou, Rui, Wang, Rui, Hosseini, Saghar, Chennabasappa, Sahana, Singh, Sanjay, Bell, Sean, Kim, Seohyun Sonia, Edunov, Sergey, Nie, Shaoliang, Narang, Sharan, Raparthy, Sharath, Shen, Sheng, Wan, Shengye, Bhosale, Shruti, Zhang, Shun, Vandenhende, Simon, Batra, Soumya, Whitman, Spencer, Sootla, Sten, Collot, Stephane, Gururangan, Suchin, Borodinsky, Sydney, Herman, Tamar, Fowler, Tara, Sheasha, Tarek, Georgiou, Thomas, Scialom, Thomas, Speckbacher, Tobias, Mihaylov, Todor, Xiao, Tong, Karn, Ujjwal, Goswami, Vedanuj, Gupta, Vibhor, Ramanathan, Vignesh, Kerkez, Viktor, Gonguet, Vincent, Do, Virginie, Vogeti, Vish, Albiero, Vítor, Petrovic, Vladan, Chu, Weiwei, Xiong, Wenhan, Fu, Wenyin, Meers, Whitney, Martinet, Xavier, Wang, Xiaodong, Wang, Xiaofang, Tan, Xiaoqing Ellen, Xia, Xide, Xie, Xinfeng, Jia, Xuchao, Wang, Xuewei, Goldschlag, Yaelle, Gaur, Yashesh, Babaei, Yasmine, Wen, Yi, Song, Yiwen, Zhang, Yuchen, Li, Yue, Mao, Yuning, Coudert, Zacharie Delpierre, Yan, Zheng, Chen, Zhengxing, Papakipos, Zoe, Singh, Aaditya, Srivastava, Aayushi, Jain, Abha, Kelsey, Adam, Shajnfeld, Adam, Gangidi, Adithya, Victoria, Adolfo, Goldstand, Ahuva, Menon, Ajay, Sharma, Ajay, Boesenberg, Alex, Baevski, Alexei, Feinstein, Allie, Kallet, Amanda, Sangani, Amit, Teo, Amos, Yunus, Anam, Lupu, Andrei, Alvarado, Andres, Caples, Andrew, Gu, Andrew, Ho, Andrew, Poulton, Andrew, Ryan, Andrew, Ramchandani, Ankit, Dong, Annie, Franco, Annie, Goyal, Anuj, Saraf, Aparajita, Chowdhury, Arkabandhu, Gabriel, Ashley, Bharambe, Ashwin, Eisenman, Assaf, Yazdan, Azadeh, James, Beau, Maurer, Ben, Leonhardi, Benjamin, Huang, Bernie, Loyd, Beth, De Paola, Beto, Paranjape, Bhargavi, Liu, Bing, Wu, Bo, Ni, Boyu, Hancock, Braden, Wasti, Bram, Spence, Brandon, Stojkovic, Brani, Gamido, Brian, Montalvo, Britt, Parker, Carl, Burton, Carly, Mejia, Catalina, Liu, Ce, Wang, Changhan, Kim, Changkyu, Zhou, Chao, Hu, Chester, Chu, Ching-Hsiang, Cai, Chris, Tindal, Chris, Feichtenhofer, Christoph, Gao, Cynthia, Civin, Damon, Beaty, Dana, Kreymer, Daniel, Li, Daniel, Adkins, David, Xu, David, Testuggine, Davide, David, Delia, Parikh, Devi, Liskovich, Diana, Foss, Didem, Wang, Dingkang, Le, Duc, Holland, Dustin, Dowling, Edward, Jamil, Eissa, Montgomery, Elaine, Presani, Eleonora, Hahn, Emily, Wood, Emily, Le, Eric-Tuan, Brinkman, Erik, Arcaute, Esteban, Dunbar, Evan, Smothers, Evan, Sun, Fei, Kreuk, Felix, Tian, Feng, Kokkinos, Filippos, Ozgenel, Firat, Caggioni, Francesco, Kanayet, Frank, Seide, Frank, Florez, Gabriela Medina, Schwarz, Gabriella, Badeer, Gada, Swee, Georgia, Halpern, Gil, Herman, Grant, Sizov, Grigory, Guangyi, Zhang, Lakshminarayanan, Guna, Inan, Hakan, Shojanazeri, Hamid, Zou, Han, Wang, Hannah, Zha, Hanwen, Habeeb, Haroun, Rudolph, Harrison, Suk, Helen, Aspegren, Henry, Goldman, Hunter, Zhan, Hongyuan, Damlaj, Ibrahim, Molybog, Igor, Tufanov, Igor, Leontiadis, Ilias, Veliche, Irina-Elena, Gat, Itai, Weissman, Jake, Geboski, James, Kohli, James, Lam, Janice, Asher, Japhet, Gaya, Jean-Baptiste, Marcus, Jeff, Tang, Jeff, Chan, Jennifer, Zhen, Jenny, Reizenstein, Jeremy, Teboul, Jeremy, Zhong, Jessica, Jin, Jian, Yang, Jingyi, Cummings, Joe, Carvill, Jon, Shepard, Jon, McPhie, Jonathan, Torres, Jonathan, Ginsburg, Josh, Wang, Junjie, Wu, Kai, U, Kam Hou, Saxena, Karan, Khandelwal, Kartikay, Zand, Katayoun, Matosich, Kathy, Veeraraghavan, Kaushik, Michelena, Kelly, Li, Keqian, Jagadeesh, Kiran, Huang, Kun, Chawla, Kunal, Huang, Kyle, Chen, Lailin, Garg, Lakshya, A, Lavender, Silva, Leandro, Bell, Lee, Zhang, Lei, Guo, Liangpeng, Yu, Licheng, Moshkovich, Liron, Wehrstedt, Luca, Khabsa, Madian, Avalani, Manav, Bhatt, Manish, Mankus, Martynas, Hasson, Matan, Lennie, Matthew, Reso, Matthias, Groshev, Maxim, Naumov, Maxim, Lathi, Maya, Keneally, Meghan, Liu, Miao, Seltzer, Michael L., Valko, Michal, Restrepo, Michelle, Patel, Mihir, Vyatskov, Mik, Samvelyan, Mikayel, Clark, Mike, Macey, Mike, Wang, Mike, Hermoso, Miquel Jubert, Metanat, Mo, Rastegari, Mohammad, Bansal, Munish, Santhanam, Nandhini, Parks, Natascha, White, Natasha, Bawa, Navyata, Singhal, Nayan, Egebo, Nick, Usunier, Nicolas, Mehta, Nikhil, Laptev, Nikolay Pavlovich, Dong, Ning, Cheng, Norman, Chernoguz, Oleg, Hart, Olivia, Salpekar, Omkar, Kalinli, Ozlem, Kent, Parkin, Parekh, Parth, Saab, Paul, Balaji, Pavan, Rittner, Pedro, Bontrager, Philip, Roux, Pierre, Dollar, Piotr, Zvyagina, Polina, Ratanchandani, Prashant, Yuvraj, Pritish, Liang, Qian, Alao, Rachad, Rodriguez, Rachel, Ayub, Rafi, Murthy, Raghotham, Nayani, Raghu, Mitra, Rahul, Parthasarathy, Rangaprabhu, Li, Raymond, Hogan, Rebekkah, Battey, Robin, Wang, Rocky, Howes, Russ, Rinott, Ruty, Mehta, Sachin, Siby, Sachin, Bondu, Sai Jayesh, Datta, Samyak, Chugh, Sara, Hunt, Sara, Dhillon, Sargun, Sidorov, Sasha, Pan, Satadru, Mahajan, Saurabh, Verma, Saurabh, Yamamoto, Seiji, Ramaswamy, Sharadh, Lindsay, Shaun, Lindsay, Shaun, Feng, Sheng, Lin, Shenghao, Zha, Shengxin Cindy, Patil, Shishir, Shankar, Shiva, Zhang, Shuqiang, Zhang, Shuqiang, Wang, Sinong, Agarwal, Sneha, Sajuyigbe, Soji, Chintala, Soumith, Max, Stephanie, Chen, Stephen, Kehoe, Steve, Satterfield, Steve, Govindaprasad, Sudarshan, Gupta, Sumit, Deng, Summer, Cho, Sungmin, Virk, Sunny, Subramanian, Suraj, Choudhury, Sy, Goldman, Sydney, Remez, Tal, Glaser, Tamar, Best, Tamara, Koehler, Thilo, Robinson, Thomas, Li, Tianhe, Zhang, Tianjun, Matthews, Tim, Chou, Timothy, Shaked, Tzook, Vontimitta, Varun, Ajayi, Victoria, Montanez, Victoria, Mohan, Vijai, Kumar, Vinay Satish, Mangla, Vishal, Ionescu, Vlad, Poenaru, Vlad, Mihailescu, Vlad Tiberiu, Ivanov, Vladimir, Li, Wei, Wang, Wenchen, Jiang, Wenwen, Bouaziz, Wes, Constable, Will, Tang, Xiaocheng, Wu, Xiaojian, Wang, Xiaolan, Wu, Xilun, Gao, Xinbo, Kleinman, Yaniv, Chen, Yanjun, Hu, Ye, Jia, Ye, Qi, Ye, Li, Yenda, Zhang, Yilin, Zhang, Ying, Adi, Yossi, Nam, Youngjin, Yu, Wang, Zhao, Yu, Hao, Yuchen, Qian, Yundi, Li, Yunlu, He, Yuzi, Rait, Zach, DeVito, Zachary, Rosnbrick, Zef, Wen, Zhaoduo, Yang, Zhenyu, Zhao, Zhiwei, Ma, Zhiyu

Dynamic instruction-tuning benchmark for universal speech and audio processing models.

1. Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks — published result table ↗

   Huang, Chien-yu, Chen, Wei-Chih, Yang, Shu-wen, Liu, Andy T., Li, Chen-An, Lin, Yu-Xiang et al. · 2024

   All 80 authors

   Huang, Chien-yu, Chen, Wei-Chih, Yang, Shu-wen, Liu, Andy T., Li, Chen-An, Lin, Yu-Xiang, Tseng, Wei-Cheng, Diwan, Anuj, Shih, Yi-Jen, Shi, Jiatong, Chen, William, Yang, Chih-Kai, Ren, Wenze, Chen, Xuanjun, Hsiao, Chi-Yuan, Peng, Puyuan, Wang, Shih-Heng, Kuan, Chun-Yi, Lu, Ke-Han, Chang, Kai-Wei, Ritter-Gutierrez, Fabian, Huang, Kuan-Po, Arora, Siddhant, Lin, You-Kuan, Chuang, Ming To, Yeo, Eunjung, Chang, Kalvin, Chien, Chung-Ming, Choi, Kwanghee, Wang, Jun-You, Hsieh, Cheng-Hsiu, Lin, Yi-Cheng, Yu, Chee-En, Chiu, I-Hsiang, Guimarães, Heitor R., Han, Jionghao, Lin, Tzu-Quan, Lin, Tzu-Yuan, Chang, Homu, Chang, Ting-Wu, Chen, Chun Wei, Chen, Shou-Jen, Chen, Yu-Hua, Cheng, Hsi-Chun, Dhawan, Kunal, Fang, Jia-Lin, Fang, Shi-Xin, Chiang, Kuan-Yu Fang, Fu, Chi An, Hsiao, Hsien-Fu, Hsu, Ching Yu, Huang, Shao-Syuan, Wei, Lee Chen, Lin, Hsi-Che, Lin, Hsuan-Hao, Lin, Hsuan-Ting, Lin, Jian-Ren, Liu, Ting-Chun, Lu, Li-Chun, Pai, Tsung-Min, Pasad, Ankita, Kuan, Shih-Yun Shan, Shon, Suwon, Tang, Yuxun, Tsai, Yun-Shao, Wei, Jui-Chiang, Wei, Tzu-Chieh, Wu, Chengxi, Wu, Dien-Ruei, Yang, Chao-Han Huck, Yang, Chieh-Chi, Yip, Jia Qi, Yuan, Shao-Xiang, Noroozi, Vahid, Chen, Zhehuai, Wu, Haibin, Livescu, Karen, Harwath, David, Watanabe, Shinji, Lee, Hung-yi

Evaluates unified multimodal text-to-image generation models on compositional prompt adherence, counting, spatial relations, and dense attribute binding.

1. GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment — published result table ↗

   Dhruba Ghosh, Hanna Hajishirzi, Ludwig Schmidt · 2023

Evaluates generative video and 3D world models on physical consistency, camera control, temporal coherence, and instruction adherence.

1. WorldScore: A Unified Evaluation Benchmark for World Generation — published result table ↗

   Haoyi Duan, Hong-Xing Yu, Sirui Chen, Li Fei-Fei, Jiajun Wu · 2025