BIG-Bench Hard

A challenging subset of BIG-bench tests varied reasoning skills beyond straightforward knowledge recall.26 Oct 2026

One row per model and compatible score measure. Mean = equal-weight arithmetic average of imported reports. Distinct configurations reported for the same model in a source table remain separate. Reports can overlap; the range is not a confidence interval. Unbundle to inspect each observation.
		Evidence	What the result shows
Gemini 1.5 Pro (May 2024) →Google	89.2% 1 report	BIG-Bench Hard[11]gemini-1.5-pro-001Released 2024-05-14	Leads this displayed comparison.
DeepSeek V3 →DeepSeek	85.2% 2 reports · range 82.9–87.5%	BIG-Bench Hard[9][3] View 2 reports 87.5% · BBH · Average · % · DeepSeek-V3 [9] 82.9% · BBH · Average · % · DeepSeek-V3 [3]	Published observations span 4.6 percentage points. Configuration and source differences contribute to this spread; it is not measured run-to-run uncertainty. #2 of 50 in this displayed comparison; 4 pp below the leader.
Gemini 1.5 Pro (Feb 2024) →Google	84% 1 report	BIG-Bench Hard[11]gemini-1.5-pro-001-feb24Released 2024-02-15	#3 of 50 in this displayed comparison; 5.2 pp below the leader.
Llama 3.1-405B →Meta	82.9% 2 reports	BIG-Bench Hard[3][9] View 2 reports 82.9% · BBH · Average · % · Llama-3.1-405B [3] 82.9% · BBH · Average · % · Llama-3.1-405B [9]	The imported reports agree at the published precision. Repeated citations may describe the same underlying evaluation. #4 of 50 in this displayed comparison; 6.3 pp below the leader.
phi-3-medium 14B →Microsoft	81.4% 1 report	BIG-Bench Hard[7]Phi-3-medium-128k-instructReleased 2024-04-23	#5 of 50 in this displayed comparison; 7.8 pp below the leader. Source notes 3-shot; CoT
Qwen2.5-72B →Qwen	79.8% 2 reports	BIG-Bench Hard[3][9] View 2 reports 79.8% · BBH · Average · % · Qwen2.5-72B [3] 79.8% · BBH · Average · % · Qwen2.5-72B [9]	The imported reports agree at the published precision. Repeated citations may describe the same underlying evaluation. #6 of 50 in this displayed comparison; 9.4 pp below the leader.
phi-3-small 7.4B →Microsoft	79.1% 1 report	BIG-Bench Hard[7]Phi-3-small-8k-instructReleased 2024-04-23	#7 of 50 in this displayed comparison; 10.1 pp below the leader. Source notes 3-shot; CoT
DeepSeek V2 (MoE-236B, May 2024) →DeepSeek	78.8% 2 reports	BIG-Bench Hard[3][9] View 2 reports 78.8% · BBH · Average · % · DeepSeek-V2 [3] 78.8% · BBH · Average · % · DeepSeek-V2 [9]	The imported reports agree at the published precision. Repeated citations may describe the same underlying evaluation. #8 of 50 in this displayed comparison; 10.4 pp below the leader.
GPT-4 (Jun 2023) →OpenAI	75.12% 1 report	BIG-Bench Hard[6]gpt-4-0613Released 2023-06-13	#9 of 50 in this displayed comparison; 14.08 pp below the leader.
phi-3-mini 3.8B →Microsoft	71.7% 1 report	BIG-Bench Hard[7]Phi-3-mini-4k-instructReleased 2024-04-23	#10 of 50 in this displayed comparison; 17.5 pp below the leader. Source notes 3-shot; CoT
Stable Beluga 2 →Stability AI	69.3% 1 report	BIG-Bench Hard[1]StableBeluga2Released 2023-07-20	#11 of 50 in this displayed comparison; 19.9 pp below the leader.
GPT-3.5 Turbo (June 2023) →OpenAI	61.59% 1 report	BIG-Bench Hard[6]gpt-3.5-turbo-0613Released 2023-06-13	#12 of 50 in this displayed comparison; 27.61 pp below the leader.
Yi-34B →01.AIYi-34B-Chat	61.55% 2 reports · range 51.4–71.7%	BIG-Bench Hard[4] View 2 reports 71.7% · BBH · Average · % · Yi-34B-Chat [4] 51.4% · BBH · Average · % · Yi-34B-Chat [4]	Published observations span 20.3 percentage points. Configuration and source differences contribute to this spread; it is not measured run-to-run uncertainty. #13 of 50 in this displayed comparison; 27.65 pp below the leader.
Phi-2 →Microsoft	59.4% 1 report	BIG-Bench Hard[7]phi-2Released 2023-12-12	#14 of 50 in this displayed comparison; 29.8 pp below the leader. Source notes 3-shot; CoT
Nemotron-4 15B →NVIDIA	58.7% 1 report	BIG-Bench Hard[5]Nemotron-4 15BReleased 2024-02-26	#15 of 50 in this displayed comparison; 30.5 pp below the leader.
Gemma 7B →Google	55.1% 2 reports	BIG-Bench Hard[2][5] View 2 reports 55.1% · BBH · Average · % · gemma-7b [2] 55.1% · BBH · Average · % · gemma-7b [5]	The imported reports agree at the published precision. Repeated citations may describe the same underlying evaluation. #16 of 50 in this displayed comparison; 34.1 pp below the leader.
Yi-34B →01.AIYi-34B	54.3% 1 report	BIG-Bench Hard[4]Yi-34BReleased 2023-11-02	#17 of 50 in this displayed comparison; 34.9 pp below the leader.
Llama 2-70B →Meta	54.25% 4 reports · range 42.4–64.9%	BIG-Bench Hard[1][4][8] View 4 reports 64.9% · BBH · Average · % · Llama-2-70b-hf [1] 58.5% · BBH · Average · % · Llama-2-70b-chat [4] 51.2% · BBH · Average · % · Llama-2-70b-hf [8] 42.4% · BBH · Average · % · Llama-2-70b-chat [4]	Published observations span 22.5 percentage points. Configuration and source differences contribute to this spread; it is not measured run-to-run uncertainty. #18 of 50 in this displayed comparison; 34.95 pp below the leader.
Qwen-14B →Qwen	52.98% 5 reports · range 49.7–55%	BIG-Bench Hard[4][1][5] View 5 reports 55% · BBH · Average · % · Qwen-14B-Chat [4] 53.4% · BBH · Average · % · Qwen-14B [1] 53.4% · BBH · Average · % · Qwen-14B [1] 53.4% · BBH · Average · % · Qwen-14B [5] 49.7% · BBH · Average · % · Qwen-14B-Chat [4]	Published observations span 5.3 percentage points. Configuration and source differences contribute to this spread; it is not measured run-to-run uncertainty. #19 of 50 in this displayed comparison; 36.22 pp below the leader.
internlm-20b →	52.5% 1 report	BIG-Bench Hard[1]internlm-20bReleased 2023-09-18	#20 of 50 in this displayed comparison; 36.7 pp below the leader.
LLaMA-65B →Meta	50.95% 2 reports · range 43.5–58.4%	BIG-Bench Hard[1][8] View 2 reports 58.4% · BBH · Average · % · LLaMA-65B [1] 43.5% · BBH · Average · % · LLaMA-65B [8]	Published observations span 14.9 percentage points. Configuration and source differences contribute to this spread; it is not measured run-to-run uncertainty. #21 of 50 in this displayed comparison; 38.25 pp below the leader.
Baichuan2-13B →Baichuan	48.86% 3 reports · range 48.78–49%	BIG-Bench Hard[1][5][6] View 3 reports 49% · BBH · Average · % · Baichuan-2-13B-Base [1] 48.8% · BBH · Average · % · Baichuan-2-13B-Base [5] 48.78% · BBH · Average · % · Baichuan-2-13B-Base [6]	Published observations span 0.22 percentage points. Configuration and source differences contribute to this spread; it is not measured run-to-run uncertainty. #22 of 50 in this displayed comparison; 40.34 pp below the leader.
Mistral 7B v0.1 →Mistral	47.8% 2 reports · range 39.5–56.1%	BIG-Bench Hard[2][5] View 2 reports 56.1% · BBH · Average · % · Mistral-7B-v0.1 [2] 39.5% · BBH · Average · % · Mistral-7B-v0.1 [5]	Published observations span 16.6 percentage points. Configuration and source differences contribute to this spread; it is not measured run-to-run uncertainty. #23 of 50 in this displayed comparison; 41.4 pp below the leader.
Qwen-7B →Qwen	45% 2 reports	BIG-Bench Hard[1] View 2 reports 45% · BBH · Average · % · Qwen-7B [1] 45% · BBH · Average · % · Qwen-7B [1]	The imported reports agree at the published precision. Repeated citations may describe the same underlying evaluation. #24 of 50 in this displayed comparison; 44.2 pp below the leader.
LLaMA-33B →Meta	44.9% 2 reports · range 39.8–50%	BIG-Bench Hard[1][8] View 2 reports 50% · BBH · Average · % · LLaMA-33B [1] 39.8% · BBH · Average · % · LLaMA-33B [8]	Published observations span 10.2 percentage points. Configuration and source differences contribute to this spread; it is not measured run-to-run uncertainty. #25 of 50 in this displayed comparison; 44.3 pp below the leader.
Llama 2-34B →Meta	44.1% 3 reports	BIG-Bench Hard[1][5][8] View 3 reports 44.1% · BBH · Average · % · Llama-2-34b [1] 44.1% · BBH · Average · % · Llama-2-34b [5] 44.1% · BBH · Average · % · Llama-2-34b [8]	The imported reports agree at the published precision. Repeated citations may describe the same underlying evaluation. #26 of 50 in this displayed comparison; 45.1 pp below the leader.
Yi 6B →01.AIYi-6B-Chat	43.45% 2 reports · range 39.7–47.2%	BIG-Bench Hard[4] View 2 reports 47.2% · BBH · Average · % · Yi-6B-Chat [4] 39.7% · BBH · Average · % · Yi-6B-Chat [4]	Published observations span 7.5 percentage points. Configuration and source differences contribute to this spread; it is not measured run-to-run uncertainty. #27 of 50 in this displayed comparison; 45.75 pp below the leader.
vicuna-13b-v1.1 →	43.04% 1 report	BIG-Bench Hard[6]vicuna-13b-v1.1Released 2023-04-12	#28 of 50 in this displayed comparison; 46.16 pp below the leader.
Baichuan 1-13B →Baichuan	43.01% 1 report	BIG-Bench Hard[6]Baichuan-13B-BaseReleased 2023-07-11	#29 of 50 in this displayed comparison; 46.19 pp below the leader.
Baichuan2-13B-Chat →	43% 2 reports · range 38.8–47.2%	BIG-Bench Hard[4] View 2 reports 47.2% · BBH · Average · % · Baichuan2-13B-Chat [4] 38.8% · BBH · Average · % · Baichuan2-13B-Chat [4]	Published observations span 8.4 percentage points. Configuration and source differences contribute to this spread; it is not measured run-to-run uncertainty. #30 of 50 in this displayed comparison; 46.2 pp below the leader.
Yi 6B →01.AIYi-6B	42.8% 1 report	BIG-Bench Hard[4]Yi-6BReleased 2023-11-02	#31 of 50 in this displayed comparison; 46.4 pp below the leader.
Llama 2-13B →Meta	42.71% 6 reports · range 32.9–58.2%	BIG-Bench Hard[4][6][2][5][8] View 6 reports 58.2% · BBH · Average · % · Llama-2-13b-chat [4] 46.98% · BBH · Average · % · Llama-2-13b [6] 39.4% · BBH · Average · % · Llama-2-13b [2] 39.4% · BBH · Average · % · Llama-2-13b [5] 39.4% · BBH · Average · % · Llama-2-13b [8] 32.9% · BBH · Average · % · Llama-2-13b-chat [4]	Published observations span 25.3 percentage points. Configuration and source differences contribute to this spread; it is not measured run-to-run uncertainty. #32 of 50 in this displayed comparison; 46.49 pp below the leader.
Baichuan 2-7B →Baichuan	41.58% 2 reports · range 41.56–41.6%	BIG-Bench Hard[1][6] View 2 reports 41.6% · BBH · Average · % · Baichuan-2-7B-Base [1] 41.56% · BBH · Average · % · Baichuan-2-7B-Base [6]	Published observations span 0.04 percentage points. Configuration and source differences contribute to this spread; it is not measured run-to-run uncertainty. #33 of 50 in this displayed comparison; 47.62 pp below the leader.
internlm-chat-20b →	39.55% 2 reports · range 36.7–42.4%	BIG-Bench Hard[4] View 2 reports 42.4% · BBH · Average · % · internlm-chat-20b [4] 36.7% · BBH · Average · % · internlm-chat-20b [4]	Published observations span 5.7 percentage points. Configuration and source differences contribute to this spread; it is not measured run-to-run uncertainty. #34 of 50 in this displayed comparison; 49.65 pp below the leader.
XVERSE-13B →	38.06% 1 report	BIG-Bench Hard[6]	#35 of 50 in this displayed comparison; 51.14 pp below the leader.
MPT-30B →MosaicML	38% 1 report	BIG-Bench Hard[1]mpt-30bReleased 2023-06-22	#36 of 50 in this displayed comparison; 51.2 pp below the leader.
LLaMA-13B →Meta	37.6% 3 reports · range 37–37.9%	BIG-Bench Hard[1][6][8] View 3 reports 37.9% · BBH · Average · % · LLaMA-13B [1] 37.89% · BBH · Average · % · LLaMA-13B [6] 37% · BBH · Average · % · LLaMA-13B [8]	Published observations span 0.9 percentage points. Configuration and source differences contribute to this spread; it is not measured run-to-run uncertainty. #37 of 50 in this displayed comparison; 51.6 pp below the leader.
Falcon-40B →Technology Innovation Institute	37.1% 1 report	BIG-Bench Hard[1]falcon-40bReleased 2023-03-15	#38 of 50 in this displayed comparison; 52.1 pp below the leader.
internlm-7b →	37% 1 report	BIG-Bench Hard[1]internlm-7bReleased 2023-07-05	#39 of 50 in this displayed comparison; 52.2 pp below the leader.
Gemma 2B →Google	35.2% 1 report	BIG-Bench Hard[2]gemma-2bReleased 2024-02-21	#40 of 50 in this displayed comparison; 54 pp below the leader. Source notes Couldn't find shot count
INTELLECT-1 →Prime Intellect,Hugging Face,Arcee AI	34.85% 1 report	BIG-Bench Hard[10]INTELLECT-1-InstructReleased 2024-11-29	#41 of 50 in this displayed comparison; 54.35 pp below the leader.
Llama 2-7B →Meta	34.79% 3 reports · range 32.6–39.16%	BIG-Bench Hard[6][2][8] View 3 reports 39.16% · BBH · Average · % · Llama-2-7b [6] 32.6% · BBH · Average · % · Llama-2-7b [2] 32.6% · BBH · Average · % · Llama-2-7b [8]	Published observations span 6.56 percentage points. Configuration and source differences contribute to this spread; it is not measured run-to-run uncertainty. #42 of 50 in this displayed comparison; 54.41 pp below the leader.
MPT-7B →MosaicML	33.93% 3 reports · range 31–35.6%	BIG-Bench Hard[1][6][8] View 3 reports 35.6% · BBH · Average · % · mpt-7b [1] 35.2% · BBH · Average · % · mpt-7b [6] 31% · BBH · Average · % · mpt-7b [8]	Published observations span 4.6 percentage points. Configuration and source differences contribute to this spread; it is not measured run-to-run uncertainty. #43 of 50 in this displayed comparison; 55.27 pp below the leader.
chatglm2-6b →Z.ai	33.69% 2 reports · range 33.68–33.7%	BIG-Bench Hard[1][6] View 2 reports 33.7% · BBH · Average · % · chatglm2-6b [1] 33.68% · BBH · Average · % · chatglm2-6b [6]	Published observations span 0.02 percentage points. Configuration and source differences contribute to this spread; it is not measured run-to-run uncertainty. #44 of 50 in this displayed comparison; 55.51 pp below the leader.
Baichuan1-7B →Baichuan	32.48% 1 report	BIG-Bench Hard[6]Baichuan-7BReleased 2023-06-01	#45 of 50 in this displayed comparison; 56.72 pp below the leader.
LLaMA-7B →Meta	32.06% 3 reports · range 30.3–33.5%	BIG-Bench Hard[1][6][8] View 3 reports 33.5% · BBH · Average · % · LLaMA-7B [1] 32.38% · BBH · Average · % · LLaMA-7B [6] 30.3% · BBH · Average · % · LLaMA-7B [8]	Published observations span 3.2 percentage points. Configuration and source differences contribute to this spread; it is not measured run-to-run uncertainty. #46 of 50 in this displayed comparison; 57.14 pp below the leader.
Chinese-Alpaca-Plus-13B →	28.94% 1 report	BIG-Bench Hard[6]	#47 of 50 in this displayed comparison; 60.26 pp below the leader.
Falcon-7B →Technology Innovation Institute	28.38% 2 reports · range 28–28.77%	BIG-Bench Hard[6][1] View 2 reports 28.77% · BBH · Average · % · falcon-7b [6] 28% · BBH · Average · % · falcon-7b [1]	Published observations span 0.77 percentage points. Configuration and source differences contribute to this spread; it is not measured run-to-run uncertainty. #48 of 50 in this displayed comparison; 60.82 pp below the leader.
Qwen-1.8B →Qwen	28.2% 2 reports	BIG-Bench Hard[1] View 2 reports 28.2% · BBH · Average · % · Qwen-1_8B [1] 28.2% · BBH · Average · % · Qwen-1_8B [1]	The imported reports agree at the published precision. Repeated citations may describe the same underlying evaluation. #49 of 50 in this displayed comparison; 61 pp below the leader.
AquilaChat2 34B →	27.2% 2 reports · range 20.1–34.3%	BIG-Bench Hard[4] View 2 reports 34.3% · BBH · Average · % [4] 20.1% · BBH · Average · % [4]	Published observations span 14.2 percentage points. Configuration and source differences contribute to this spread; it is not measured run-to-run uncertainty. #50 of 50 in this displayed comparison; 62 pp below the leader.

Sources

Qwen Technical Report ↗
Bai, Jinze, Bai, Shuai, Chu, Yunfei, Cui, Zeyu, Dang, Kai, Deng, Xiaodong et al. · 2023
All 48 authors
Bai, Jinze, Bai, Shuai, Chu, Yunfei, Cui, Zeyu, Dang, Kai, Deng, Xiaodong, Fan, Yang, Ge, Wenbin, Han, Yu, Huang, Fei, Hui, Binyuan, Ji, Luo, Li, Mei, Lin, Junyang, Lin, Runji, Liu, Dayiheng, Liu, Gao, Lu, Chengqiang, Lu, Keming, Ma, Jianxin, Men, Rui, Ren, Xingzhang, Ren, Xuancheng, Tan, Chuanqi, Tan, Sinan, Tu, Jianhong, Wang, Peng, Wang, Shijie, Wang, Wei, Wu, Shengguang, Xu, Benfeng, Xu, Jin, Yang, An, Yang, Hao, Yang, Jian, Yang, Shusheng, Yao, Yang, Yu, Bowen, Yuan, Hongyi, Yuan, Zheng, Zhang, Jianwei, Zhang, Xingxuan, Zhang, Yichang, Zhang, Zhenru, Zhou, Chang, Zhou, Jingren, Zhou, Xiaohuan, Zhu, Tianhang
Dates & evaluation details
BBH · Average · % · 2026-10-26
Epoch AI public data snapshot, CC BY 4.0. Externally collected result; original evaluation provider is linked. CSV: bbh_external.csv; score column: Average; fraction scale: 1. Model version is retained in configuration. Original source authors retain credit and any additional licensing terms.
Gemma: Open Models Based on Gemini Research and Technology ↗
Gemma Team, Mesnard, Thomas, Hardin, Cassidy, Dadashi, Robert, Bhupatiraju, Surya, Pathak, Shreya et al. · 2024
All 108 authors
Gemma Team, Mesnard, Thomas, Hardin, Cassidy, Dadashi, Robert, Bhupatiraju, Surya, Pathak, Shreya, Sifre, Laurent, Rivière, Morgane, Kale, Mihir Sanjay, Love, Juliette, Tafti, Pouya, Hussenot, Léonard, Sessa, Pier Giuseppe, Chowdhery, Aakanksha, Roberts, Adam, Barua, Aditya, Botev, Alex, Castro-Ros, Alex, Slone, Ambrose, Héliou, Amélie, Tacchetti, Andrea, Bulanova, Anna, Paterson, Antonia, Tsai, Beth, Shahriari, Bobak, Lan, Charline Le, Choquette-Choo, Christopher A., Crepy, Clément, Cer, Daniel, Ippolito, Daphne, Reid, David, Buchatskaya, Elena, Ni, Eric, Noland, Eric, Yan, Geng, Tucker, George, Muraru, George-Christian, Rozhdestvenskiy, Grigory, Michalewski, Henryk, Tenney, Ian, Grishchenko, Ivan, Austin, Jacob, Keeling, James, Labanowski, Jane, Lespiau, Jean-Baptiste, Stanway, Jeff, Brennan, Jenny, Chen, Jeremy, Ferret, Johan, Chiu, Justin, Mao-Jones, Justin, Lee, Katherine, Yu, Kathy, Millican, Katie, Sjoesund, Lars Lowe, Lee, Lisa, Dixon, Lucas, Reid, Machel, Mikuła, Maciej, Wirth, Mateo, Sharman, Michael, Chinaev, Nikolai, Thain, Nithum, Bachem, Olivier, Chang, Oscar, Wahltinez, Oscar, Bailey, Paige, Michel, Paul, Yotov, Petko, Chaabouni, Rahma, Comanescu, Ramona, Jana, Reena, Anil, Rohan, McIlroy, Ross, Liu, Ruibo, Mullins, Ryan, Smith, Samuel L, Borgeaud, Sebastian, Girgin, Sertan, Douglas, Sholto, Pandya, Shree, Shakeri, Siamak, De, Soham, Klimenko, Ted, Hennigan, Tom, Feinberg, Vlad, Stokowiec, Wojciech, Chen, Yu-hui, Ahmed, Zafarali, Gong, Zhitao, Warkentin, Tris, Peran, Ludovic, Giang, Minh, Farabet, Clément, Vinyals, Oriol, Dean, Jeff, Kavukcuoglu, Koray, Hassabis, Demis, Ghahramani, Zoubin, Eck, Douglas, Barral, Joelle, Pereira, Fernando, Collins, Eli, Joulin, Armand, Fiedel, Noah, Senter, Evan, Andreev, Alek, Kenealy, Kathleen
Dates & evaluation details
BBH · Average · % · 2026-10-26
Epoch AI public data snapshot, CC BY 4.0. Externally collected result; original evaluation provider is linked. CSV: bbh_external.csv; score column: Average; fraction scale: 1. Model version is retained in configuration. Original source authors retain credit and any additional licensing terms.
DeepSeek-V3 Technical Report ↗
DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu et al. · 2024
All 200 authors
DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, Zizheng Pan
Dates & evaluation details
BBH · Average · % · 2026-10-26
Epoch AI public data snapshot, CC BY 4.0. Externally collected result; original evaluation provider is linked. CSV: bbh_external.csv; score column: Average; fraction scale: 1. Model version is retained in configuration. Original source authors retain credit and any additional licensing terms.
Yi: Open Foundation Models by 01.AI ↗
AI, 01., :, Young, Alex, Chen, Bei, Li, Chao, Huang, Chengen et al. · 2024
All 33 authors
AI, 01., :, Young, Alex, Chen, Bei, Li, Chao, Huang, Chengen, Zhang, Ge, Zhang, Guanwei, Wang, Guoyin, Li, Heng, Zhu, Jiangcheng, Chen, Jianqun, Chang, Jing, Yu, Kaidong, Liu, Peng, Liu, Qiang, Yue, Shawn, Yang, Senbin, Yang, Shiming, Xie, Wen, Huang, Wenhao, Hu, Xiaohui, Ren, Xiaoyi, Niu, Xinyao, Nie, Pengcheng, Li, Yanpeng, Xu, Yuchi, Liu, Yudong, Wang, Yue, Cai, Yuxuan, Gu, Zhenyu, Liu, Zhiyuan, Dai, Zonghong
Dates & evaluation details
BBH · Average · % · 2026-10-26
Epoch AI public data snapshot, CC BY 4.0. Externally collected result; original evaluation provider is linked. CSV: bbh_external.csv; score column: Average; fraction scale: 1. Model version is retained in configuration. Original source authors retain credit and any additional licensing terms.
Nemotron-4 15B Technical Report ↗
Parmar, Jupinder, Prabhumoye, Shrimai, Jennings, Joseph, Patwary, Mostofa, Subramanian, Sandeep, Su, Dan et al. · 2024
All 27 authors
Parmar, Jupinder, Prabhumoye, Shrimai, Jennings, Joseph, Patwary, Mostofa, Subramanian, Sandeep, Su, Dan, Zhu, Chen, Narayanan, Deepak, Jhunjhunwala, Aastha, Dattagupta, Ayush, Jawa, Vibhu, Liu, Jiwei, Mahabaleshwarkar, Ameya, Nitski, Osvald, Brundyn, Annika, Maki, James, Martinez, Miguel, You, Jiaxuan, Kamalu, John, LeGresley, Patrick, Fridman, Denys, Casper, Jared, Aithal, Ashwath, Kuchaiev, Oleksii, Shoeybi, Mohammad, Cohen, Jonathan, Catanzaro, Bryan
Dates & evaluation details
BBH · Average · % · 2026-10-26
Epoch AI public data snapshot, CC BY 4.0. Externally collected result; original evaluation provider is linked. CSV: bbh_external.csv; score column: Average; fraction scale: 1. Model version is retained in configuration. Original source authors retain credit and any additional licensing terms.
Baichuan 2: Open Large-scale Language Models ↗
Yang, Aiyuan, Xiao, Bin, Wang, Bingning, Zhang, Borong, Bian, Ce, Yin, Chao et al. · 2023
All 55 authors
Yang, Aiyuan, Xiao, Bin, Wang, Bingning, Zhang, Borong, Bian, Ce, Yin, Chao, Lv, Chenxu, Pan, Da, Wang, Dian, Yan, Dong, Yang, Fan, Deng, Fei, Wang, Feng, Liu, Feng, Ai, Guangwei, Dong, Guosheng, Zhao, Haizhou, Xu, Hang, Sun, Haoze, Zhang, Hongda, Liu, Hui, Ji, Jiaming, Xie, Jian, Dai, JunTao, Fang, Kun, Su, Lei, Song, Liang, Liu, Lifeng, Ru, Liyun, Ma, Luyao, Wang, Mang, Liu, Mickel, Lin, MingAn, Nie, Nuolan, Guo, Peidong, Sun, Ruiyang, Zhang, Tao, Li, Tianpeng, Li, Tianyu, Cheng, Wei, Chen, Weipeng, Zeng, Xiangrong, Wang, Xiaochuan, Chen, Xiaoxi, Men, Xin, Yu, Xin, Pan, Xuehai, Shen, Yanjun, Wang, Yiding, Li, Yiyu, Jiang, Youxin, Gao, Yuchen, Zhang, Yupeng, Zhou, Zenan, Wu, Zhiying
Dates & evaluation details
BBH · Average · % · 2026-10-26
Epoch AI public data snapshot, CC BY 4.0. Externally collected result; original evaluation provider is linked. CSV: bbh_external.csv; score column: Average; fraction scale: 1. Model version is retained in configuration. Original source authors retain credit and any additional licensing terms.
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone ↗
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach et al. · 2024
All 129 authors
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matthew Dixon, Ronen Eldan, Victor Fragoso, Jianfeng Gao, Mei Gao, Min Gao, Amit Garg, Allie Del Giorno, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Wenxiang Hu, Jamie Huynh, Dan Iter, Sam Ade Jacobs, Mojan Javaheripi, Xin Jin, Nikos Karampatziakis, Piero Kauffmann, Mahoud Khademi, Dongwoo Kim, Young Jin Kim, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Yunsheng Li, Chen Liang, Lars Liden, Xihui Lin, Zeqi Lin, Ce Liu, Liyuan Liu, Mengchen Liu, Weishung Liu, Xiaodong Liu, Chong Luo, Piyush Madan, Ali Mahmoudzadeh, David Majercak, Matt Mazzola, Caio César Teodoro Mendes, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Liliang Ren, Gustavo de Rosa, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Yelong Shen, Swadheen Shukla, Xia Song, Masahiro Tanaka, Andrea Tupini, Praneetha Vaddamanu, Chunyu Wang, Guanhua Wang, Lijuan Wang, Shuohang Wang, Xin Wang, Yu Wang, Rachel Ward, Wen Wen, Philipp Witte, Haiping Wu, Xiaoxia Wu, Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu, Weijian Xu, Jilong Xue, Sonali Yadav, Fan Yang, Jianwei Yang, Yifan Yang, Ziyi Yang, Donghan Yu, Lu Yuan, Chenruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, Xiren Zhou
Dates & evaluation details
BBH · Average · % · 2026-10-26
Epoch AI public data snapshot, CC BY 4.0. Externally collected result; original evaluation provider is linked. CSV: bbh_external.csv; score column: Average; fraction scale: 1. Model version is retained in configuration. Original source authors retain credit and any additional licensing terms.
Llama 2: Open Foundation and Fine-Tuned Chat Models ↗
Touvron, Hugo, Martin, Louis, Stone, Kevin, Albert, Peter, Almahairi, Amjad, Babaei, Yasmine et al. · 2023
All 68 authors
Touvron, Hugo, Martin, Louis, Stone, Kevin, Albert, Peter, Almahairi, Amjad, Babaei, Yasmine, Bashlykov, Nikolay, Batra, Soumya, Bhargava, Prajjwal, Bhosale, Shruti, Bikel, Dan, Blecher, Lukas, Ferrer, Cristian Canton, Chen, Moya, Cucurull, Guillem, Esiobu, David, Fernandes, Jude, Fu, Jeremy, Fu, Wenyin, Fuller, Brian, Gao, Cynthia, Goswami, Vedanuj, Goyal, Naman, Hartshorn, Anthony, Hosseini, Saghar, Hou, Rui, Inan, Hakan, Kardas, Marcin, Kerkez, Viktor, Khabsa, Madian, Kloumann, Isabel, Korenev, Artem, Koura, Punit Singh, Lachaux, Marie-Anne, Lavril, Thibaut, Lee, Jenya, Liskovich, Diana, Lu, Yinghai, Mao, Yuning, Martinet, Xavier, Mihaylov, Todor, Mishra, Pushkar, Molybog, Igor, Nie, Yixin, Poulton, Andrew, Reizenstein, Jeremy, Rungta, Rashi, Saladi, Kalyan, Schelten, Alan, Silva, Ruan, Smith, Eric Michael, Subramanian, Ranjan, Tan, Xiaoqing Ellen, Tang, Binh, Taylor, Ross, Williams, Adina, Kuan, Jian Xiang, Xu, Puxin, Yan, Zheng, Zarov, Iliyan, Zhang, Yuchen, Fan, Angela, Kambadur, Melanie, Narang, Sharan, Rodriguez, Aurelien, Stojnic, Robert, Edunov, Sergey, Scialom, Thomas
Dates & evaluation details
BBH · Average · % · 2026-10-26
Epoch AI public data snapshot, CC BY 4.0. Externally collected result; original evaluation provider is linked. CSV: bbh_external.csv; score column: Average; fraction scale: 1. Model version is retained in configuration. Original source authors retain credit and any additional licensing terms.
DeepSeek-V3 Technical Report ↗
DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu et al. · 2024
All 200 authors
DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, Zizheng Pan
Dates & evaluation details
BBH · Average · % · 2026-10-26
Epoch AI public data snapshot, CC BY 4.0. Externally collected result; original evaluation provider is linked. CSV: bbh_external.csv; score column: Average; fraction scale: 1. Model version is retained in configuration. Original source authors retain credit and any additional licensing terms.
INTELLECT-1 Technical Report ↗
Jaghouar, Sami, Ong, Jack Min, Basra, Manveer, Obeid, Fares, Straube, Jannik, Keiblinger, Michael et al. · 2024
All 12 authors
Jaghouar, Sami, Ong, Jack Min, Basra, Manveer, Obeid, Fares, Straube, Jannik, Keiblinger, Michael, Bakouch, Elie, Atkins, Lucas, Panahi, Maziyar, Goddard, Charles, Ryabinin, Max, Hagemann, Johannes
Dates & evaluation details
BBH · Average · % · 2026-10-26
Epoch AI public data snapshot, CC BY 4.0. Externally collected result; original evaluation provider is linked. CSV: bbh_external.csv; score column: Average; fraction scale: 1. Model version is retained in configuration. Original source authors retain credit and any additional licensing terms.
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context ↗
Gemini Team, Google · 2024
Dates & evaluation details
BBH · Average · % · 2026-10-26
Epoch AI public data snapshot, CC BY 4.0. Externally collected result; original evaluation provider is linked. CSV: bbh_external.csv; score column: Average; fraction scale: 1. Model version is retained in configuration. Original source authors retain credit and any additional licensing terms.

TRACE THE SCORE · Per-model notes use published breakdowns where available.

Score vs. release date

Epoch AI · 88 observations

BBH · Average · % · higher is better

Dates are model release dates recorded by Epoch AI, not evaluation dates. Each point retains its published score and configuration. Separate releases are not a matched longitudinal experiment. Missing dates are omitted. Explore all reported results ↓

The dashed line traces the best observed score through the model release dates. It describes this selection of reports, with their original configurations. It does not estimate unmeasured models or costs.

View plotted values and sources (88)

BIG-Bench Hard · BBH · Average · %. Published measurements in their original score units.
Model	Score	Model release date	Configuration & source
Stable Beluga 2 →	69.3%	2023-07-20	StableBeluga2 Epoch AI · BBH · Average · % ↗
Llama 2-70B →	64.9%	2023-07-18	Llama-2-70b-hf Epoch AI · BBH · Average · % ↗
LLaMA-65B →	58.4%	2023-02-24	LLaMA-65B Epoch AI · BBH · Average · % ↗
Qwen-14B →	53.4%	2023-09-24	Qwen-14B Epoch AI · BBH · Average · % ↗
Qwen-14B →	53.4%	2023-09-24	Qwen-14B Epoch AI · BBH · Average · % ↗
internlm-20b →	52.5%	2023-09-18	internlm-20b Epoch AI · BBH · Average · % ↗
LLaMA-33B →	50%	2023-02-24	LLaMA-33B Epoch AI · BBH · Average · % ↗
Baichuan2-13B →	49%	2023-09-06	Baichuan-2-13B-Base Epoch AI · BBH · Average · % ↗
Qwen-7B →	45%	2023-09-28	Qwen-7B Epoch AI · BBH · Average · % ↗
Qwen-7B →	45%	2023-09-28	Qwen-7B Epoch AI · BBH · Average · % ↗
Llama 2-34B →	44.1%	2023-07-18	Llama-2-34b Epoch AI · BBH · Average · % ↗
Baichuan 2-7B →	41.6%	2023-09-20	Baichuan-2-7B-Base Epoch AI · BBH · Average · % ↗
MPT-30B →	38%	2023-06-22	mpt-30b Epoch AI · BBH · Average · % ↗
LLaMA-13B →	37.9%	2023-02-24	LLaMA-13B Epoch AI · BBH · Average · % ↗
Falcon-40B →	37.1%	2023-03-15	falcon-40b Epoch AI · BBH · Average · % ↗
internlm-7b →	37%	2023-07-05	internlm-7b Epoch AI · BBH · Average · % ↗
MPT-7B →	35.6%	2023-05-05	mpt-7b Epoch AI · BBH · Average · % ↗
chatglm2-6b →	33.7%	2023-06-24	chatglm2-6b Epoch AI · BBH · Average · % ↗
LLaMA-7B →	33.5%	2023-02-24	LLaMA-7B Epoch AI · BBH · Average · % ↗
Qwen-1.8B →	28.2%	2023-11-30	Qwen-1_8B Epoch AI · BBH · Average · % ↗
Qwen-1.8B →	28.2%	2023-11-30	Qwen-1_8B Epoch AI · BBH · Average · % ↗
Falcon-7B →	28%	2023-04-24	falcon-7b Epoch AI · BBH · Average · % ↗
Mistral 7B v0.1 →	56.1%	2023-09-27	Mistral-7B-v0.1 Epoch AI · BBH · Average · % ↗
Gemma 7B →	55.1%	2024-02-21	gemma-7b Epoch AI · BBH · Average · % ↗
Llama 2-13B →	39.4%	2023-07-18	Llama-2-13b Epoch AI · BBH · Average · % ↗
Gemma 2B →	35.2%	2024-02-21	gemma-2b Epoch AI · BBH · Average · % ↗
Llama 2-7B →	32.6%	2023-07-18	Llama-2-7b Epoch AI · BBH · Average · % ↗
Llama 3.1-405B →	82.9%	2024-07-23	Llama-3.1-405B Epoch AI · BBH · Average · % ↗
DeepSeek-V3 →	82.9%	2024-12-26	DeepSeek-V3 Epoch AI · BBH · Average · % ↗
Qwen2.5-72B →	79.8%	2024-09-19	Qwen2.5-72B Epoch AI · BBH · Average · % ↗
DeepSeek-V2 (MoE-236B, May 2024) →	78.8%	2024-05-07	DeepSeek-V2 Epoch AI · BBH · Average · % ↗
Yi-34B →	71.7%	2023-11-02	Yi-34B-Chat Epoch AI · BBH · Average · % ↗
Llama 2-70B →	58.5%	2023-07-18	Llama-2-70b-chat Epoch AI · BBH · Average · % ↗
Llama 2-13B →	58.2%	2023-07-18	Llama-2-13b-chat Epoch AI · BBH · Average · % ↗
Qwen-14B →	55%	2023-09-24	Qwen-14B-Chat Epoch AI · BBH · Average · % ↗
Yi-34B →	54.3%	2023-11-02	Yi-34B Epoch AI · BBH · Average · % ↗
Yi-34B →	51.4%	2023-11-02	Yi-34B-Chat Epoch AI · BBH · Average · % ↗
Qwen-14B →	49.7%	2023-09-24	Qwen-14B-Chat Epoch AI · BBH · Average · % ↗
Baichuan2-13B-Chat →	47.2%	2023-09-06	Baichuan2-13B-Chat Epoch AI · BBH · Average · % ↗
Yi 6B →	47.2%	2023-11-22	Yi-6B-Chat Epoch AI · BBH · Average · % ↗
Yi 6B →	42.8%	2023-11-02	Yi-6B Epoch AI · BBH · Average · % ↗
Llama 2-70B →	42.4%	2023-07-18	Llama-2-70b-chat Epoch AI · BBH · Average · % ↗
internlm-chat-20b →	42.4%	2023-09-17	internlm-chat-20b Epoch AI · BBH · Average · % ↗
Yi 6B →	39.7%	2023-11-22	Yi-6B-Chat Epoch AI · BBH · Average · % ↗
Baichuan2-13B-Chat →	38.8%	2023-09-06	Baichuan2-13B-Chat Epoch AI · BBH · Average · % ↗
internlm-chat-20b →	36.7%	2023-09-17	internlm-chat-20b Epoch AI · BBH · Average · % ↗
Llama 2-13B →	32.9%	2023-07-18	Llama-2-13b-chat Epoch AI · BBH · Average · % ↗
Nemotron-4 15B →	58.7%	2024-02-26	Nemotron-4 15B Epoch AI · BBH · Average · % ↗
Gemma 7B →	55.1%	2024-02-21	gemma-7b Epoch AI · BBH · Average · % ↗
Qwen-14B →	53.4%	2023-09-24	Qwen-14B Epoch AI · BBH · Average · % ↗
Baichuan2-13B →	48.8%	2023-09-06	Baichuan-2-13B-Base Epoch AI · BBH · Average · % ↗
Llama 2-34B →	44.1%	2023-07-18	Llama-2-34b Epoch AI · BBH · Average · % ↗
Mistral 7B v0.1 →	39.5%	2023-09-27	Mistral-7B-v0.1 Epoch AI · BBH · Average · % ↗
Llama 2-13B →	39.4%	2023-07-18	Llama-2-13b Epoch AI · BBH · Average · % ↗
GPT-4 (Jun 2023) →	75.12%	2023-06-13	gpt-4-0613 Epoch AI · BBH · Average · % ↗
GPT-3.5 Turbo (June 2023) →	61.59%	2023-06-13	gpt-3.5-turbo-0613 Epoch AI · BBH · Average · % ↗
Baichuan2-13B →	48.78%	2023-09-06	Baichuan-2-13B-Base Epoch AI · BBH · Average · % ↗
Llama 2-13B →	46.98%	2023-07-18	Llama-2-13b Epoch AI · BBH · Average · % ↗
vicuna-13b-v1.1 →	43.04%	2023-04-12	vicuna-13b-v1.1 Epoch AI · BBH · Average · % ↗
Baichuan 1-13B →	43.01%	2023-07-11	Baichuan-13B-Base Epoch AI · BBH · Average · % ↗
Baichuan 2-7B →	41.56%	2023-09-20	Baichuan-2-7B-Base Epoch AI · BBH · Average · % ↗
Llama 2-7B →	39.16%	2023-07-18	Llama-2-7b Epoch AI · BBH · Average · % ↗
LLaMA-13B →	37.89%	2023-02-24	LLaMA-13B Epoch AI · BBH · Average · % ↗
MPT-7B →	35.2%	2023-05-05	mpt-7b Epoch AI · BBH · Average · % ↗
chatglm2-6b →	33.68%	2023-06-24	chatglm2-6b Epoch AI · BBH · Average · % ↗
Baichuan1-7B →	32.48%	2023-06-01	Baichuan-7B Epoch AI · BBH · Average · % ↗
LLaMA-7B →	32.38%	2023-02-24	LLaMA-7B Epoch AI · BBH · Average · % ↗
Falcon-7B →	28.77%	2023-04-24	falcon-7b Epoch AI · BBH · Average · % ↗
phi-3-medium 14B →	81.4%	2024-04-23	Phi-3-medium-128k-instruct Epoch AI · BBH · Average · % ↗
phi-3-small 7.4B →	79.1%	2024-04-23	Phi-3-small-8k-instruct Epoch AI · BBH · Average · % ↗
phi-3-mini 3.8B →	71.7%	2024-04-23	Phi-3-mini-4k-instruct Epoch AI · BBH · Average · % ↗
Phi-2 →	59.4%	2023-12-12	phi-2 Epoch AI · BBH · Average · % ↗
Llama 2-70B →	51.2%	2023-07-18	Llama-2-70b-hf Epoch AI · BBH · Average · % ↗
Llama 2-34B →	44.1%	2023-07-18	Llama-2-34b Epoch AI · BBH · Average · % ↗
LLaMA-65B →	43.5%	2023-02-24	LLaMA-65B Epoch AI · BBH · Average · % ↗
LLaMA-33B →	39.8%	2023-02-24	LLaMA-33B Epoch AI · BBH · Average · % ↗
Llama 2-13B →	39.4%	2023-07-18	Llama-2-13b Epoch AI · BBH · Average · % ↗
LLaMA-13B →	37%	2023-02-24	LLaMA-13B Epoch AI · BBH · Average · % ↗
Llama 2-7B →	32.6%	2023-07-18	Llama-2-7b Epoch AI · BBH · Average · % ↗
MPT-7B →	31%	2023-05-05	mpt-7b Epoch AI · BBH · Average · % ↗
LLaMA-7B →	30.3%	2023-02-24	LLaMA-7B Epoch AI · BBH · Average · % ↗
DeepSeek-V3 →	87.5%	2024-12-26	DeepSeek-V3 Epoch AI · BBH · Average · % ↗
Llama 3.1-405B →	82.9%	2024-07-23	Llama-3.1-405B Epoch AI · BBH · Average · % ↗
Qwen2.5-72B →	79.8%	2024-09-19	Qwen2.5-72B Epoch AI · BBH · Average · % ↗
DeepSeek-V2 (MoE-236B, May 2024) →	78.8%	2024-05-07	DeepSeek-V2 Epoch AI · BBH · Average · % ↗
INTELLECT-1 →	34.85%	2024-11-29	INTELLECT-1-Instruct Epoch AI · BBH · Average · % ↗
Gemini 1.5 Pro (May 2024) →	89.2%	2024-05-14	gemini-1.5-pro-001 Epoch AI · BBH · Average · % ↗
Gemini 1.5 Pro (Feb 2024) →	84%	2024-02-15	gemini-1.5-pro-001-feb24 Epoch AI · BBH · Average · % ↗

What it measures

A challenging subset of BIG-bench tests varied reasoning skills beyond straightforward knowledge recall.

How to read the result

Inspect per-task performance and prompting sensitivity, since a single average combines different reasoning demands.

Limits and comparison traps

Direct-answer and chain-of-thought protocols differ, and public solutions may have appeared in training data.

Benchmark reference notes

23 challenging BIG-Bench tasks (boolean expressions, causal judgment, dyck languages, logical deduction, tracking shuffled objects).

Epoch data and evaluation setting

Epoch AI publishes this benchmark in its capability collection. Imported rows retain the published score column and model configuration. Original authors and evaluation providers remain credited in the result references.

Epoch attribution and score coverage

Data is attributed to Epoch AI under CC BY 4.0, with original evaluation providers credited in the linked source snapshots. Scores are imported only where a published mapping identifies the field and its scale; missing or incompatible records are omitted rather than assigned zero.

Where this fits in an evaluation suite

Ask what the task actually requires: recall, deduction, search, or a mixture. Separate closed-book reasoning from retrieval and tool-assisted problem solving.

Start with a capability hypothesis, then choose a second test with a different failure mode. Keep the checkpoint, reasoning effort, agent tools, sampling policy, and scoring procedure fixed when measuring a training intervention. Record failures and missing outputs in the denominator.

A reproducible comparison

Pin the benchmark release, task split, model configuration, tools, sampling budget, and evaluation harness described by the primary source.

Publish the exact run configuration, item coverage, score units, and available uncertainty. If a source supplies only an aggregate or a single run, preserve that limit. A new model name or a rounded score alone does not establish a capability improvement.

PRIMARY EVIDENCE & FURTHER READING

Reviewed 2026-10-26. Explanation and interpretation by RLTasks. Benchmark ownership remains with its original authors.

Continue exploring

All reasoning benchmarks →