gemini-1.5-pro-001 · Aider coding benchmarks · Refactoring · percent correct ↗
Google / MODEL EVIDENCE
gemini-1.5-pro-001
5 benchmarks · 41 observations · 5 sources
Scores at a glance
5 benchmarks · large dot: gemini-1.5-pro-001; small dots: closest models where reported
LongProc (Long Procedural Generation) ↗89.2 points
Gemini-1.5-pro-001 · LongProc (Long Procedural Generation) · 0.5K · average task score · original evaluation ↗
2 more reported results
79.4 points · Gemini-1.5-pro-001 · LongProc (Long Procedural Generation) · 2K · average task score · original evaluation · LongProc: Benchmarking Long-Context Language Models on Long Procedural Generation ↗
54 points · Gemini-1.5-pro-001 · LongProc (Long Procedural Generation) · 8K · average task score · original evaluation · LongProc: Benchmarking Long-Context Language Models on Long Procedural Generation ↗
Chatbot Arena ↗1158 Elo
Gemini-1.5-Pro-001 · Chatbot Arena · Chatbot Arena Vision · English Elo · 2024-11-13 · November 13, 2024 snapshot ↗
Gemini 1.5 Pro (001) · MMLU-Redux (2.0) · Virology · corrected-instance accuracy · MMLU-Redux 2.0 paper · audited subject comparison ↗
4 more reported results
99% · Gemini 1.5 Pro (001) · MMLU-Redux (2.0) · Logical Fallacies · corrected-instance accuracy · MMLU-Redux 2.0 paper · audited subject comparison · Are We Done with MMLU? ↗
72% · Gemini 1.5 Pro (001) · MMLU-Redux (2.0) · College Chemistry · corrected-instance accuracy · MMLU-Redux 2.0 paper · audited subject comparison · Are We Done with MMLU? ↗
67% · Gemini 1.5 Pro (001) · MMLU-Redux (2.0) · Professional Law · corrected-instance accuracy · MMLU-Redux 2.0 paper · audited subject comparison · Are We Done with MMLU? ↗
94% · Gemini 1.5 Pro (001) · MMLU-Redux (2.0) · Human Sexuality · corrected-instance accuracy · MMLU-Redux 2.0 paper · audited subject comparison · Are We Done with MMLU? ↗
gemini-1.5-pro-001 · LAB-Bench (v1) · Accuracy · LitQA2 ↗
32 more reported results
1% · gemini-1.5-pro-001 · LAB-Bench (v1) · Accuracy · SuppQA · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
25% · gemini-1.5-pro-001 · LAB-Bench (v1) · Accuracy · FigQA · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
59% · gemini-1.5-pro-001 · LAB-Bench (v1) · Accuracy · TableQA · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
3% · gemini-1.5-pro-001 · LAB-Bench · DbQA · subtask macro average · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
47% · gemini-1.5-pro-001 · LAB-Bench (v1) · Accuracy · ProtocolQA · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
29.9333% · gemini-1.5-pro-001 · LAB-Bench · SeqQA · subtask macro average · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
10% · gemini-1.5-pro-001 · LAB-Bench (v1) · Accuracy · CloningScenarios · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
0% · gemini-1.5-pro-001 · LAB-Bench (v1) · Accuracy · DbQA_dga_task-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
3% · gemini-1.5-pro-001 · LAB-Bench (v1) · Accuracy · DbQA_gene_location_task-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
0% · gemini-1.5-pro-001 · LAB-Bench (v1) · Accuracy · DbQA_mirna_targets_task-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
10% · gemini-1.5-pro-001 · LAB-Bench (v1) · Accuracy · DbQA_mouse_tumor_gene_sets-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
4% · gemini-1.5-pro-001 · LAB-Bench (v1) · Accuracy · DbQA_oncogenic_signatures_task-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
0% · gemini-1.5-pro-001 · LAB-Bench (v1) · Accuracy · DbQA_tfbs_GTRD_task-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
0% · gemini-1.5-pro-001 · LAB-Bench (v1) · Accuracy · DbQA_variant_from_sequence_task-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
0% · gemini-1.5-pro-001 · LAB-Bench (v1) · Accuracy · DbQA_variant_multi_sequence_task-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
13% · gemini-1.5-pro-001 · LAB-Bench (v1) · Accuracy · DbQA_vax_response_task-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
0% · gemini-1.5-pro-001 · LAB-Bench (v1) · Accuracy · DbQA_viral_ppi_task-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
2% · gemini-1.5-pro-001 · LAB-Bench (v1) · Accuracy · SeqQA_ORF-seq-AAid-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
41% · gemini-1.5-pro-001 · LAB-Bench (v1) · Accuracy · SeqQA_ORF-seq-AAseq-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
5% · gemini-1.5-pro-001 · LAB-Bench (v1) · Accuracy · SeqQA_ORF-seq-numlen-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
47% · gemini-1.5-pro-001 · LAB-Bench (v1) · Accuracy · SeqQA_ORF-transeff-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
47% · gemini-1.5-pro-001 · LAB-Bench (v1) · Accuracy · SeqQA_PCR-gene-enzprimers-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
43% · gemini-1.5-pro-001 · LAB-Bench (v1) · Accuracy · SeqQA_PCR-gene-gibshindprimers-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
16% · gemini-1.5-pro-001 · LAB-Bench (v1) · Accuracy · SeqQA_PCR-gene-gibssmaprimers-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
66% · gemini-1.5-pro-001 · LAB-Bench (v1) · Accuracy · SeqQA_PCR-geneprimers-enz-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
2% · gemini-1.5-pro-001 · LAB-Bench (v1) · Accuracy · SeqQA_PCR-len-primers-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
15% · gemini-1.5-pro-001 · LAB-Bench (v1) · Accuracy · SeqQA_PCR-primers-len-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
54% · gemini-1.5-pro-001 · LAB-Bench (v1) · Accuracy · SeqQA_PCR-seq-enzprimers-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
48% · gemini-1.5-pro-001 · LAB-Bench (v1) · Accuracy · SeqQA_PCR-seq-primers-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
39% · gemini-1.5-pro-001 · LAB-Bench (v1) · Accuracy · SeqQA_Prop-seq-gcpercent-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
7% · gemini-1.5-pro-001 · LAB-Bench (v1) · Accuracy · SeqQA_RE-seq-lenfrags-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
17% · gemini-1.5-pro-001 · LAB-Bench (v1) · Accuracy · SeqQA_RE-seq-numfrags-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
Closest models by score
Similar positions on shared benchmark leaderboards. Smaller gaps are closer.
There are not enough shared scored settings to identify close comparisons reliably.
Sources
Every plotted observation links to its evaluation. Reported configurations, benchmark versions and task subsets can differ; similarity describes shared scores, not training lift.
- Aider refactoring benchmark ↗
- Are We Done with MMLU? — published result table ↗
All 16 authors
Gema, Aryo Pradipta, Leang, Joshua Ong Jun, Hong, Giwon, Devoto, Alessio, Mancino, Alberto Carlo Maria, Saxena, Rohit, He, Xuanli, Zhao, Yu, Du, Xiaotang, Madani, Mohammad Reza Ghasemi, Barale, Claire, McHardy, Robert, Harris, Joshua, Kaddour, Jean, van Krieken, Emile, Minervini, Pasquale
- LAB-Bench: Measuring Capabilities of Language Models for Biology Research — published result table ↗
All 9 authors
Jon M. Laurent, Joseph D. Janizek, Michael Ruzo, Michaela M. Hinks, Michael J. Hammerling, Siddharth Narayanan, Manvitha Ponnapati, Andrew D. White, Samuel G. Rodriques
- Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models — published result table ↗
All 50 authors
Deitke, Matt, Clark, Christopher, Lee, Sangho, Tripathi, Rohun, Yang, Yue, Park, Jae Sung, Salehi, Mohammadreza, Muennighoff, Niklas, Lo, Kyle, Soldaini, Luca, Lu, Jiasen, Anderson, Taira, Bransom, Erin, Ehsani, Kiana, Ngo, Huong, Chen, YenSung, Patel, Ajay, Yatskar, Mark, Callison-Burch, Chris, Head, Andrew, Hendrix, Rose, Bastani, Favyen, VanderBilt, Eli, Lambert, Nathan, Chou, Yvonne, Chheda, Arnavi, Sparks, Jenna, Skjonsberg, Sam, Schmitz, Michael, Sarnat, Aaron, Bischoff, Byron, Walsh, Pete, Newell, Chris, Wolters, Piper, Gupta, Tanmay, Zeng, Kuo-Hao, Borchardt, Jon, Groeneveld, Dirk, Nam, Crystal, Lebrecht, Sophie, Wittlif, Caitlin, Schoenick, Carissa, Michel, Oscar, Krishna, Ranjay, Weihs, Luca, Smith, Noah A., Hajishirzi, Hannaneh, Girshick, Ross, Farhadi, Ali, Kembhavi, Aniruddha
- LongProc: Benchmarking Long-Context Language Models on Long Procedural Generation — published result table ↗
All 8 authors
Xi Ye, Fangcong Yin, Yinghui He, Joie Zhang, Howard Yen, Tianyu Gao, Greg Durrett, Danqi Chen
Names used in source tables
Gemini 1.5 Pro (001) · Gemini-1.5-Pro-001 · Gemini-1.5-pro-001 · gemini-1.5-pro-001