Reported model / MODEL EVIDENCE
1 benchmarks · 3 observations · 1 sources
1 benchmarks · large dot: AlphaCode 41B; small dots: closest models where reported
AlphaCode 41B · CodeContests · Test solve rate · 10@1k · AlphaCode · CodeContests test set ↗
23.2% · AlphaCode 41B · CodeContests · Test solve rate · 10@10k · AlphaCode · CodeContests test set · Competition-Level Code Generation with AlphaCode ↗
27.7% · AlphaCode 41B · CodeContests · Test solve rate · 10@100k · AlphaCode · CodeContests test set · Competition-Level Code Generation with AlphaCode ↗
Similar positions on shared benchmark leaderboards. Smaller gaps are closer.
1 of 1 shared benchmark comparisons
1 of 1 shared benchmark comparisons
1 of 1 shared benchmark comparisons
Mean absolute gap in percentile positions on shared benchmark settings; lower is closer. At least 60% overlap is required, with a minimum of three when available. Independent evaluations are used when there are at least three; otherwise published report comparisons are used. Repeated model observations within one setting are averaged before ranking. Writing-style traits and composite indices are excluded. This describes similarity, not an overall quality ranking.
Every plotted observation links to its evaluation. Reported configurations, benchmark versions and task subsets can differ; similarity describes shared scores, not training lift.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d'Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, Oriol Vinyals
AlphaCode 41B