O1 Preview (Sept 2024)

3 benchmarks · 3 observations · 1 sources

Scores at a glance

3 benchmarks · large dot: O1 Preview (Sept 2024); small dots: closest models where reported

HumanEval+ ↗89%

O1 Preview (Sept 2024) · HumanEval+ · Pass@1 · EvalPlus · official result data ↗

MBPP+ ↗80.2%

O1 Preview (Sept 2024) · MBPP+ · Pass@1 · EvalPlus · official result data ↗

HumanEval ↗96.3%

O1 Preview (Sept 2024) · HumanEval · Pass@1 · EvalPlus · official result data ↗

Closest models by score

Similar positions on shared benchmark leaderboards. Smaller gaps are closer.

O1 Mini (Sept 2024) →OpenAI

0.4 percentile-point gap

3 of 3 shared benchmark comparisons

Qwen2.5-Coder-32B-Instruct →Qwen

2.0 percentile-point gap

3 of 3 shared benchmark comparisons

GPT-4o (Aug 2024) →OpenAI

4.9 percentile-point gap

3 of 3 shared benchmark comparisons

How similarity is measured

Mean absolute gap in percentile positions on shared benchmark settings; lower is closer. At least 60% overlap is required, with a minimum of three when available. Independent evaluations are used when there are at least three; otherwise published report comparisons are used. Repeated model observations within one setting are averaged before ranking. Writing-style traits and composite indices are excluded. This describes similarity, not an overall quality ranking.

Filter comparisons to this model →

Sources

Every plotted observation links to its evaluation. Reported configurations, benchmark versions and task subsets can differ; similarity describes shared scores, not training lift.

Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation — published result table ↗
Liu, Jiawei, Xia, Chunqiu Steven, Wang, Yuyao, Zhang, Lingming · 2023

Names used in source tables

O1 Preview (Sept 2024)