RLTasksby ulam.ai
Models / Anthropic

Anthropic / MODEL EVIDENCE

Claude 3.5 Sonnet (Oct 2024)

22 benchmarks · 23 observations · 22 sources · released 2024-10-22

Scores at a glance

22 benchmarks · large dot: Claude 3.5 Sonnet (Oct 2024); small dots: closest models where reported

GPQA Diamond ↗55.3%
9.2896.4

Claude 3.5 Sonnet (Oct 2024) · GPQA diamond · Best score (across scorers) · % ↗

MATH Level 5 ↗56.95%
3.2998.13

Claude 3.5 Sonnet (Oct 2024) · MATH level 5 · Best score (across scorers) · % ↗

BALROG ↗32.6%
3.768.3

Claude 3.5 Sonnet (Oct 2024) · Balrog · Average progress · % ↗

CadEval ↗48%
1274

Claude 3.5 Sonnet (Oct 2024) · CadEval · Overall pass (%) · % ↗

GeoBench ↗62%
3286

Claude 3.5 Sonnet (Oct 2024) · GeoBench · ACW Country % · % ↗

GSO ↗4.6%
026.5

Claude 3.5 Sonnet (Oct 2024) · GSO-Bench · Score OPT@1 · % ↗

MMLU ↗87.3%
29.587.3

Claude 3.5 Sonnet (Oct 2024) · MMLU · EM · % ↗

SimpleBench ↗41.4%
10.781.9

Claude 3.5 Sonnet (Oct 2024) · SimpleBench · Score (AVG@5) · % ↗

VPCT ↗33%
3091

Claude 3.5 Sonnet (Oct 2024) · VPCT · Correct · % ↗

DTBench ↗67.84%
41.6598.93

Claude 3.5 Sonnet (Oct 2024) · DTBench · Accuracy · % ↗

LiveBench ↗59.03%
22.0882.35

Claude 3.5 Sonnet (Oct 2024) · LiveBench · Global average · % ↗

Video-MME ↗60%
6079.7

Claude 3.5 Sonnet (Oct 2024) · VideoMME · Overall (no subtitles) · % ↗

EnigmaEval ↗0.91%
0.3839.28

Claude 3.5 Sonnet (Oct 2024) · EnigmaEval · Accuracy · % ↗

Closest models by score

Similar positions on shared benchmark leaderboards. Smaller gaps are closer.

2.9 percentile-point gap

3 of 5 shared benchmark comparisons

Phi-4 →Microsoft
3.3 percentile-point gap

3 of 5 shared benchmark comparisons

3.4 percentile-point gap

4 of 5 shared benchmark comparisons

How similarity is measured

Mean absolute gap in percentile positions on shared benchmark settings; lower is closer. At least 60% overlap is required, with a minimum of three when available. Independent evaluations are used when there are at least three; otherwise published report comparisons are used. Repeated model observations within one setting are averaged before ranking. Writing-style traits and composite indices are excluded. This describes similarity, not an overall quality ranking.

Filter comparisons to this model →

Sources

Every plotted observation links to its evaluation. Reported configurations, benchmark versions and task subsets can differ; similarity describes shared scores, not training lift.

  1. Aider LLM Leaderboards — Polyglot coding benchmark ↗

    Aider

  2. BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games ↗

    Paglieri, Davide, Cupiał, Bartłomiej, Coward, Samuel, Piterbarg, Ulyana, Wolczyk, Maciej, Khan, Akbir et al. · 2024

    All 13 authors

    Paglieri, Davide, Cupiał, Bartłomiej, Coward, Samuel, Piterbarg, Ulyana, Wolczyk, Maciej, Khan, Akbir, Pignatelli, Eduardo, Kuciński, Łukasz, Pinto, Lerrel, Fergus, Rob, Foerster, Jakob Nicolaus, Parker-Holder, Jack, Rocktäschel, Tim

  3. DTBench — Conceptual Reasoning Index ↗

    Conceptual Reasoning Index

  4. Holistic Evaluation of Language Models (HELM) ↗

    Stanford Center for Research on Foundation Models

  5. FrontierMath Tier 4 (v1, old) ↗

    Epoch AI

  6. FrontierMath Tiers 1-3 (v1, old) ↗

    Epoch AI

  7. GPQA Diamond ↗

    Epoch AI

  8. Humanity's Last Exam ↗

    Epoch AI

  9. MATH Level 5 ↗

    Epoch AI

  10. OTIS Mock AIME 2024-2025 ↗

    Epoch AI

  11. SimpleBench ↗

    Epoch AI

  12. The Agent Company ↗

    Epoch AI

  13. Capabilities & benchmarking — retained external evaluation records ↗

    Epoch AI · 2026

  14. Video-MME ↗

    Epoch AI

  15. GeoBench — GeoGuessr model benchmark ↗

    GeoBench

  16. Writing benchmark — incorporating mandatory story elements ↗

    lechmazur

  17. GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agents ↗

    Shetty, Manish, Jain, Naman, Liu, Jinjian, Kethanaboyina, Vijay, Sen, Koushik, Stoica, Ion · 2025

  18. WeirdML ↗

    Håvard Tveit Ihle

  19. EnigmaEval ↗

    Scale Labs / SEAL

  20. LiveBench — model evaluation leaderboard ↗

    LiveBench Team

  21. Measuring AI Ability to Complete Long Software Tasks ↗

    METR

  22. CadEval Dashboard — run 20250422_095709 ↗

    willpatrick · 2025

Names used in source tables

Claude 3.5 Sonnet (Oct 2024)