Anthropic / MODEL EVIDENCE
4 benchmarks · 18 observations · 4 sources
4 benchmarks · large dot: claude-3-7-sonnet-20250219; small dots: closest models where reported
claude-3-7-sonnet-20250219 (32k thinking tokens) · Aider coding benchmarks · Polyglot · percent correct ↗
60.4% · claude-3-7-sonnet-20250219 (no thinking) · Aider coding benchmarks · Polyglot · percent correct · Aider code editing benchmark ↗
claude-3-7-sonnet-20250219 · FrontierMath-2025-02-28-Public · FrontierMath public · 10 questions · source run score · Epoch AI · public February 2025 sample · historical release ↗
claude-3.7-sonnet-20250219 · NYU CTF Bench · pass@1 · 200-challenge full set ↗
17.5% · claude-3.7-sonnet-20250219 · NYU CTF Bench · pass@1 · 200-challenge full set · NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security — official leaderboard ↗
13.4615% · claude-3.7-sonnet-20250219 · NYU CTF Bench · pass@1 · crypto subset · NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security — official leaderboard ↗
11.5385% · claude-3.7-sonnet-20250219 · NYU CTF Bench · pass@1 · crypto subset · NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security — official leaderboard ↗
20% · claude-3.7-sonnet-20250219 · NYU CTF Bench · pass@1 · forensics subset · NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security — official leaderboard ↗
29.1667% · claude-3.7-sonnet-20250219 · NYU CTF Bench · pass@1 · misc subset · NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security — official leaderboard ↗
12.8205% · claude-3.7-sonnet-20250219 · NYU CTF Bench · pass@1 · pwn subset · NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security — official leaderboard ↗
15.3846% · claude-3.7-sonnet-20250219 · NYU CTF Bench · pass@1 · pwn subset · NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security — official leaderboard ↗
25.4902% · claude-3.7-sonnet-20250219 · NYU CTF Bench · pass@1 · rev subset · NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security — official leaderboard ↗
21.5686% · claude-3.7-sonnet-20250219 · NYU CTF Bench · pass@1 · rev subset · NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security — official leaderboard ↗
10.5263% · claude-3.7-sonnet-20250219 · NYU CTF Bench · pass@1 · web subset · NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security — official leaderboard ↗
claude-3-7-sonnet-20250219† · Overall difficulty-adjusted Brier index ↗
Similar positions on shared benchmark leaderboards. Smaller gaps are closer.
There are not enough shared scored settings to identify close comparisons reliably.
Every plotted observation links to its evaluation. Reported configurations, benchmark versions and task subsets can differ; similarity describes shared scores, not training lift.
Minghao Shao, Sofija Jancheska, Meet Udeshi, Brendan Dolan-Gavitt, Haoran Xi, Kimberly Milner, Boyuan Chen, Max Yin, Siddharth Garg, Prashanth Krishnamurthy, Farshad Khorrami, Ramesh Karri, Muhammad Shafique
claude-3-7-sonnet-20250219 · claude-3-7-sonnet-20250219 (32k thinking tokens) · claude-3-7-sonnet-20250219 (no thinking) · claude-3-7-sonnet-20250219† · claude-3.7-sonnet-20250219