RLTasksby ulam.ai
Models / Anthropic

Anthropic / MODEL EVIDENCE

3.5 Sonnet

1 benchmarks · 6 observations · 1 sources

Scores at a glance

1 benchmarks · large dot: 3.5 Sonnet; small dots: closest models where reported

SWE-Lancer ↗26.2%
826.2

3.5 Sonnet · SWE-Lancer · IC SWE (Diamond) · pass@1 · original evaluation ↗

5 more reported results

21.1% · 3.5 Sonnet · SWE-Lancer · IC SWE (Full) · pass@1 · original evaluation · SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? ↗

44.9% · 3.5 Sonnet · SWE-Lancer · SWE Manager (Diamond) · pass@1 · original evaluation · SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? ↗

47% · 3.5 Sonnet · SWE-Lancer · SWE Manager (Full) · pass@1 · original evaluation · SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? ↗

36.1% · 3.5 Sonnet · SWE-Lancer · SWE-Lancer Diamond · pass@1 · original evaluation · SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? ↗

33.7% · 3.5 Sonnet · SWE-Lancer · SWE-Lancer Full · pass@1 · original evaluation · SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? ↗

Closest models by score

Similar positions on shared benchmark leaderboards. Smaller gaps are closer.

o1 →OpenAI
50.0 percentile-point gap

1 of 1 shared benchmark comparisons

100.0 percentile-point gap

1 of 1 shared benchmark comparisons

How similarity is measured

Mean absolute gap in percentile positions on shared benchmark settings; lower is closer. At least 60% overlap is required, with a minimum of three when available. Independent evaluations are used when there are at least three; otherwise published report comparisons are used. Repeated model observations within one setting are averaged before ranking. Writing-style traits and composite indices are excluded. This describes similarity, not an overall quality ranking.

Filter comparisons to this model →

Sources

Every plotted observation links to its evaluation. Reported configurations, benchmark versions and task subsets can differ; similarity describes shared scores, not training lift.

  1. SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? — published result table ↗

    Miserendino, Samuel, Wang, Michele, Patwardhan, Tejal, Heidecke, Johannes · 2025

Names used in source tables

3.5 Sonnet