Forty research-level physics problems focus the revised CritPt suite on challenges that remained difficult at selection time.
28 model results
One row per model and compatible score measure. Mean = equal-weight arithmetic average of imported reports. Distinct configurations reported for the same model in a source table remain separate. Reports can overlap; the range is not a confidence interval. Unbundle to inspect each observation.
CritPt v2 · hard subset · AA implementation · 2026-10-28
Publisher-selected recent-model cohort from the public evaluation page. Model labels preserve effort and fallback configurations. Values are a dated snapshot, not a live feed. Consult the publisher’s methodology for the harness, task split and judging; do not merge this cohort with vendor-run results.
TRACE THE SCORE · Per-model notes use published breakdowns where available.
What it measures
The selected problems probe difficult physics reasoning with computation available. Inspect the subset’s selection rule when interpreting a model’s position.
Subset boundary
The hard subset and full 70-problem CritPt v2 suite have different denominators. Neither can substitute for the original CritPt release.
Where this fits in an evaluation suite
A correct multiple-choice answer and a supported scientific contribution demand different evidence. Check whether the score rewards a proposal, an executed experiment, or a completed research target.
Start with a capability hypothesis, then choose a second test with a different failure mode. Keep the checkpoint, reasoning effort, agent tools, sampling policy, and scoring procedure fixed when measuring a training intervention. Record failures and missing outputs in the denominator.
A reproducible comparison
The fixed private 40-problem subset excludes 30 of CritPt v2’s 70 problems solved by seven frontier models on all five launch-time attempts. AA uses its code-enabled Stirrup protocol and averages five repeats.
Publish the exact run configuration, item coverage, score units, and available uncertainty. If a source supplies only an aggregate or a single run, preserve that limit. A new model name or a rounded score alone does not establish a capability improvement.