FIMO (Formal IMO)

What it measures

IMO Shortlist algebra and number theory problems formalized in Lean.

Reading the measurement

Partial credit can distinguish a useful approach from an empty attempt, but it is not the same as a completed task. Read the rubric, full-success count and score distribution together. For mathematical work, preserve the difference between expert review and acceptance by a formal proof checker.

Published evaluation

The source baseline experiments found no successful FIMO formal proofs under either informal-proof setting. These zero scores are explicitly reported observations, not substituted missing data. miniF2F results in the same table are a different benchmark.

Where this fits in an evaluation suite

Final-answer accuracy, proof credit, and expert-reviewed research progress are distinct signals. A proof-aware evaluation should retain partial work and audit strong claims.

Start with a capability hypothesis, then choose a second test with a different failure mode. Keep the checkpoint, reasoning effort, agent tools, sampling policy, and scoring procedure fixed when measuring a training intervention. Record failures and missing outputs in the denominator.

A reproducible comparison

The source baseline experiments found no successful FIMO formal proofs under either informal-proof setting. These zero scores are explicitly reported observations, not substituted missing data. miniF2F results in the same table are a different benchmark.

Publish the exact run configuration, item coverage, score units, and available uncertainty. If a source supplies only an aggregate or a single run, preserve that limit. A new model name or a rounded score alone does not establish a capability improvement.