LeanDojo Benchmark (Lean 4)

What it measures

98,734 theorems extracted from Mathlib with novel-premise split testing formal theorem proving.

Reading the measurement

The number of permitted attempts changes the question. Pass@1 estimates whether the first submission succeeds; pass@k asks whether at least one of several candidates succeeds. Match sampling, execution limits and candidate selection before reading a gap as better capability.

Published evaluation

Lean 4 theorem proving on the named held-out split. The older Lean 3 results in Table 2 are not substituted.

Where this fits in an evaluation suite

Final-answer accuracy, proof credit, and expert-reviewed research progress are distinct signals. A proof-aware evaluation should retain partial work and audit strong claims.

Start with a capability hypothesis, then choose a second test with a different failure mode. Keep the checkpoint, reasoning effort, agent tools, sampling policy, and scoring procedure fixed when measuring a training intervention. Record failures and missing outputs in the denominator.

A reproducible comparison

Lean 4 theorem proving on the named held-out split. The older Lean 3 results in Table 2 are not substituted.

Publish the exact run configuration, item coverage, score units, and available uncertainty. If a source supplies only an aggregate or a single run, preserve that limit. A new model name or a rounded score alone does not establish a capability improvement.