TruthfulQA

What it measures

Questions target common misconceptions to test whether a model repeats familiar falsehoods.

How to read the result

Read truthfulness alongside informativeness so a model cannot look strong merely by refusing every question.

Limits and comparison traps

Multiple-choice and generation settings are distinct. A curated misconception test does not cover every form of factual unreliability.

Published evaluation

Official benchmark result table. The named measure is retained in its native units; separate task or scoring measures are not combined.

Where this fits in an evaluation suite

Read accuracy together with error and abstention rates. Domain averages can hide language gaps, specialist failures, and poorly calibrated confidence.

Start with a capability hypothesis, then choose a second test with a different failure mode. Keep the checkpoint, reasoning effort, agent tools, sampling policy, and scoring procedure fixed when measuring a training intervention. Record failures and missing outputs in the denominator.

A reproducible comparison

Official benchmark result table. The named measure is retained in its native units; separate task or scoring measures are not combined.

Publish the exact run configuration, item coverage, score units, and available uncertainty. If a source supplies only an aggregate or a single run, preserve that limit. A new model name or a rounded score alone does not establish a capability improvement.