HealthBench Hard

What it measures

A difficult HealthBench subset concentrates evaluation on challenging health conversations.

How to read the result

Use it to inspect failures that may be hidden by a strong full-suite average.

Limits and comparison traps

Subset selection and rubric version matter. A low hard-subset score cannot be interpreted as an error rate on routine user traffic.

Published evaluation

The 1,000-example Hard subset. Scores transcribed from the printed overall labels, not estimated from bar heights; the two-decimal source precision is preserved. Averages are clipped at zero by the source protocol. Per-axis bars are not substituted for overall scores.

Where this fits in an evaluation suite

Read accuracy together with error and abstention rates. Domain averages can hide language gaps, specialist failures, and poorly calibrated confidence.

Start with a capability hypothesis, then choose a second test with a different failure mode. Keep the checkpoint, reasoning effort, agent tools, sampling policy, and scoring procedure fixed when measuring a training intervention. Record failures and missing outputs in the denominator.

A reproducible comparison

The 1,000-example Hard subset. Scores transcribed from the printed overall labels, not estimated from bar heights; the two-decimal source precision is preserved. Averages are clipped at zero by the source protocol. Per-axis bars are not substituted for overall scores.

Publish the exact run configuration, item coverage, score units, and available uncertainty. If a source supplies only an aggregate or a single run, preserve that limit. A new model name or a rounded score alone does not establish a capability improvement.