What it measures

Reason about motives, feelings and likely outcomes in everyday social situations.

Where this fits in an evaluation suite

Ask what the task actually requires: recall, deduction, search, or a mixture. Separate closed-book reasoning from retrieval and tool-assisted problem solving.

Start with a capability hypothesis, then choose a second test with a different failure mode. Keep the checkpoint, reasoning effort, agent tools, sampling policy, and scoring procedure fixed when measuring a training intervention. Record failures and missing outputs in the denominator.

A reproducible comparison

Social IQa answer accuracy from the pretrained-model comparison in The Llama 3 Herd of Models. Retain source model sizes, prompting and task split; different commonsense tasks are separate evaluations.

Publish the exact run configuration, item coverage, score units, and available uncertainty. If a source supplies only an aggregate or a single run, preserve that limit. A new model name or a rounded score alone does not establish a capability improvement.

PRIMARY EVIDENCE & FURTHER READING

  1. SocialIQA: Commonsense Reasoning about Social Interactions ↗

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, Yejin Choi · 2019

Reviewed 2026-10-26. Explanation and interpretation by RLTasks. Benchmark ownership remains with its original authors.