Parrish, Alicia, Chen, Angelica, Nangia, Nikita, Padmakumar, Vishakh, Phang, Jason, Thompson, Jana et al. · 2021
Dates & evaluation details
BBQ · Age · ambig · Accuracy · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Age · ambig · Signed bias score · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Age · disambig · Accuracy · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Age · disambig · Signed bias score · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · All released categories · ambig · Accuracy · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · All released categories · disambig · Accuracy · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Disability status · ambig · Accuracy · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Disability status · ambig · Signed bias score · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Disability status · disambig · Accuracy · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Disability status · disambig · Signed bias score · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Gender identity · ambig · Accuracy · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Gender identity · ambig · Signed bias score · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Gender identity · disambig · Accuracy · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Gender identity · disambig · Signed bias score · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Gender identity (names) · ambig · Accuracy · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Gender identity (names) · ambig · Signed bias score · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Gender identity (names) · disambig · Accuracy · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Gender identity (names) · disambig · Signed bias score · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Nationality · ambig · Accuracy · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Nationality · ambig · Signed bias score · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Nationality · disambig · Accuracy · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Nationality · disambig · Signed bias score · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Physical appearance · ambig · Accuracy · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Physical appearance · ambig · Signed bias score · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Physical appearance · disambig · Accuracy · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Physical appearance · disambig · Signed bias score · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Race ethnicity · ambig · Accuracy · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Race ethnicity · ambig · Signed bias score · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Race ethnicity · disambig · Accuracy · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Race ethnicity · disambig · Signed bias score · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Race ethnicity (names) · ambig · Accuracy · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Race ethnicity (names) · ambig · Signed bias score · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Race ethnicity (names) · disambig · Accuracy · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Race ethnicity (names) · disambig · Signed bias score · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Race x SES · ambig · Accuracy · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Race x SES · ambig · Signed bias score · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Race x SES · disambig · Accuracy · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Race x SES · disambig · Signed bias score · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Race x SES (names) · ambig · Accuracy · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Race x SES (names) · ambig · Signed bias score · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Race x SES (names) · disambig · Accuracy · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Race x SES (names) · disambig · Signed bias score · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Race x gender · ambig · Accuracy · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Race x gender · ambig · Signed bias score · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Race x gender · disambig · Accuracy · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Race x gender · disambig · Signed bias score · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Race x gender (names) · ambig · Accuracy · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Race x gender (names) · ambig · Signed bias score · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Race x gender (names) · disambig · Accuracy · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Race x gender (names) · disambig · Signed bias score · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Religion · ambig · Accuracy · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Religion · ambig · Signed bias score · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Religion · disambig · Accuracy · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Religion · disambig · Signed bias score · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · SES · ambig · Accuracy · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · SES · ambig · Signed bias score · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · SES · disambig · Accuracy · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · SES · disambig · Signed bias score · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Sexual orientation · ambig · Accuracy · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Sexual orientation · ambig · Signed bias score · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Sexual orientation · disambig · Accuracy · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.
BBQ · Sexual orientation · disambig · Signed bias score · 2026-10-27
Recomputed from the authors’ released answer likelihoods and gold labels using the published strict-argmax and target_loc definitions. The sixteen metadata exclusions per model match the authors’ R analysis. Repeated metadata rows with identical scoring fields are deduplicated by example ID so each prediction counts once. Ambiguous-context signed bias is multiplied by 1−accuracy; disambiguated bias is unscaled. Zero is neutral; the sign identifies direction, so a larger signed value is not a better capability score. Name proxies and category labels remain separate. No model was run by RLTasks.