GenRM (Generative Reward Models & CoT Verifiers)

Replaces the discriminative scalar classification head $R_\phi(x,y)$ with next-token generation: the reward model first generates an explicit natural-language verification chain-of-thought $z_{\text{critique}}$ comparing or checking the candidate solution(s), followed by a verdict token (Yes/No or [[A]]/[[B]]), scored via the log-probability of the positive verdict token averaged across $K$ sampled CoT critiques (GenRM-Majority).

Allows the reward model itself to benefit from test-time compute scaling!