SCoRe (Training Language Models to Self-Correct via Multi-Turn RL)

Proves that offline SFT or naive single-turn RL on self-correction tasks fails due to distribution shift and mode collapse (the model either outputs the final answer on Turn 1 and learns nothing about Turn 2 correction, or intentionally sabotages Turn 1 to earn a high improvement delta on Turn 2). SCoRe solves this via a two-stage on-policy RL pipeline:

1. Stage I: Train Turn 2 to maximize $R(y_2)$ while strongly KL-anchoring Turn 1 to $\pi_{\text{ref}}$ so the model cannot sabotage Turn 1.
2. Stage II: Jointly optimize $R(y_1) + R(y_2) + \alpha(R(y_2) - R(y_1))$ with a progress-shaping bonus.