Step-DPO & MCTS-DPO (Process-Level Stepwise DPO)

Instead of contrasting two complete responses $(y_w, y_l)$ that share many valid early reasoning steps, identifies the exact first erroneous reasoning step $s_k^l$ vs. a verified continuation step $s_k^w$ given the shared valid prefix $(x, s_{1:k-1})$ (using Monte Carlo rollouts, MCTS, or a process judge) and applies DPO solely on the diverging step: $\mathcal{L}_{\text{Step-DPO}}(\theta) = -\mathbb{E}_{(x, s_{1:k-1}, s_k^w, s_k^l)}\left[\log \sigma\left(\beta \log \frac{\pi_\theta(s_k^w\mid x, s_{1:k-1})}{\pi_{\text{ref}}(s_k^w\mid x, s_{1:k-1})} - \beta \log \frac{\pi_\theta(s_k^l\mid x, s_{1:k-1})}{\pi_{\text{ref}}(s_k^l\mid x, s_{1:k-1})}\right)\right]$

Prevents DPO from inadvertently penalizing valid prefix steps that happen to precede a late calculation error in $y_l$.