DPO (Direct Preference Optimization)

Derives the closed-form optimal policy $\pi^\star(y\mid x) = \frac{1}{Z(x)}\pi_{\text{ref}}(y\mid x)\exp\left(\frac{1}{\beta}R(x,y)\right)$ of the KL-regularized RLHF objective, solving for $R(x,y) = \beta \log \frac{\pi_\theta(y\mid x)}{\pi_{\text{ref}}(y\mid x)} + \beta \log Z(x)$ and substituting it directly into the Bradley-Terry preference likelihood over winning/losing pairs $(y_w \succ y_l \mid x)$: $\mathcal{L}_{\text{DPO}}(\theta) = -\mathbb{E}_{(x, y_w, y_l)\sim\mathcal{D}}\left[\log \sigma\left(\beta \log \frac{\pi_\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)} - \beta \log \frac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)}\right)\right]$

Eliminates the separate reward model and online RL rollout loop, but offline DPO can decrease the absolute log-probability of both $y_w$ and $y_l$ simultaneously while widening their margin, hurting multi-step reasoning unless mitigated by SFT regularization, online sampling, or DPOP.