R-DPO (Regularized / Length-Desensitized DPO)

Explicitly penalizes sequence-length exploitation in DPO by adding an analytical length margin $\alpha(|y_w| - |y_l|)$ inside the Bradley-Terry sigmoid argument: $\mathcal{L}_{\text{R-DPO}}(\theta) = -\mathbb{E}\left[\log \sigma\left(\beta \log \frac{\pi_\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)} - \beta \log \frac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)} + \alpha(|y_w| - |y_l|)\right)\right]$