IPO (Identity Preference Optimization / ΨPO)

Shows that under deterministic or near-deterministic preferences $P(y_w \succ y_l) \to 1$, the Bradley-Terry transformation in DPO drives $\log \frac{\pi_\theta(y_w)}{\pi_\theta(y_l)} \to \infty$ regardless of the KL regularization coefficient $\beta$, causing severe overfitting. IPO replaces the logit-sigmoid loss with a root-finding squared loss toward a target margin $\frac{1}{2\beta}$: $\mathcal{L}_{\text{IPO}}(\theta) = \mathbb{E}_{(x, y_w, y_l)}\left[\left(\log \frac{\pi_\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)} - \log \frac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)} - \frac{1}{2\beta}\right)^2\right]$

Provably respects the KL constraint even when preference pairs are deterministic.