SPPO (Self-Play Preference Optimization)

Formulates alignment with general, potentially intransitive preferences $\mathbb{P}(y \succ y' \mid x)$ as a constant-sum two-player game, seeking the von Neumann winner (Nash equilibrium) via multiplicative weights / mirror descent: $\mathcal{L}_{\text{SPPO}}(\theta_{t+1}) = \mathbb{E}_{x, y \sim \pi_{\theta_t}}\left[\left(\log \frac{\pi_{\theta_{t+1}}(y\mid x)}{\pi_{\theta_t}(y\mid x)} - \eta \left(\widehat{P}(y \succ \pi_{\theta_t}\mid x) - \frac{1}{2}\right)\right)^2\right]$ where $\widehat{P}(y \succ \pi_{\theta_t}\mid x) = \frac{1}{K}\sum_{k=1}^K \mathbb{P}(y \succ y_k\mid x)$ is the empirical win rate of $y$ against $K$ cohort samples.