ORPO (Odds Ratio Preference Optimization)

Reference-free monolithic post-training method that combines SFT cross-entropy on $y_w$ with a log-odds ratio penalty between $y_w$ and $y_l$ in a single stage (no $\pi_{\text{ref}}$ copy needed): $\mathcal{L}_{\text{ORPO}}(\theta) = \mathcal{L}_{\text{SFT}}(x, y_w) - \lambda \log \sigma\left(\log \frac{\text{odds}_\theta(y_w\mid x)}{\text{odds}_\theta(y_l\mid x)}\right), \quad \text{odds}_\theta(y\mid x) = \frac{P_\theta(y\mid x)}{1 - P_\theta(y\mid x)}$ where $P_\theta(y\mid x) = \exp\left(\frac{1}{|y|}\sum_{t=1}^{|y|} \log \pi_\theta(y_t\mid x, y_{<t})\right)$ is the geometric mean token probability.