LiPO & LiPO-λ (Listwise Preference Optimization via Learning-to-Rank)

Adapts LambdaRank from classic Information Retrieval to LLM alignment: given a list of $K$ responses with real-valued reward labels $\psi = (s_1, \dots, s_K)$, weights each pairwise DPO log-ratio gradient by the change in Normalized Discounted Cumulative Gain $|\Delta \text{NDCG}(i, j)|$: $\mathcal{L}_{\text{LiPO-}\lambda}(\theta) = -\mathbb{E}_{x, \{y_i\}}\left[\sum_{s_i > s_j} |\Delta \text{NDCG}(i, j)| \log \sigma\left(\beta \log \frac{\pi_\theta(y_i\mid x)}{\pi_{\text{ref}}(y_i\mid x)} - \beta \log \frac{\pi_\theta(y_j\mid x)}{\pi_{\text{ref}}(y_j\mid x)}\right)\right]$