RRHF (Rank Responses to Align Language Models with Human Feedback)

Samples $K$ candidate responses $\{y_1, \dots, y_K\}$ with external scores $\{r_1, \dots, r_K\}$ and optimizes a pairwise ranking margin hinge loss over length-normalized conditional log-probabilities $p_i = \frac{1}{|y_i|}\sum_t \log \pi_\theta(y_{i,t}\mid x, y_{i,<t})$ plus an SFT loss on the highest-ranked candidate $y_{i^\star}$: $\mathcal{L}_{\text{RRHF}}(\theta) = \sum_{r_i < r_j} \max(0, p_i - p_j) - \lambda \sum_{t} \log \pi_\theta(y_{i^\star, t}\mid x, y_{i^\star, <t})$