Mechanism

Constructs an unbiased loss estimator for the true noise-free DPO objective under symmetric preference flip probability ε\varepsilon by linearly combining the forward and flipped log-sigmoid losses: LrDPO(θ)=−(1−ε)log⁡σ(Δθ)+εlog⁡σ(−Δθ)1−2ε\mathcal{L}_{\text{rDPO}}(\theta) = \frac{-(1-\varepsilon)\log \sigma(\Delta_\theta) + \varepsilon \log \sigma(-\Delta_\theta)}{1 - 2\varepsilon}

References

  1. Provably Robust DPO: Aligning Language Models with Noisy Feedback ↗

    Sayak Ray Chowdhury, Anush Kini, Nagarajan Natarajan · 2024