f-DPO (Generalized f-Divergence DPO)

Generalizes DPO from reverse KL divergence to any convex $f$-divergence (forward KL, Jensen-Shannon, $\alpha$-divergence) by replacing $\log \frac{\pi_\theta}{\pi_{\text{ref}}}$ with the derivative of the convex conjugate $f'\left(\frac{\pi_\theta(y\mid x)}{\pi_{\text{ref}}(y\mid x)}\right)$, trading off mode-seeking vs. mass-covering generation behavior.