Mechanism

DFT reweights supervised token losses using detached target-token probabilities. This changes the influence of low-probability teacher tokens. The probability-space derivative of cross-entropy contains an inverse probability, but the gradient with respect to softmax logits is bounded; that identity alone does not establish exploding optimization gradients. Consult the paper for the actual objective and empirical evidence.

Practical Notes

A one-line change to the SFT loss (loss = - sg(p_t) * log(p_t)) that bridges the generalization gap between offline SFT and online RL.

References

  1. On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification ↗

    Yongliang Wu, Yizhou Zhou, Zhou Ziheng, Yingzhe Peng, Xinyu Ye, Xinting Hu et al. · 2025

    All 10 authors

    Yongliang Wu, Yizhou Zhou, Zhou Ziheng, Yingzhe Peng, Xinyu Ye, Xinting Hu, Wenbo Zhu, Lu Qi, Ming-Hsuan Yang, Xu Yang