Mechanism

Standard offline distillation trains the student on teacher-generated sequences y∼πteachery \sim \pi_{\text{teacher}}, creating a train-test exposure mismatch when the student samples from its own distribution y∼πθy \sim \pi_\theta at inference time. GKD samples trajectories on-policy from the student y∼πθ(⋅∣x)y \sim \pi_\theta(\cdot\mid x) (or a λ\lambda-mixture of student and teacher) and minimizes token-level generalized Jensen-Shannon / Reverse-KL divergence against the teacher's logit distribution πteacher(yt∣x,y<t)\pi_{\text{teacher}}(y_t \mid x, y_{<t}) at every step of the student's trajectory: LGKD(θ)=Ex,y∼πθ(⋅∣x)[∑t=1∣y∣DJSD(β)(πteacher(⋅∣x,y<t) ∥ πθ(⋅∣x,y<t))]\mathcal{L}_{\text{GKD}}(\theta) = \mathbb{E}_{x, y \sim \pi_\theta(\cdot\mid x)}\left[\sum_{t=1}^{|y|} \mathbb{D}_{JSD}^{(\beta)}\left(\pi_{\text{teacher}}(\cdot\mid x, y_{<t}) \,\|\, \pi_\theta(\cdot\mid x, y_{<t})\right)\right]

References

  1. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes ↗

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist et al. · 2023

    All 7 authors

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, Olivier Bachem

  2. Qwen3 Technical Report ↗

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng et al. · 2025

    All 60 authors

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, Zihan Qiu