DistiLLM (Skew KL & Adaptive Off-Policy Distillation)

Optimizes a parameterized Skew KL divergence $\mathbb{D}_{\text{KL}}(\pi_{\text{teacher}} \,\|\, \alpha \pi_{\text{teacher}} + (1-\alpha)\pi_\theta)$ (which has bounded, smooth gradients unlike raw reverse KL) paired with an adaptive replay schedule of student-generated trajectories.