TDPO (Token-Level Direct Preference Optimization)

Incorporates forward token-level sequential KL divergence constraints $\sum_{t=1}^{|y|} \mathbb{D}_{\text{SeqKL}}(x, y_{<t}; \pi_{\text{ref}}\|\pi_\theta)$ directly into the DPO objective (TDPO1 and TDPO2), regulating token-by-token divergence rather than only sequence-sum log-ratios.