WPO (Weighted Preference Optimization for Off-Policy-to-On-Policy Simulation)

Reweights off-policy preference pairs $(x, y_w, y_l)$ by their sequence probability under the current policy $w(x,y) = \exp\left(\frac{1}{|y|}\sum_t \log \pi_\theta(y_t\mid x, y_{<t})\right)$, making off-policy public preference datasets behave like on-policy samples at zero rollout cost.