WARP (Weight Averaged Rewarded Policies)

Three-level weight-space RLHF algorithm developed by Google DeepMind to achieve Pareto-optimal reward vs. KL trade-offs:

1. Inner Loop: Run $M$ parallel RLHF runs with distinct KL/objective weights, using a moving EMA anchor $\theta_{\text{EMA}} \leftarrow (1-\mu)\theta_{\text{EMA}} + \mu \theta$ as the dynamic KL reference policy $\pi_{\text{ref}}$.
2. Middle Loop: Merge the $M$ RL policies via SLERP into a single policy $\theta_{\text{SLERP}}$.
3. Outer Loop: Linearly interpolate (LITI) the merged RL policy back toward the initial SFT initialization $\theta_{\text{init}} \leftarrow (1-\eta)\theta_{\text{init}} + \eta \theta_{\text{SLERP}}$ to start the next iteration.