VAPO (Value-Augmented Proximal Policy Optimization for Long-CoT)

Demonstrates that value-based PPO can outperform critic-free GRPO/DAPO on long-CoT reasoning once value-function pathologies are fixed via: (1) Value-Pretraining (warming up the critic from the reward signal before policy updates), (2) Decoupled-GAE ($\lambda_{\text{policy}} = 1.0$ for unbiased policy updates vs. $\lambda_{\text{value}} = 0.95$ for low-variance value bootstrapping), (3) Length-Adaptive GAE (scaling $\lambda$ with trajectory length so TD errors don't vanish over 32k+ tokens), (4) Clip-Higher, and (5) Positive-Example LM Loss (adding a small SFT loss on correct rollouts).

Provides much finer token-level credit assignment than sequence-level outcome rewards on 32k–64k token trajectories.