DAPO (Decoupled Clip and Dynamic Sampling Policy Optimization)

Introduces four synergistic modifications to GRPO for large-scale long-CoT RL:

1. Clip-Higher (Decoupled Clipping): Uses asymmetric clipping bounds $[1-\epsilon_{\text{low}}, 1+\epsilon_{\text{high}}]$ (e.g., $\epsilon_{\text{low}}=0.2, \epsilon_{\text{high}}=0.28$) so low-probability exploratory tokens can increase in probability without premature clipping, preventing entropy collapse.
2. Dynamic Sampling: Over-samples and filters out prompt groups where accuracy is exactly $1$ (all correct) or $0$ (all wrong), ensuring every batch has non-zero advantage gradients.
3. Token-Level Policy Gradient Loss: Normalizes the loss by total tokens across the group $\sum_{i=1}^G |y_i|$ rather than per-sequence $\frac{1}{G}\sum \frac{1}{|y_i|}$ so long reasoning chains are weighted properly.
4. Overlong Reward Shaping: Applies a soft linear punishment interval near the context limit and masks out truncated rollouts to prevent reward noise.

Directly covered on rltasks.com/post-training/rl-training-systems-and-dapo; Clip-Higher is now standard across nearly all frontier RL runs.