GRPO (Group Relative Policy Optimization)

Samples a group of $G$ outputs $\{y_1, \dots, y_G\}$ from the old policy $\pi_{\theta_{\text{old}}}(\cdot\mid x)$ for each prompt $x$, computes their rewards $\{r_1, \dots, r_G\}$, and estimates the advantage of token $t$ in output $i$ via group z-score normalization without any value critic: $\hat{A}_{i,t} = \frac{r_i - \text{mean}(\{r_1,\dots,r_G\})}{\text{std}(\{r_1,\dots,r_G\}) + \varepsilon}$ Maximizes the clipped surrogate with a direct per-token KL penalty estimator $\mathbb{D}_{\text{KL}}(\pi_\theta\|\pi_{\text{ref}}) = \frac{\pi_{\text{ref}}}{\pi_\theta} - \log\frac{\pi_{\text{ref}}}{\pi_\theta} - 1$: $\mathcal{J}_{\text{GRPO}}(\theta) = \mathbb{E}_{x, \{y_i\}}\left[\frac{1}{G}\sum_{i=1}^G \frac{1}{|y_i|}\sum_{t=1}^{|y_i|} \left(\min\left(\rho_{i,t}(\theta)\hat{A}_{i,t}, \, \text{clip}(\rho_{i,t}(\theta), 1-\epsilon, 1+\epsilon)\hat{A}_{i,t}\right) - \beta \mathbb{D}_{\text{KL}}(\pi_\theta\|\pi_{\text{ref}})\right)\right]$

The most widely adopted RL algorithm of 2024–2025; however, the $\frac{1}{|y_i|}$ length normalization and $\text{std}(r)$ denominator introduce length and difficulty biases addressed by Dr.GRPO and DAPO.