GSPO (Group Sequence Policy Optimization)

Identifies a fundamental flaw in token-level importance ratios $\rho_{i,t} = \frac{\pi_\theta(y_{i,t}\mid x,y_{i,<t})}{\pi_{\theta_{\text{old}}}(y_{i,t}\mid x,y_{i,<t})}$: while the basic policy gradient identity holds at the sequence level, off-policy token-level ratios accumulate high variance and cause catastrophic Mixture-of-Experts (MoE) routing instability. GSPO defines the importance ratio at the sequence level with length normalization (geometric mean of token ratios): $s_i(\theta) = \left(\frac{\pi_\theta(y_i \mid x)}{\pi_{\theta_{\text{old}}}(y_i \mid x)}\right)^{\frac{1}{|y_i|}} = \exp\left(\frac{1}{|y_i|}\sum_{t=1}^{|y_i|} \log \frac{\pi_\theta(y_{i,t}\mid x, y_{i,<t})}{\pi_{\theta_{\text{old}}}(y_{i,t}\mid x, y_{i,<t})}\right)$ and clips $s_i(\theta)$ at the sequence level: $\min\left(s_i(\theta)\hat{A}_i, \text{clip}(s_i(\theta), 1-\epsilon, 1+\epsilon)\hat{A}_i\right)$. Also includes GSPO-token for token-specific advantages.

Completely eliminates the need for MoE Routing Replay hacks and stabilizes massive MoE RL runs.