REINFORCE++ & REINFORCE++-Baseline

Combines the simplicity of critic-free REINFORCE with key PPO stability optimizations: (1) token-level KL penalty added to rewards, (2) PPO-style ratio clipping, and (3) global batch advantage normalization (normalizing $R(x,y) - b(x)$ across the entire global batch rather than within small per-prompt groups, avoiding division by near-zero group variance). REINFORCE++-Baseline subtracts the group mean reward first and then applies global batch std normalization.

Works with $G=1$ (REINFORCE++) or $G>1$ (REINFORCE++-Baseline) and avoids prompt-level std explosion.