Proximal Policy Optimization (PPO) for RLHF

Optimizes the LLM policy $\pi_\theta$ against a token-level reward $r_t = -\beta \log\frac{\pi_\theta(y_t\mid x,y_{<t})}{\pi_{\text{ref}}(y_t\mid x,y_{<t})} + \mathbb{I}[t=|y|]R_\phi(x,y)$ using Generalized Advantage Estimation ($\hat{A}_t^{\text{GAE}(\gamma,\lambda)}$) computed with a learned value critic $V_\psi(x, y_{<t})$ and a clipped importance-sampling surrogate objective: $\mathcal{L}_{\text{PPO}}(\theta) = \mathbb{E}_t\left[\min\left(\rho_t(\theta)\hat{A}_t, \, \text{clip}(\rho_t(\theta), 1-\epsilon, 1+\epsilon)\hat{A}_t\right)\right], \quad \rho_t(\theta) = \frac{\pi_\theta(y_t\mid x,y_{<t})}{\pi_{\theta_{\text{old}}}(y_t\mid x,y_{<t})}$

Requires 4 models in GPU memory ($\pi_\theta, \pi_{\text{ref}}, R_\phi, V_\psi$); critic warmup and advantage whitening are critical for stability.