ReMax (Reward Maximization with Greedy Baseline)

Eliminates the value network in PPO and replaces the multi-sample Monte Carlo baseline of RLOO with the reward of a single deterministic greedy-decoded trajectory $y^{\text{greedy}} = \arg\max_y \pi_\theta(y\mid x)$: $\hat{A}_{\text{ReMax}}(x, y) = R(x, y) - R(x, y^{\text{greedy}})$

Requires only 2 forward generations per prompt (1 sampled + 1 greedy) and 0 critic memory, though higher variance than RLOO/GRPO on hard reasoning tasks where greedy decoding consistently scores 0.