P3O (Pairwise Proximal Policy Optimization)

Notices that standard PPO is sensitive to additive shifts in the reward model $R(x,y) + c(x)$ unless the value critic is perfectly calibrated. P3O directly uses pairwise reward differences $R(x, y_1) - R(x, y_2)$ between two sampled trajectories inside a clipped PPO surrogate, achieving provable invariance to prompt-dependent reward offsets without a critic.

Bridges pairwise preference optimization (DPO) and on-policy clipped policy gradients (PPO).