RSO (Statistical Rejection Sampling Optimization)

Notices that selecting the simple $\arg\max_{1\le i\le N} R(x, y_i)$ skews the empirical distribution away from the theoretical KL-regularized optimal policy $\pi^\star(y\mid x) \propto \pi_{\text{ref}}(y\mid x)\exp(R(x,y)/\beta)$. RSO uses statistical Metropolis-Hastings / rejection-sampling acceptance probabilities $u < \exp\left(\frac{R(x,y) - R_{\max}}{\beta}\right)$ so accepted pairs truly follow $\pi^\star$, then runs DPO/SLiC on those pairs.