APO (Anchored Preference Optimization: APO-zero & APO-down)

Controls whether the chosen likelihood $\log \pi_\theta(y_w)$ and rejected likelihood $\log \pi_\theta(y_l)$ increase or decrease during offline alignment depending on whether the model is already stronger or weaker than the dataset:

• APO-zero (when $y_w \succ \pi_\theta$): pushes $\log \frac{\pi_\theta(y_w)}{\pi_{\text{ref}}(y_w)} > 0$ up and $\log \frac{\pi_\theta(y_l)}{\pi_{\text{ref}}(y_l)} < 0$ down using two separate anchored sigmoid terms.
• APO-down (when $\pi_\theta$ is already better than $y_w$): mildly decreases $y_w$ while strongly decreasing $y_l$.