Different preference objectives, different assumptions

Preference methods differ in their data requirements and objective. KTO explores optimization from desirable or undesirable examples; ORPO combines supervised and preference-oriented learning without a separate reference-model requirement in its proposed formulation; SimPO uses a reference-free sequence-probability-based reward formulation. These are specific proposals, not universal replacements for every training setup.

A comparison between algorithms is weak if one receives cleaner examples, more tokens, or a more useful preference distribution. Keep the base, data construction, prompt mix, training budget, and evaluation fixed before interpreting a gain as evidence for the new objective.

Preference optimization can change how much a model writes, how directly it answers, and how often it declines. Measure these alongside correctness. A judge can prefer a stylistic change even when the system has not become better at the underlying task.

Use held-out pairwise judgments, objective task checks, and regression tests appropriate to the application. Include cases where a concise correct answer should beat an elaborate incorrect one. Report the judging setup and examples of changed behavior so the outcome can be understood beyond one aggregate score.