DPO (Direct Preference Optimization) is a model alignment method. Training data is not a single standard answer but a pair of responses under the same prompt: which better matches preference and which is relatively worse. DPO directly increases the relative probability of model generation preference responses without needing to train reward models separately and then use PPO for reinforcement learning optimization.
What does a preference data look like?
The most basic samples include prompts, winning answers, and missing answers. Annotators compare helpfulness, factuality, style, or safety boundaries. During training, the model learns to create a probability gap between the two within the same context, while referencing model constraints to avoid deviating too far from the original model just to cater to preferences.
This also explains why DPO is often considered simpler than the classic RLHF process: it transforms reward modeling and strategy optimization into a binary objective, eliminating online sampling, reward model services, and complex reinforcement learning loops. It remains in the post-training stage and cannot replace the foundational knowledge gained from pretraining.
What do DPO, SFT, and RLHF each learn?
| Method | Key data | Core Role | Common limitations |
|---|---|---|---|
| SFT | Hints and ideal answers | Imitation target demonstration | It's hard to express a subtle preference between two available answers |
| DPO | Preferences and non-preferences are correct | Direct learning is relatively preferred | Highly dependent on comparing data quality and coverage |
| Classic RLHF | Sample preference data, reward models, and strategies | Continuously optimize strategies with reward signals | The process is complex, and training stability and cost are harder to control |
These three are not simple substitution relationships. A common approach is to first establish available behaviors with SFT, then use DPO or other preference optimization methods to calibrate tone, rejection boundaries, and response quality.
"Easier training" does not automatically mean better results
If the winning answer is only longer and better panders to the annotator, the model may learn surface style rather than true quality. If preferences are too similar, the training signal will be weak; If coverage is insufficient, the model may only improve on a few tasks.
DPO is best suited for scenarios where goals can be expressed through pairwise comparison while also seeking to reduce RLHF engineering complexity. Teams should still retain independent evaluation sets to check facts, safety, length bias, and preferences of different groups; One "which one prefers" cannot be taken as universally correct.
