training
DPO
Direct Preference Optimization, a simpler alternative to RLHF that directly optimizes a language model using preference data without needing a separate reward model. DPO achieves comparable results to RLHF with less computational complexity and greater training stability.
In practice
DPO trains the model directly on pairs of preferred and rejected responses, eliminating the need for a reward model step.