training

Reward Model

A model trained to predict human preferences, used to provide feedback signals for reinforcement learning. Reward models learn from human comparison data to score outputs, enabling automated training without continuous human involvement.

In practice

A reward model scores 'helpful, detailed response' higher than 'vague, dismissive response' based on learned human preferences.

RLHF
Reinforcement Learning from Human Feedback, a training technique where human evaluators rank model outputs to train a reward model, which then guides the language model to generate more helpful, harmless, and honest responses through reinforcement learning.
PPO
Proximal Policy Optimization, a reinforcement learning algorithm widely used in RLHF training of language models. PPO updates the policy in small, controlled steps to improve stability, preventing the model from changing too dramatically in any single update.
DPO
Direct Preference Optimization, a simpler alternative to RLHF that directly optimizes a language model using preference data without needing a separate reward model. DPO achieves comparable results to RLHF with less computational complexity and greater training stability.