training

RLHF

Reinforcement Learning from Human Feedback, a training technique where human evaluators rank model outputs to train a reward model, which then guides the language model to generate more helpful, harmless, and honest responses through reinforcement learning.

In practice

ChatGPT was trained with RLHF where human raters compared pairs of responses and indicated which was better.

Reinforcement Learning
A machine learning paradigm where an agent learns to make decisions by taking actions in an environment and receiving rewards or penalties. The agent optimizes its behavior to maximize cumulative reward over time through trial and error.
Reward Model
A model trained to predict human preferences, used to provide feedback signals for reinforcement learning. Reward models learn from human comparison data to score outputs, enabling automated training without continuous human involvement.
PPO
Proximal Policy Optimization, a reinforcement learning algorithm widely used in RLHF training of language models. PPO updates the policy in small, controlled steps to improve stability, preventing the model from changing too dramatically in any single update.
DPO
Direct Preference Optimization, a simpler alternative to RLHF that directly optimizes a language model using preference data without needing a separate reward model. DPO achieves comparable results to RLHF with less computational complexity and greater training stability.

In the index

Tools that mention RLHF

Scale AITrust 72

Matched on each tool’s own description and feature list, highest trust score first.

Briefings

RLHF

Tools that mention RLHF

Further reading