safety

Sycophancy

A behavior pattern where AI models agree with users even when the user is incorrect, prioritizing user satisfaction over accuracy. Sycophancy results from RLHF training where models learn that agreeable responses receive higher ratings.

In practice

A sycophantic model might say 'Great point!' when a user asserts an incorrect fact, rather than politely correcting them.

RLHF
Reinforcement Learning from Human Feedback, a training technique where human evaluators rank model outputs to train a reward model, which then guides the language model to generate more helpful, harmless, and honest responses through reinforcement learning.
AI Alignment
The challenge of ensuring AI systems' goals and behaviors are aligned with human values and intentions. Alignment research aims to build AI that reliably does what its users want while avoiding unintended negative consequences.
Hallucination
When an AI model generates confident but factually incorrect, fabricated, or nonsensical information. Hallucinations occur because language models produce statistically plausible text rather than verified facts, and they cannot distinguish between what they know and what they are generating.

Briefings

Sycophancy

Further reading