Reinforcement Learning from Human Feedback (RLHF)

A method that uses human preferences to shape model behavior after pretraining.

What can a preference comparison teach?

Two draft answers explain a fictional museum’s access policy. One acknowledges that a lift’s availability is unknown. The other invents a repair time but sounds confident.

  1. Human comparison

    Reviewers see the underlying notice and prefer the answer that accurately reports the uncertainty. Their comparison gives a relative judgment for this pair.

  2. Reward model

    Across many comparisons a model learns to predict preferences. Tone or length can become shortcuts if the examples rarely separate polish from accuracy.

  3. Policy update

    In a classic RLHF pipeline, the language model is optimized using the learned reward alongside constraints that limit how far it changes.

How would you test what it learned?

Use held-out pairs where the polished answer is sometimes right and sometimes wrong. Inspect whether the system preserves uncertainty when the notice lacks a date. A higher predicted reward alone cannot establish source accuracy.

What RLHF Adds After Pretraining

A pretrained model learns broad language patterns, but that does not automatically make it a good assistant. RLHF helps shape how the model behaves in practical interaction. Human reviewers compare candidate answers, rate quality, or express preferences, and that signal is used to steer the model toward better responses.

In simple terms, RLHF teaches the model not just how language works, but what kinds of answers people tend to find more useful, safer, and more appropriate. It is one reason modern assistants can follow instructions more naturally than raw next-token predictors.

Follow the idea

Reinforcement Learning (RL) · See how a reward shapes later choices.

AI Alignment · Examine the gap between a desired outcome and a measured proxy.

Related concepts (5)

Source · Ouyang and colleagues on training with human feedback

Return to Reinforcement Learning from Human Feedback (RLHF) in the A–Z glossary