Reinforcement learning from human feedback

TL;DR

• Base LLMs autocomplete text well but are not optimized for instruction following or a product's safety and helpfulness goals. RLHF is one post-training pipeline for adapting that behavior.
• The classic pipeline has three stages: supervised fine-tuning (SFT), reward model training on human preference pairs, and PPO optimization against that reward model.
• PPO is complex and can be unstable. DPO (Direct Preference Optimization) is often simpler because it trains directly on preference pairs without a separately deployed reward model.
• InstructGPT used PPO, and Llama 2 used PPO. Later open-weight projects have used DPO or related methods such as KTO and ORPO; the choice depends on data and objectives.
• Alignment can trade some raw benchmark capability for behavior that better matches a product's safety and usefulness goals; the size and direction of that tradeoff must be measured.
• For a design review or interview: know the three-stage pipeline, explain the DPO tradeoff, and be ready to discuss reward hacking.

30-second mental model

RLHF is preference-based post-training. Supervised fine-tuning teaches a base model the desired response format, preference data teaches a reward model or direct preference objective what people prefer, and optimization shifts the policy toward those preferences. It changes behavior; it does not automatically add reliable knowledge or remove the need for evaluation and safety controls.

5-minute explanation

The classic pipeline is SFT, reward-model training, and PPO with a KL constraint that keeps the policy near a reference model. DPO and related methods use preference pairs more directly and can remove much of the operational complexity. RLAIF and constitutional approaches replace or supplement some human labels with model-generated critiques, but they inherit the biases and blind spots of the principles, models, and evaluation process.

The engineering problem is proxy optimization. A model can maximize a reward that correlates with helpfulness in the labels while becoming verbose, sycophantic, evasive, or unsafe outside that distribution. Use held-out human and adversarial evaluations, inspect behavior by slice, and monitor both capability and safety after deployment.

The problem it solves

A freshly pretrained base LLM has one job: predict the next token given the previous tokens. Ask "How do I make a bomb?" and it produces instructions, not because it's malicious, but because that's what follows that sequence in training data. Ask "What year did Einstein win the Nobel Prize?" and it might confidently say 1915 (it was 1921). It doesn't follow instructions. It completes them.

You can't deploy that to users. OpenAI faced this exact problem when building ChatGPT on top of GPT-3. The model was technically impressive, but the raw output was dangerous, unreliable, and often unhelpful. The gap between "predicts next token well" and "helpful, harmless, honest assistant" was enormous.

Think of it like hiring a brilliant but unfiltered intern. They know everything in the textbooks, they can write fluently, but they haven't learned workplace norms. They'll share confidential information if you ask nicely. They'll give you medical advice with complete confidence. They'll write offensive jokes if that's what the prompt implies. The knowledge is there, but the judgment isn't.

Prompt engineering alone can improve behavior on common cases but is brittle against paraphrases, adversarial inputs, and distribution shifts. A model may need post-training and runtime controls for deeper behavioral change.

RLHF is a pipeline that narrows this gap. It uses preference signals to teach a proxy for what people value, then optimizes the model toward that proxy. It can make an assistant feel more instruction-following than an autocomplete model, but it does not guarantee safe or correct behavior.

Here's a concrete example. Ask a base model "Summarize this legal document in plain English" and it might: (1) continue writing MORE legal text instead of summarizing, (2) produce a summary but include fabricated clauses, or (3) summarize perfectly but add unsolicited legal advice. The model doesn't know what "helpful" means in this context. RLHF teaches it by showing many examples of what humans prefer when they ask for a summary.

What is it?

RLHF (Reinforcement Learning from Human Feedback) is a post-training alignment technique that takes a capable base model and fine-tunes it to follow instructions, refuse harmful requests, and produce useful answers, shaped by human judgment rather than just next-token prediction loss.

OpenAI described this approach in the InstructGPT paper (March 2022). Ouyang et al. reported that a 1.3B-parameter model trained with RLHF was preferred by human raters over the raw 175B GPT-3 model in that evaluation. The result illustrates how post-training can change product usefulness independently of parameter count.

That result turned alignment from a research curiosity into an engineering priority.

The analogy that lands: RLHF is like a performance review loop. The model (employee) does work. Humans (managers) compare outputs and say which is better. The model adjusts its behavior based on that feedback. Over thousands of iterations, the model converges on behavior that humans prefer, just as an employee learns what the company values through repeated feedback.

One important nuance: RLHF doesn't teach the model new knowledge. It changes the model's behavior to surface knowledge it already has in ways humans find helpful. A base model "knows" how to answer a question politely. It also "knows" how to continue toxic text. RLHF amplifies the helpful behaviors and suppresses the harmful ones. This is why alignment researchers describe RLHF as steering, not teaching.

Many assistant models use RLHF or related preference-based post-training, but the exact pipeline differs by provider and release. Understanding the components helps when evaluating an already-aligned model or designing additional post-training.

The transformation is dramatic. Before RLHF, you get a model that might answer "What's the capital of France?" with "France is a country in Western Europe known for its wine..." (it's continuing text, not answering a question). After RLHF, you get "The capital of France is Paris." Same model, same knowledge, completely different behavior.

How it works

The three-stage pipeline

The original InstructGPT paper established a three-stage pipeline that became an influential baseline. Later alignment efforts implement, reorder, or simplify these stages.

The classic stages are sequential and each builds on the previous one: (1) Supervised Fine-Tuning (SFT), (2) Reward Model training, and (3) PPO optimization. The compute, annotation time, and infrastructure depend on model size and implementation.

It is useful to view the stages as one pipeline where each addresses a weakness of the previous one. SFT teaches format, the reward model captures preferences, and PPO optimizes for those preferences while maintaining a reference-model constraint.

Stage 1: Supervised fine-tuning (SFT)

Human annotators write ideal responses to a diverse set of prompts. The base model is fine-tuned on these (prompt, ideal_response) pairs using standard cross-entropy loss. This teaches the model the basic shape of helpful conversation: follow instructions, answer questions, refuse clearly harmful requests.

For InstructGPT, OpenAI used about 13,000 demonstration examples from a team of 40 contractors. That's a small dataset by pretraining standards, but the quality bar was high. Each demonstration showed the model what a "good" response looked like for a specific type of prompt.

The prompts covered a wide range of tasks: open-ended generation ("Write a story about..."), summarization, Q&A, rewriting, classification, brainstorming, and code. This diversity matters because the SFT model needs to generalize across task types, not just memorize response templates for one category.

The SFT model is sometimes called the "reference model" or "reference policy" in later stages. It represents the baseline behavior that PPO will optimize from. This model is already noticeably better than the base model at following instructions, but it still produces mediocre responses for many prompts because it was only trained to imitate the annotators, not to optimize for what users actually prefer.

Engineering implication: SFT can establish the response format but may not cover edge cases, subtle helpfulness, or adversarial safety behavior. Preference optimization addresses those gaps only to the extent that the data and reward capture them.

Stage 2: Reward model training

The SFT model generates two or more responses to the same prompt. Human raters compare the responses and pick the better one. These A-is-better-than-B preference pairs are used to train a separate reward model.

The reward model is typically another LLM (often the same architecture as the SFT model) with the final layer replaced by a scalar output head. Given a (prompt, response) pair, it outputs a single number representing how "good" the response is. It learns to assign higher scores to responses that humans preferred and lower scores to responses they rejected.

The training objective is a Bradley-Terry pairwise ranking loss: given a preferred response and a rejected response, maximize the probability that the reward model scores the preferred one higher. In practice, the loss function looks like:

# Simplified reward model training loss
loss = -log(sigmoid(reward(preferred) - reward(rejected)))

For InstructGPT, OpenAI collected about 33,000 comparison pairs. Llama 2 used over 1 million preference pairs across multiple rounds of collection. The quality and diversity of these pairs is the single biggest predictor of alignment quality.

Annotator agreement varies by task, rubric, and reviewer pool. Disagreement is expected: the reward model learns an aggregate preference signal, not any individual annotator's opinion, and it cannot reliably learn preferences that reviewers do not apply consistently.

This is a stage that is often skipped in high-level descriptions. A useful explanation should state what the reward model learns, how preference pairs are collected, and where annotator disagreement limits the proxy.

Stage 3: PPO optimization

Now comes the reinforcement learning. The SFT model (now called the "policy") generates a response to a prompt. The reward model scores it. PPO (Proximal Policy Optimization) updates the policy's weights to generate higher-scoring responses.

PPO is an RL algorithm from Schulman et al. (2017) that makes small, stable updates. It clips the policy update to prevent the model from changing too much in a single step. This stability matters because language models are enormous and a single bad update can catastrophically degrade performance.

Each PPO iteration:

1. Sample a batch of prompts
2. Generate completions from the current policy
3. Score them with the reward model
4. Compute KL divergence between the policy and the reference (SFT) model
5. Compute the final reward: reward_model_score - beta x KL_divergence
6. Update the policy using the clipped PPO objective

The process repeats for thousands of iterations. The policy gradually learns to generate responses that score well on the reward model while staying close to the SFT baseline.

Why PPO and not other RL algorithms? PPO's clipping mechanism is useful because standard policy-gradient updates can destabilize a model. PPO constrains each update so the policy changes gradually; how much stability it provides depends on the implementation and hyperparameters.