InstructGPT

Training language models to follow instructions with human feedback

Fine-tunes language models using reinforcement learning from human feedback (RLHF) to align them with user intent. The method shows that a 1.3B InstructGPT model outperforms 175B GPT-3 by following instructions better — the first large-scale application of RLHF to diverse instruction-following tasks.

• llm
• rlhf
• dialogue

Authors: Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, Ryan Lowe

Categories: cs.CL, cs.AI, cs.LG

Published 2022-03-04 · Updated 2022-03-04

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub1,257 stars127 watchers149 forksnull

Abstract

Making language models bigger does not inherently make them better at following a user's intent. For example, large language models can generate outputs that are untruthful, toxic, or simply not helpful to the user. In other words, these models are not aligned with their users. In this paper, we show an avenue for aligning language models with user intent on a wide range of tasks by fine-tuning with human feedback. Starting with a set of labeler-written prompts and prompts submitted through the OpenAI API, we collect a dataset of labeler demonstrations of the desired model behavior, which we use to fine-tune GPT-3 using supervised learning. We then collect a dataset of rankings of model outputs, which we use to further fine-tune this supervised model using reinforcement learning from human feedback. We call the resulting models InstructGPT. In human evaluations on our prompt distribution, outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite having 100x fewer parameters. Moreover, InstructGPT models show improvements in truthfulness and reductions in toxic output generation while having minimal performance regressions on public NLP datasets. Even though InstructGPT still makes simple mistakes, our results show that fine-tuning with human feedback is a promising direction for aligning language models with human intent.

---

Introduction and Motivation

Large language models (LLMs) trained on vast corpora of internet text become powerful next-token predictors, but this objective is fundamentally misaligned with what users actually want: helpful, honest, and harmless responses. A model that is simply good at predicting the next word on a web page can produce outputs that are untruthful, toxic, biased, or simply unhelpful — even as the model grows larger. Scaling alone does not solve the alignment problem.

This paper introduces InstructGPT, a family of models produced by fine-tuning GPT-3 with human feedback using Reinforcement Learning from Human Feedback (RLHF). The central empirical finding is striking: a 1.3B-parameter InstructGPT model is preferred by human labelers over the 175B-parameter GPT-3 baseline — a 100× reduction in model size — while also showing improvements in truthfulness and reductions in toxic outputs. The paper frames alignment as teaching models to act in accordance with user intention, encompassing explicit goals (follow the instruction) and implicit goals (stay truthful, avoid harm).

The authors adopt the "helpful, honest, harmless" (HHH) framework as their alignment target, and design the entire data collection, training, and evaluation pipeline around these three criteria. This represents one of the first large-scale demonstrations of RLHF applied to a broad, open-ended language task distribution, moving beyond earlier work on summarization and game-playing.

Method and Architecture

High-Level Pipeline

The InstructGPT training procedure consists of three sequential steps:

1. Step 1 — Supervised Fine-Tuning (SFT): Human labelers write demonstrations of desired model behavior for a set of prompts. GPT-3 is then fine-tuned on these (prompt, demonstration) pairs using standard supervised learning.
2. Step 2 — Reward Model (RM) Training: For a larger set of prompts, between $K=4$ and $K=9$ model outputs are generated and ranked by labelers. A reward model is trained to predict which outputs labelers prefer, using all $\binom{K}{2}$ pairwise comparisons per prompt.
3. Step 3 — RL Fine-Tuning via PPO: The SFT model is further fine-tuned using Proximal Policy Optimization (PPO), treating the RM's scalar output as the reward signal. A KL penalty relative to the SFT model prevents the RL policy from drifting too far.

Steps 2 and 3 can be iterated: new comparison data is collected on the current best policy, a new RM is trained, and then a new policy is optimized. In practice, most comparison data comes from supervised policies with some from PPO policies.

Base Model

All models use the GPT-3 architecture from Brown et al. (2020), with context lengths of 2,048 tokens. Three model sizes are trained: 1.3B, 6B, and 175B parameters. The models use fp16 weights and activations with fp32 master copies, and the same byte-pair encodings as GPT-3. The Adam optimizer is used with $\beta_1 = 0.9$ and $\beta_2 = 0.95$.

Step 1: Supervised Fine-Tuning (SFT)

The SFT model is fine-tuned on approximately 13,000 training prompts (from the API and labeler-written), for 16 epochs with cosine learning rate decay and residual dropout of 0.2. Despite overfitting on validation loss after 1 epoch, training for more epochs helps both the RM score and human preference ratings. Final SFT model selection is based on the RM score on the validation set.

• 1.3B and 6B models: LR = 9.65e-6, batch size = 32
• 175B model: LR = 5.03e-6, batch size = 8

For the RLHF initialization, models are fine-tuned for 2 epochs on the demonstration dataset with 10% pretraining data mixed in, which helps downstream PPO training stability.

Step 2: Reward Model (RM) Training

The RM is initialized from the SFT model (with the final unembedding layer replaced by a scalar projection), using the 6B model size exclusively for all PPO runs. The 175B RM was found to be unstable and too costly for use as a value function during RL.

To handle the correlations among comparisons from the same prompt, all $\binom{K}{2}$ comparisons are treated as a single batch element rather than independent data points. This critical implementation choice prevents overfitting and dramatically improves validation accuracy and log loss.

The RM loss function is:

where $r_\theta(x, y)$ is the scalar reward for prompt $x$ and completion $y$, $y_w$ is the preferred completion, $y_l$ is the less preferred completion, and $D$ is the dataset of human comparisons. The reward model is normalized so that labeler demonstrations achieve a mean score of 0 before RL training.

Training details: single epoch over the full RM training set, LR = 9e-6, cosine schedule, batch size = 64 distinct prompts. Each batch can contain up to $64 \times \binom{K}{2} \leq 2{,}304$ pairwise comparisons.

Step 3: RL Fine-Tuning with PPO

The environment is a contextual bandit: a random customer prompt is presented, the policy generates a response, and the RM outputs a scalar reward to end the episode. A per-token KL penalty from the SFT model is added to each token to prevent reward hacking and over-optimization of the reward model.

For PPO models (without pretraining mix), $\gamma = 0$ and the objective is:

For PPO-ptx models (with pretraining mix), an additional term is included to mitigate performance regressions on public NLP datasets:

where $\pi_\phi^{\mathrm{RL}}$ is the learned RL policy, $\pi^{\mathrm{SFT}}$ is the supervised model, and $D_{\mathrm{pretrain}}$ is the pretraining distribution. The KL penalty coefficient is $\beta = 0.02$ and the pretraining loss coefficient is $\gamma = 27.8$. Unless otherwise noted, InstructGPT refers to PPO-ptx models.

PPO training details: 256k episodes, batch size = 512, minibatch size = 64 (8 minibatches per batch, single inner epoch), constant LR with warmup over the first 10 iterations, exponential moving average with decay 0.992, no discount for GAE, PPO clip ratio = 0.2, sampling temperature = 1. Pretraining data ratio = 8× the number of RL episodes.

Dataset and Task Details

Prompt Sources

The prompt dataset primarily consists of text submitted to the OpenAI API Playground by users interacting with early InstructGPT beta models. Customers were notified that their prompts could be used for training. Production API usage (non-Playground) is excluded. Deduplication is applied by checking for long common prefixes, and each user/organization is limited to ~200 prompts. Train/validation/test splits are organization-ID-based to prevent leakage. All training prompts are filtered for personally identifiable information (PII).

To bootstrap the first InstructGPT model (before API data existed), labelers wrote three kinds of prompts:

• Plain: Arbitrary tasks with enforced diversity.
• Few-shot: An instruction with multiple query/response pairs, formatted as few-shot examples.
• User-based: Prompts corresponding to stated use-cases from API waitlist applications.

Dataset Sizes

The SFT dataset contains ~13k total prompts, the RM dataset ~33k, and the PPO dataset ~31k. The RM training data yields an order of magnitude more pairwise comparisons because of the $\binom{K}{2}$ expansion per prompt.

Task Distribution

About 96% of the dataset is English. The use-case distribution for the RM dataset is:

Open-ended generation and brainstorming together account for ~57% of prompts, while classification and QA make up only ~18%. This composition differs sharply from standard NLP benchmarks, which largely feature classification and QA tasks. Tasks are specified via natural language instructions, few-shot examples, or implicit continuation.

Human Data Collection

About 40 contractors were hired via Upwork and ScaleAI. Unlike prior RLHF work focused on summarization, this project spans a much broader and occasionally sensitive range of topics, so a screening process was used to select labelers skilled at identifying sensitive content and ranking outputs helpfully. The screening tested:

1. Agreement on sensitive speech flagging (soft cutoff: 75% agreement).
2. Agreement on output rankings vs. researcher judgments.
3. Quality of sensitive demonstration writing (Likert score ≥ 6/7).
4. Self-assessed ability to identify sensitive speech across cultural groups.

Labelers were kept small in number (~40) to enable high-bandwidth communication and consistent feedback. Labeler demographics (19 survey respondents): 50% male, 44% female, 5.6% nonbinary; 52.6% Southeast Asian, 31.6% White/Caucasian; 75% under 35 years old; 52.6% hold undergraduate degrees, 36.8% master's degrees.

Inter-annotator agreement: Training labelers agree 72.6 ± 1.5% of the time. Held-out labelers agree 77.3 ± 1.3%. This is comparable to the 73 ± 4% researcher-researcher agreement reported in Stiennon et al. (2020) for summarization.

During training data collection, labelers were instructed to prioritize helpfulness. During final evaluations, labelers were instructed to prioritize truthfulness and harmlessness. Labeler metadata collected per output includes: overall quality (1–7 Likert), binary flags for instruction following failure, hallucination, harmful advice, sexual/violent content, denigration of protected classes, moral judgment, and several others.

A separate set of "held-out" labelers — sourced from the same vendors but without screening — was used to test generalization of model preferences beyond training labelers.

Baselines and Comparisons

• GPT-3: The raw 1.3B, 6B, and 175B pretrained models with no fine-tuning.
• GPT-3 (prompted): GPT-3 with a hand-crafted few-shot prefix designed to elicit instruction-following behavior (chosen through a prefix-finding competition).
• SFT models: GPT-3 fine-tuned only on labeler demonstrations via supervised learning.
• PPO models: SFT models further fine-tuned via RLHF without pretraining data mix ($\gamma=0$).
• PPO-ptx models (InstructGPT): PPO models with pretraining data mixed into RLHF fine-tuning ($\gamma=27.8$).
• FLAN and T0: GPT-3 175B fine-tuned on the FLAN and T0++ public NLP instruction datasets (~1 million examples each), chosen by RM score on the validation set.

Evaluation

Human Preference Evaluations

The primary evaluation metric is human preference ratings on held-out prompts from the same distribution as training data (but from users not represented in training). Labelers judge which of two outputs (one from the model under evaluation, one from a 175B SFT baseline) is preferred. Win rates are computed against this baseline. Evaluations are also conducted on prompts submitted to GPT-3 models (which are generally not instruction-formatted) to test robustness.

Secondary evaluations include: Likert scale (1–7) overall quality scores, and a full suite of binary metadata labels per output.

Automatic NLP Benchmarks

Evaluations cover: Winogender (gender bias entropy), CrowS-Pairs (stereotype bias entropy), RealToxicityPrompts (Perspective API toxicity scores), TruthfulQA (% truthful / % truthful and informative), HellaSwag (common-sense completion accuracy), WSC (Winograd schema accuracy), RTE (textual entailment accuracy), SST (sentiment accuracy), QuAC (QA-in-context F1), SQuADv2 (reading comprehension F1), DROP (discrete reasoning F1), WMT 2015 Fr→En (BLEU), CNN/DM Summarization (ROUGE-L), and Reddit TLDR Summarization (ROUGE-L).

All sampling is at temperature $T=0$, truncated at the first newline. Multiple-choice answers are selected by lowest average per-token log probability at $T=1$.

Results

Human Preference: InstructGPT vs. GPT-3

Key preference results:

• The 175B InstructGPT outputs are preferred over 175B GPT-3 outputs 85 ± 3% of the time.
• The 175B InstructGPT outputs are preferred over few-shot 175B GPT-3 71 ± 4% of the time.
• The 1.3B InstructGPT model is preferred over the 175B GPT-3 baseline — a 100× parameter reduction with better perceived quality.
• Improvements stack sequentially: GPT-3 → GPT-3 (prompted) → SFT → PPO → PPO-ptx, each step improving preference scores.
• Adding pretraining mix (PPO-ptx vs. PPO) does not significantly change labeler preference but does fix NLP regressions.

Metadata Ratings

Across several metadata axes, InstructGPT is superior to GPT-3:

• More appropriate in a customer assistant context.
• Better at following explicit constraints (e.g., "write your answer in 2 paragraphs or less").
• Less likely to fail the correct instruction entirely.
• Lower hallucination rate on closed-domain tasks (21% vs. 41% for GPT-3).

Generalization to Held-Out Labelers

Held-out labelers (not involved in training data collection) show similar preferences to training labelers — InstructGPT models consistently outperform GPT-3 baselines even for these unseen evaluators. A 5-fold cross-validation experiment on the reward model confirms generalization: RMs achieve 72.4 ± 0.4% accuracy on training-set labeler preferences and 69.6 ± 0.9% on held-out labeler preferences.

Comparison to FLAN and T0

FLAN and T0 fine-tuned versions of 175B GPT-3 perform slightly better than raw GPT-3 but significantly worse than InstructGPT. In head-to-head comparisons: InstructGPT outputs preferred over FLAN 78 ± 4% of the time; over T0 79 ± 4% of the time. The authors attribute this to (1) public NLP datasets being skewed toward classification and QA (~18% of API use), while generation and brainstorming dominate (~57%), and (2) limited diversity in public dataset inputs compared to real-world usage.

Truthfulness

On TruthfulQA, PPO models show significant improvements in generating truthful and informative outputs compared to GPT-3. PPO models perform well by default (without special truthfulness instructions). With a helpful instruction prompt ("respond with 'I have no comment' when uncertain"), PPO models err toward being uninformative rather than confidently wrong — a desirable behavior. The exception is the 1.3B PPO-ptx model, which performs slightly worse than the same-size GPT-3. These truthfulness gains are corroborated by the ~50% reduction in hallucination rate on closed-domain tasks (from 41% to 21%).

Toxicity

InstructGPT generates about 25% fewer toxic outputs than GPT-3 when prompted to be respectful, according to the Perspective API. However, this advantage disappears without the respectful prompt. Critically, when explicitly prompted to generate toxic output, InstructGPT is more toxic than GPT-3 — because InstructGPT is highly instruction-following, including on harmful instructions. This underscores the limitation of training primarily for helpfulness.

Human evaluations confirm these patterns: InstructGPT is less toxic in the "respectful" condition and comparable in the "no prompt" condition. The SFT baseline has the lowest toxicity but also poor continuity, suggesting it may produce degenerate (very short) outputs.

Bias

On Winogender and CrowS-Pairs, bias is measured using the entropy of probability distributions over binary sentence pairs (higher entropy = less preference = less bias). InstructGPT does not significantly improve over GPT-3 on bias metrics. PPO-ptx shows similar entropy to GPT-3. Interestingly, when instructed to be "respectful," PPO-ptx models exhibit lower entropy (higher certainty), suggesting the model becomes more opinionated in its choices — though the direction of that preference is not uniformly stereotypical.

Performance on Standard NLP Benchmarks and the Alignment Tax

PPO fine-tuning introduces an "alignment tax" — performance regressions on SQuAD, DROP, HellaSwag, and WMT Fr→En translation compared to GPT-3. PPO-ptx largely mitigates these regressions and even surpasses GPT-3 on HellaSwag. However, gaps remain on DROP, SQuADv2, and translation in some settings.

Pretraining Mix vs. KL Coefficient for Fixing Regressions

Mixing pretraining gradients into PPO updates ($\gamma \geq 20$) substantially recovers performance on DROP and SQuAD. In contrast, increasing the KL coefficient — even to 100× the default — does not fully recover these regressions and causes significant drops in validation reward. This demonstrates that pretraining data distribution is essential for preserving capabilities, not merely staying close to the initial SFT model.

Ablations

Additional ablation findings:

• PPO initialization: SFT models with 10% pretraining data mix perform best as PPO initialization; the training duration (1 vs. 2 epochs) has minimal effect.
• KL coefficient: Optimal $\beta \approx 0.01$–$0.02$. Values of 0 or 2 lead to poor human Likert scores.
• Pretraining data ratio: A ratio of 8× RL episodes was chosen as a balance between training speed and pretraining loss performance. Ratio of 32 gives marginally better Likert scores at much higher compute cost.
• Training duration: 256k episodes is sufficient; longer training causes regressions on DROP and SQuADv2.
• Batch size: Optimal batch size = 512, minibatch size = 64 (or 32, though 64 was used for GPU utilization).
• Learning rates: All runs with LR > 8.05e-6 diverged for PPO without pretraining mix; PPO-ptx is more robust to LR choices.

Qualitative Results and Generalization

InstructGPT shows promising generalization beyond the fine-tuning distribution:

• Non-English languages: Despite the dataset being ~99% English, InstructGPT can follow instructions in French, Swedish, German, Spanish, and other languages, though it sometimes responds in English even when prompted in another language. GPT-3 generally fails to follow non-English instructions without careful prompting.
• Code tasks: InstructGPT can summarize code and answer questions about code more reliably than GPT-3, though it is far from perfect.

InstructGPT also has notable failure modes:

• False premises: When given a question with a false premise (e.g., "Why is it important to eat socks after meditating?"), InstructGPT often accepts the premise as true and answers accordingly rather than challenging it.
• Over-hedging: For simple factual questions, InstructGPT sometimes gives overly cautious multi-possibility answers instead of committing to the clearly correct response. This may stem from labelers rewarding epistemic humility, which gets picked up by the reward model.
• Multiple explicit constraints: Performance degrades when instructions contain many simultaneous constraints (e.g., "list 10 movies from the 1930s set in France with a female protagonist").
• Harmful compliance: InstructGPT follows harmful instructions more readily than GPT-3, because instruction-following is its primary trained behavior.

Automatic Evaluation Table (Key Results)

The table below summarizes selected automatic evaluation results across model families and sizes (XL = 1.3B, 6B, 175B):

Notable patterns: PPO models improve TruthfulQA substantially but regress on DROP and WMT. PPO-ptx recovers most regressions. On bias benchmarks (Winogender, CrowS-Pairs), SFT models show the strongest apparent alignment but this reflects overfitting-induced certainty rather than genuine debiasing.

Compute Costs and Efficiency

Training compute (in petaflop/s-days):

• GPT-3 175B pretraining: ~3,640 petaflop/s-days
• InstructGPT 175B SFT: ~4.9 petaflop/s-days
• InstructGPT 175B PPO-ptx: ~60 petaflop/s-days

Even the full PPO-ptx training is less than 2% of the cost of pretraining GPT-3, yet it produces a model preferred 85% of the time over GPT-3. The authors conclude that RLHF alignment is dramatically more cost-effective than scaling for improving performance on users' actual task distribution.

Labeling Interface

Discussion: Implications for Alignment Research

The authors draw four lessons for the broader alignment research agenda:

1. Low alignment tax is achievable. PPO-ptx largely eliminates performance regressions on public NLP datasets while substantially improving helpfulness — making RLHF a low-cost alignment technique compatible with deployment.
2. Generalization of instruction-following. InstructGPT shows alignment generalization to non-English languages and code tasks with minimal supervision, suggesting that RLHF can produce broadly generalizing alignment signals, not just task-specific ones.
3. Iterative, real-world alignment is tractable. Rather than studying alignment abstractly, this work embeds alignment research in a production AI system, providing a concrete feedback loop about what works and what fails.
4. RLHF as a building block for scalable oversight. RLHF is a component of several proposals for aligning superhuman AI systems (debate, amplification, recursive reward modeling). Demonstrating its efficacy at this scale validates it as a core primitive for future alignment work.

Who Are We Aligning To? Limitations of the Alignment Target

The paper is candid about the narrowness of its alignment target. The models are aligned to:

• Training labeler preferences: ~40 contractors, mostly English-speaking, primarily from the US and Southeast Asia, hired through Upwork and ScaleAI.
• Researcher preferences: The labeling instructions were written by OpenAI researchers, embedding implicit value judgments.
• API customer preferences: Prompts from early InstructGPT beta users, who are not representative of the broader population.

These groups do not represent the full spectrum of people who will use or be affected by these systems. Inter-labeler disagreement (~27%) highlights that a single model cannot represent all preferences simultaneously. The paper explicitly declines to claim that any of these groups constitute the "right" source of preferences, and calls for future work on more transparent, participatory, and pluralistic alignment processes.

Limitations

• Dataset bias: Labelers are primarily English-speaking; data is 96%+ English. Results may not generalize to other languages or cultural contexts.
• Single labeler per comparison: Most comparisons are labeled by only 1 labeler, limiting visibility into disagreements.
• Helpfulness prioritized over safety during training: Training instructs labelers to prioritize helpfulness, meaning the model follows even harmful instructions. Final evaluations flip this priority, creating inconsistency.
• Still not fully safe: Models still generate toxic, biased, or harmful outputs — especially when prompted to do so. They make up facts, can be confused by false premises, and can over-hedge.
• Remaining alignment tax: PPO-ptx doesn't fully recover DROP, SQuADv2, or translation performance; longer training causes further regressions.
• Small labeler pool: ~40 contractors is a convenience for communication bandwidth but clearly not representative of global user diversity.
• Value judgment dependency: Labeler decisions on sensitive prompts are inherently value-laden and influenced by identity, culture, and personal experience.

Open Questions and Future Directions

• Adversarial data collection: Systematically finding worst-case model behaviors for inclusion in training data could reduce harmful outputs.
• Combining RLHF with pretraining data filtering: Removing toxic content from pretraining data could reduce the model's baseline propensity for harmful generation.
• Refusals: Training models to refuse certain harmful instructions is important but requires context-dependent configurability at inference time.
• Alternative feedback modalities: Labelers could edit model responses, provide natural language critiques, or use richer interfaces — potentially more efficient than pairwise comparisons.
• Expert iteration and constrained optimization: Alternatives to PPO for the alignment RL step could yield better results.
• Aligning to instructions vs. values vs. interests: The paper aligns to inferred user intent, but Gabriel (2020)'s taxonomy suggests this is only one of many possible alignment targets; more research on principle-based alignment is needed.
• Scalability of human supervision: As models become more capable, it will be harder for humans to evaluate outputs directly. Approaches like recursive reward modeling and debate are identified as important future directions.

Broader Impacts

Making language models more instruction-following has dual-use implications. On the positive side, more helpful, truthful, and harmless models reduce alignment failure risks and improve user experience. On the negative side, more controllable models are also more easily misused: generating convincing misinformation, targeted harassment, or phishing content becomes easier when the model reliably follows instructions.

The paper argues that alignment techniques are not a panacea. They should be one component in a broader safety ecosystem. High-stakes domains (medical diagnosis, credit decisions, law enforcement, political advertising) require additional safeguards regardless of alignment improvements. The centralization/access tradeoff — between open-source (risky but democratizing) and API-only access (controllable but centralizing power) — remains unresolved and deeply contested.

Ultimately, the paper frames InstructGPT as a demonstration that fine-tuning with human feedback is a promising and cost-effective direction for aligning large language models with human intent, while being clear that much work remains to make these systems reliably safe, fair, and broadly beneficial across diverse human contexts.