Constitutional AI: Harmlessness from AI Feedback

Reading guide

Problem and motivation

Standard Reinforcement Learning from Human Feedback (RLHF) suffers from several bottlenecks when aligning language assistants:

• Heavy reliance on human harm labels: Collecting tens of thousands of human preference comparisons for harmfulness is expensive and exposes human annotators to toxic content.
• Evasiveness and tension with helpfulness: Models trained via RLHF often learn to answer sensitive prompts with canned refusals like “I can't answer that”, sacrificing helpfulness and transparency.
• Opacity: Thousands of private preference labels make the exact alignment objectives implicit and difficult to inspect or audit.

Constitutional AI (CAI) aims to scale supervision by replacing human harm annotations with explicit, natural language principles (a constitution), training an AI to self-critique, revise, and generate its own preference feedback.

Key ideas

• A Constitution of Principles: Behavior is steered entirely through a small set of natural language rules (roughly 16 principles) rather than human harm labels.
• Two-Stage Pipeline: A Supervised Learning (SL) stage brings model outputs on-distribution using self-critiques and revisions, followed by a Reinforcement Learning from AI Feedback (RLAIF) stage that trains a preference model on model-evaluated comparisons.
• Engagement Over Evasiveness: The assistant is guided to explain objections to harmful requests rather than issuing blanket refusals.
• Chain-of-Thought (CoT) Supervision: Leveraging step-by-step reasoning during evaluation substantially improves the model's ability to identify harms and evaluate multiple-choice preferences.

Method

1. Supervised Learning Stage (Critique $\to$ Revision $\to$ SL-CAI)

1. Sample an initial output from a helpful RLHF model using a red-teaming prompt $x$.
2. Prompt the model to critique its own response according to a randomly sampled constitutional principle: $$\text{Critique}\sim P_{\text{model}}(\cdot \mid x,y_{\text{initial}},\text{CritiqueRequest})$$
3. Prompt the model to revise the response to satisfy the critique: $$y_{\text{revised}}\sim P_{\text{model}}(\cdot \mid x,y_{\text{initial}},\text{Critique},\text{RevisionRequest})$$
4. Repeat critique-revision iteratively (typically up to 4 revisions).
5. Finetune a pretrained LM on revised responses paired with helpfulness demonstration data (182,831 red team prompts, 135,296 helpfulness prompts) for one epoch at batch size 1024.

2. Reinforcement Learning Stage (RLAIF)

1. Using the SL-CAI model, sample response pairs $(y_A,y_B)$ for prompts.
2. Present the conversation, response pair, and a sampled principle to a feedback model in a multiple-choice format (optionally using Chain-of-Thought prompting).
3. Compute normalized target probabilities from the feedback model's completion logits: $$P(A)=\frac{e^{z_A}}{e^{z_A}+e^{z_B}}$$ When using Chain-of-Thought, target probabilities are clamped to the range $[0.40,0.60]$ to avoid overconfidence.
4. Train a Preference Model (PM) using mixed data: 135,296 human-labeled helpfulness comparisons and 182,831 AI-generated harmlessness comparisons.
5. Optimize the SL-CAI policy against this preference model via RL.

Results

• Pareto Improvements: Crowdworker evaluations (based on 10,274 helpfulness and 8,135 harmlessness comparisons) show that 52B RL-CAI models achieve superior harmlessness Elo scores at equivalent helpfulness compared to models trained on human harmlessness labels.
• Reduced Evasiveness: When crowdworkers were explicitly instructed to prefer thoughtful explanations over evasive refusals, RL-CAI models consistently outperformed standard HH RLHF models.
• Scaling of AI Feedback: Larger models (52B) with Chain-of-Thought reasoning approached the accuracy of human-preference-trained models on HHH evaluation benchmarks (scoring well over 80–90% accuracy on binary multiple-choice harm identification).
• Critiques vs. Direct Revisions: Critiques yielded noticeable improvements in harmlessness scores for smaller models, while performing slightly better or equivalently on 52B models compared to direct revision.

Limitations and open questions

• Reliance on Human Helpfulness Labels: The experiments eliminated human feedback labels only for harmlessness; the helpfulness data still relied on human comparison labels.
• Goodharting and Over-Optimization: Extended RL training sometimes produced preachy or boilerplate language (e.g., repetitively stating “You are valid, valued, and cared for”).
• Calibration of CoT Targets: Because Chain-of-Thought reasoning causes the model to strongly favor one choice, probability clamping was required as a heuristic to maintain preference model stability.
• Dual-Use Concerns: Lowering the annotation barrier for steering language models also makes it easier to train models toward harmful objectives without human oversight.