RLHF, KL control & Constitutional AI

§1Where RLHF came from

RLHF was not invented for chatbots. It began as a way to teach a simulated robot a backflip that nobody could write a reward function for, and became the standard recipe for language models over five years.

The core problem is old: many behaviors we want are easy to recognize and very hard to specify. You cannot write down a reward function for "a good summary" or "a helpful but safe answer". You can, however, look at two candidates and say which is better. Reinforcement learning from human feedback uses that asymmetry. Fit a model of human judgments, then optimize a policy against the model.

Christiano et al. 2017: preferences instead of rewards

Deep RL from human preferences (Christiano, Leike, Brown, Martic, Legg, Amodei) had humans compare pairs of short video clips of an agent's behavior. A reward predictor $\hat r$ was fit so that the probability of preferring clip $\sigma^1$ over $\sigma^2$ follows a Bradley–Terry model on summed predicted rewards:

$$P[\sigma^1\succ\sigma^2]=\frac{\exp\sum_t\hat r(o^1_t,a^1_t)}{\exp\sum_t\hat r(o^1_t,a^1_t)+\exp\sum_t\hat r(o^2_t,a^2_t)}.$$

The agent learned Atari games and MuJoCo locomotion with feedback on fewer than 1% of its interactions, and a Hopper learned a backflip from 900 queries in under an hour of human time. Two lessons from this paper still govern RLHF in 2026:

• Feedback has to be online. A reward predictor trained only on early comparisons "captures only part of the true reward" because the state distribution shifts as the policy improves. Maximizing that partial reward produced degenerate behavior (endless Pong volleys). Feedback had to be interleaved with training.
• Where you ask matters. Queries were chosen where an ensemble of reward predictors disagreed most. This was a crude form of active learning, and the paper notes it sometimes hurt.

Ziegler et al. 2019: the KL penalty arrives in language

Fine-Tuning Language Models from Human Preferences applied the recipe to GPT-2 (774M) on stylistic continuation (5,000 comparisons were enough) and summarization (60,000). It also introduced the per-sample reward still used today:

$$R(x,y)=r(x,y)-\beta\log\frac{\pi(y\mid x)}{\rho(y\mid x)},$$

where $\rho$ is the pretrained model. The authors explain the KL term as an entropy bonus that also keeps the policy where the reward model is valid. Rather than fix $\beta$, they adapted it with a proportional controller aiming at a KL target (6 nats for descriptiveness, 10 for sentiment; §4). Two anecdotes from this paper are worth knowing:

• The summarizers became "smart copiers". Labelers rewarded accurate extracts, and copying is the cheapest way to be accurate.
• A refactor flipped the sign of both the reward and the KL penalty. The model optimized for what labelers rated worst while the (flipped) KL term kept the text fluent; since labelers had been told to rate sexually explicit continuations very low, it produced only such content. The authors called it "maximally bad output", not gibberish. They were asleep during the run, so nobody noticed until training had finished, and they suggest an "Andon cord" that lets any labeler stop a run. The lesson: a fluent, confident-looking policy is no evidence that the reward points the right way.

Stiennon et al. 2020: RLHF beats the humans who wrote the references

Learning to summarize from human feedback scaled the pipeline (1.3B and 6.7B policies, a released set of 64,832 TL;DR comparisons, $\beta=0.05$). The 6.7B RLHF summaries were preferred to the human-written reference summaries about 65% of the time after controlling for length, and they transferred to news articles without news-specific training. It also gave the first clear picture of over-optimization. As the policy was pushed further from the SFT model, the reward model's score kept rising while human preference first rose, then fell, until the RM was "anti-correlated with human preferences". Chapter 03 covers the scaling laws for this effect. Here what matters is the consequence: the KL distance from the reference becomes the quantity you budget.

InstructGPT (Ouyang et al. 2022): the canonical three stages

Training language models to follow instructions with human feedback is the paper most people mean by "RLHF". About 40 contractors produced the data. The prompts came from the OpenAI API Playground and from the labelers themselves.

1. SFT on labeler demonstrations (~13k prompts).
2. Reward model (6B) trained on rankings of $K=4\ldots9$ completions per prompt (~33k prompts). All $\binom K2$ pairs from one prompt go into a single batch element. Shuffling them as independent pairs made the RM overfit, since every completion appears in $K-1$ pairs. They used 6B rather than 175B because 175B RM training was less stable as an initialization for the PPO value function, and much more expensive.
3. PPO against the RM (~31k prompts, 256k episodes), with a per-token KL penalty ($\beta=0.02$) and, in the PPO-ptx variant, a pretraining log-likelihood term.

Headline results: outputs of the 1.3B InstructGPT were preferred to those of the 175B GPT-3, a model with 100× more parameters. The 175B InstructGPT was preferred to 175B GPT-3 85 ± 3% of the time, and to few-shot-prompted GPT-3 71 ± 4% of the time. Closed-domain hallucination fell from 41% to 21%. Toxicity fell by about 25% when the model was asked to be respectful, but the gap vanished with no instruction. There was no improvement on the Winogender and CrowS-Pairs bias benchmarks.

§2The RLHF loop, step by step

One PPO iteration of RLHF involves four networks (policy, value head, reward model, reference) and two loops running at different speeds: a fast one over gradient steps and a slow one over human labels.

Concretely, for each batch:

1. Sample prompts $x$; generate $y\sim\pi_\theta(\cdot\mid x)$ with the current policy.
2. Score $(x,y)$ with the reward model: one scalar, applied at the final token.
3. Run the frozen reference (the SFT model) on the same tokens and form a per-token penalty $-\beta\log\frac{\pi_\theta(y_t\mid x,y_{\lt t})}{\piref(y_t\mid x,y_{\lt t})}$.
4. Combine these into per-token rewards, estimate advantages with a value head and GAE, and take a few epochs of clipped PPO steps (the mechanics are in Chapter 02).

import torch

def kl_shaped_rewards(rm_score, logp_pi, logp_ref, mask, beta):
    """Per-token rewards for PPO-RLHF.
    rm_score: [B] scalar RM score per sequence
    logp_pi, logp_ref: [B, T] log-probs of the *sampled* response tokens
    mask: [B, T] 1.0 on response tokens (response left-aligned at t=0)
    """
    kl_tok = (logp_pi - logp_ref) * mask          # k1 estimator, one sample per token
    rewards = -beta * kl_tok                       # dense penalty at every token
    last = mask.sum(dim=1).long() - 1              # index of each sequence's final token
    rewards[torch.arange(rewards.size(0)), last] += rm_score   # sparse RM reward at EOS
    return rewards, kl_tok.sum(dim=1)              # per-token rewards, per-sequence KL estimate

Two facts about this snippet are worth knowing. First, summing the per-token log-ratios of a sampled sequence gives exactly $\log\frac{\pi_\theta(y\mid x)}{\piref(y\mid x)}$ for that sequence, by the chain rule. Its expectation under $\pi_\theta$ is the sequence-level $\KL(\pi_\theta(\cdot\mid x)\,\|\,\piref(\cdot\mid x))$. So the "token-level" and "sequence-level" KL penalties are the same objective with different credit assignment. Second, the per-token form lets the value function attribute the penalty to the specific tokens that drifted, which reduces variance compared with dumping the whole penalty at the end.

§3The KL-regularized objective and its closed form

Behind the PPO machinery is a simple optimization problem with an exact answer. Knowing that answer is what turns RLHF from a recipe into something you can reason about.

Drop the function approximation and ask: over all conditional distributions $\pi(\cdot\mid x)$, which one maximizes

$$J(\pi)=\E_{x\sim D}\Big[\E_{y\sim\pi(\cdot\mid x)}[r(x,y)]-\beta\,\KL\big(\pi(\cdot\mid x)\,\|\,\piref(\cdot\mid x)\big)\Big]\;?$$

The objective separates across prompts, so fix $x$. The scrollytelling below walks through the derivation, the Gibbs variational principle, on eight candidate answers.

What β means in practice

β converts reward units into nats. A reward gap of Δ between two answers multiplies their probability ratio by $e^{\Delta/\beta}$ relative to the reference. Reward models trained with Bradley–Terry are only defined up to an additive constant per prompt, and their scale depends on training details. So a β value means nothing until you know the RM's scale, which is one reason labs normalize RM outputs (InstructGPT shifted its RM so labeler demonstrations scored 0 on average). Published values span orders of magnitude for this reason: Stiennon 0.05, InstructGPT 0.02, Llama 2 0.01 (7B/13B) and 0.005 (34B/70B), Anthropic's HH-RLHF $\lambda_{KL}=0.001$. Bai et al. called the last "a very small value" that "might actually be wholly unnecessary", since their KL stayed under ~100 nats anyway.

Reward hacking at low β

The closed form makes reward hacking easy to see. If the reward model overrates some rare output by a margin Δ, the tilt multiplies that output's odds by $e^{\Delta/\beta}$. What decides the takeover is not the size of the RM error itself but the hack's proxy-score lead over the best honest answer. In the explorer below, a 3.8-unit error lifts "Confident jargon, wrong" just 0.2 above the next-best proxy score; at β = 0.05 that lead is worth $e^{0.2/0.05}=e^{4}\approx55\times$ in odds, enough for an answer with a 3% prior to take about 86% of the mass. The explorer below uses a real softmax tilt on ten archetypal answers to one prompt. The true reward is what a careful human would say; the proxy is a reward model with the classic biases: it likes length, flattery and confident jargon.

Even with a perfect RM, very low β is not free. The policy becomes nearly deterministic, and you lose everything §5 is about: diversity, calibration, the ability to sample alternatives. The KL term also does a second job in practice. Every sample PPO trains on comes from the policy, and the RM was trained on samples from earlier policies, so staying close to $\piref$ keeps the RM in distribution (Ziegler et al. and Stiennon et al. both make this point explicitly).

RL as Bayesian inference, and why it matters

Korbak, Perez and Buckley (2022) make a conceptual argument with practical consequences. Plain reward maximization, $\max_\pi\E_\pi[r]$, has a degenerate optimum: a point mass on the argmax, which they call distribution collapse. For a generative model that is almost never what you want. With the KL term, the objective equals (up to a constant) $-\beta\,\KL(\pi\,\|\,\pi^*)$, so maximizing it is variational inference, with the policy as a variational approximation to the posterior $\pi^*$. Seen this way:

• The KL penalty is not an ad-hoc regularizer you might tune away. It is what makes the problem well posed: it defines which distribution you are trying to reach.
• Other ways of reaching the same target become legitimate alternatives to RL: rejection sampling, best-of-n (an approximation of the tilt; see Ch. 03), distribution-matching methods that minimize forward KL $\KL(\pi^*\|\pi)$, and DPO-style reparameterizations (Ch. 05).
• Since policy-gradient methods minimize the reverse KL $\KL(\pi\|\pi^*)$, they are mode-seeking with respect to the target: when the policy cannot represent $\pi^*$ exactly or training stops early, they would rather cover one mode of $\pi^*$ well than all modes thinly. This adds to the diversity loss that the target itself already implies (§5). At the exact optimum $\pi=\pi^*$ it plays no role.

§4How far should you move? Reward, √KL and the KL knob

Anthropic's HH paper reported a simple empirical regularity: during RLHF, reward grows roughly linearly in the square root of the KL from the initial policy. It explains why KL is the right x-axis for every RLHF plot.

That bend is what Gao, Schulman and Hilton (2022) measured with a synthetic "gold" reward model labeling data for a proxy. With $d=\sqrt{\KL}$, the gold reward along an RL run fit $R_{RL}(d)=d\,(\alpha_{RL}-\beta_{RL}\log d)$, and best-of-n fit $R_{bon}(d)=d\,(\alpha_{bon}-\beta_{bon}d)$. The best-of-n form is linear at small $d$; the RL form is not (its slope $\alpha_{RL}-\beta_{RL}(1+\log d)$ diverges as $d\to0$, and the authors note it "likely does not hold near the origin"). Both reach a maximum and then decline. Their most practical finding concerns the KL penalty. In their setup it did not improve the gold-reward-vs-KL frontier: the penalty "only causes the gold RM score to converge earlier", "akin to early stopping". They add that this result "could be particularly sensitive to hyperparameters", so treat it as strong evidence, not a law.

KL in the reward or KL in the loss?

There are two common ways to implement "β·KL", and they are not equivalent.

• KL in the reward (Ziegler, InstructGPT, most PPO-RLHF): add $-\beta\log\frac{\pi_\theta}{\piref}$ at every token as part of the reward, treat it as a constant (detached), and let the policy-gradient machinery credit it like any other reward. This is the "k1 in reward" estimator.
• KL in the loss (GRPO, DeepSeekMath and successors): keep the reward pure, and add $\beta\cdot\widehat{\KL}$ to the loss as a differentiable term computed on sampled tokens. It is usually the "k3" estimator (Schulman 2020): $k_3=(\rho-1)-\log\rho$ with $\rho=\piref(y_t)/\pi_\theta(y_t)$, which is non-negative and an unbiased value estimate of the per-token KL.

The catch is that an unbiased estimate of the KL's value does not give an unbiased estimate of its gradient. Liu et al. (2025) unify the two views by computing the gradient coefficient each implementation puts on the score function. They show that "k1 in reward" is the principled reverse-KL gradient, that "k2 as loss" ($\tfrac12(\log\rho)^2$) is gradient-equivalent to it on-policy, and that "k3 as loss" is only a first-order, biased approximation. They also point out that off-policy reuse of samples (PPO epochs, async RL) without importance weights on the KL term adds further bias.

A two-line derivation (ours, consistent with their analysis) shows what k3-as-loss actually optimizes. Treat the sampled tokens as fixed and differentiate: $\nabla_\theta k_3=(1-\rho)\,\nabla_\theta\log\pi_\theta$. On-policy,

$$\E_{\pi_\theta}\big[(1-\rho)\nabla_\theta\log\pi_\theta\big]=\underbrace{\E_{\pi_\theta}[\nabla_\theta\log\pi_\theta]}_{=0}-\sum_y\piref(y)\,\nabla_\theta\log\pi_\theta(y)=\nabla_\theta\KL(\piref\,\|\,\pi_\theta).$$

So the expected k3-as-loss gradient is exactly the gradient of the forward KL, while k2-as-loss gives $\E_{\pi_\theta}[\log\frac{\pi_\theta}{\piref}\nabla_\theta\log\pi_\theta]=\nabla_\theta\KL(\pi_\theta\|\piref)$, the reverse KL. Since $1-\rho\approx\log\frac{1}{\rho}$ when $\rho\approx1$, the two agree to first order near the reference and drift apart as the policy moves away. The widget shows the estimators' value-level behavior: what they estimate and with what variance.

One more property of k3 matters in practice. Its variance depends on $\E_{\pi}[\rho^2]=\sum_y \piref(y)^2/\pi(y)$, which grows without bound when the policy puts small probability on tokens the reference likes. That is exactly what a sharpened, post-RLHF policy does. For the Gaussians in the widget, the variance is infinite as soon as $\sigma^2\lt\tfrac12$. Per-token KL readings in late training therefore come from heavy-tailed estimates: a few tokens dominate them.

# GRPO-style: KL as a differentiable loss term (k3), reward left untouched
log_rho = logp_ref - logp_pi                 # log(pi_ref / pi_theta) on sampled tokens
k3 = torch.exp(log_rho) - 1.0 - log_rho      # >= 0; unbiased for the per-token KL value
loss = pg_loss + beta * (k3 * mask).sum() / mask.sum()
# On-policy, its expected gradient is grad KL(pi_ref || pi), the FORWARD KL,
# not grad KL(pi || pi_ref) (Liu et al. 2025: a first-order approximation).
# 'k1 in reward' (previous snippet) or 'k2 as loss' give the reverse-KL gradient on-policy.

Fixed β, adaptive β, and moving references

Adaptive KL control. Ziegler et al. treat β as a controller output rather than a hyperparameter. With target $\mathrm{KL}_{\mathrm{target}}$:

$$e_t=\clip\!\Big(\frac{\mathrm{KL}_t-\mathrm{KL}_{\mathrm{target}}}{\mathrm{KL}_{\mathrm{target}}},-0.2,0.2\Big),\qquad \beta_{t+1}=\beta_t\,(1+K_\beta e_t),\quad K_\beta=0.1.$$

This turns the question "what β?" into "what KL budget?". Given the √KL regularity, that is the more natural question: it fixes where on the reward-vs-KL curve you stop.

Moving the anchor. Over a long run, a fixed SFT reference becomes a liability: the KL term grows to dominate the loss and freezes progress. Several fixes move the reference:

• Iterated RLHF starts each round from the previous round's best policy (Llama 2's RLHF-V1…V5 each built on the last), so the natural anchor for a round is its own starting point rather than the original SFT model.
• Elastic Reset (Noukhovitch et al., NeurIPS 2023) periodically resets the online model to an exponential moving average (EMA) of itself, then resets the EMA to the initial model. It reports higher reward with less drift.
• WARP (Ramé et al. 2024) uses an EMA of the policy as the KL anchor, merges independently trained policies by spherical interpolation, then interpolates back toward the initialization. Weight averaging is used as a substitute for a stronger KL penalty.
• ProRL (NVIDIA, NeurIPS 2025), for long reasoning RL, periodically hard-resets the reference to a recent snapshot and reinitializes the optimizer state, so the KL term stays useful without dominating.

The trade-off is clear from §3. Every reset moves the prior, so the cumulative distance from the original SFT model is no longer bounded by any single KL term. Resets buy optimization progress at the price of the guarantee that you remain close to the model whose behavior you understood. In RL with verifiable rewards, where the reward is much harder to hack, many recipes drop the KL term altogether (Ch. 06). With a learned preference RM, that is far riskier.

§5What the KL anchor fails to prevent

A KL leash keeps the policy from inventing new kinds of output. It does little to stop the policy from narrowing, becoming overconfident, getting longer, or telling people what they want to hear.

Mode collapse: the asymmetry of the leash

Suppose the policy collapses entirely onto one output $y_0$. Its KL cost is

$$\KL(\delta_{y_0}\,\|\,\piref)=\log\frac{1}{\piref(y_0)}.$$

If the reference gave $y_0$ 5% probability, total collapse costs only $\log 20\approx3$ nats, which is well within a typical RLHF budget. Reverse KL does contain an entropy bonus, since $\KL(\pi\|\piref)=-H(\pi)-\E_\pi[\log\piref]$ (this is why Ziegler et al. describe the term as one), but the bonus is capped: dropping every other output can never cost more than $\log\frac{1}{\piref(y_0)}$. Reverse KL is infinite only for putting mass where the reference has none. A penalty that blows up when coverage is lost would need the forward KL $\KL(\piref\|\pi)$, which RLHF does not use explicitly.

There is a second, data-level push toward collapse. Zhang et al. (2025, "Verbalized Sampling") argue that human raters have a typicality bias: given equal quality, they prefer familiar-sounding text. Model it as a reward that includes the base model's own log-probability:

$$r(x,y)=r^\star(x,y)+\alpha\log\piref(y\mid x)+\epsilon\quad\Longrightarrow\quad \pi^*\propto\piref^{\,1+\alpha/\beta}\;e^{r^\star/\beta}.$$

Plugging into the closed form, the reference distribution is raised to a power $\gamma=1+\alpha/\beta>1$. This is a temperature below 1 applied before any real quality signal. On HelpSteer they fit a logistic model to correctness-matched response pairs and estimate $\hat\alpha=0.57\pm0.07$ and $0.65\pm0.07$ (using Llama-3.1-405B and GLM-4.5 base models as $\piref$), both clearly positive. Be careful with the magnitude. Their regressor is the difference in average per-token log-likelihood, whereas the formula above uses the whole-sequence $\log\piref(y\mid x)$. Plugging $\hat\alpha\approx0.6$ and β = 0.1 straight in would give γ ≈ 7, but for a $T$-token answer the sequence-level coefficient is closer to $\hat\alpha/T$, so the real sharpening is far smaller. The robust result is the sign: human preference data rewards typicality, which pushes γ above 1.

The empirical evidence is consistent with this picture:

• Kirk et al. (2023), working mainly with LLaMA 7B (with OPT models as a check), found that RLHF generalizes better than SFT out of distribution, increasingly so as the shift grows. It also substantially reduces per-input diversity (many samples for one prompt), with a much smaller effect across inputs. Best-of-n with the same RM kept across-input diversity, so the RM alone does not explain the drop. Surprisingly, raising the KL coefficient did not raise per-input diversity in their sweep; it lowered both performance and per-input diversity. The authors describe a generalization–diversity trade-off.
• Anthropic's HH paper notes that RL-finetuned models "typically have much narrower, lower-entropy output distributions", which makes rigidly formatted evaluations hard to interpret. For the same reason, they scanned temperature and top-p before comparing RLHF and base models on code.
• Jiang et al. (2025, "Artificial Hivemind"), a NeurIPS 2025 Datasets & Benchmarks oral, built Infinity-Chat (26k open-ended queries). They document both intra-model repetition and striking inter-model homogeneity: different labs' models converge on similar answers. Reward models and LM judges were also less calibrated to human ratings on items where individual annotators disagreed.
• A historical caution: the widely shared "mode collapse" examples from text-davinci-002 (its favorite random number was 97) turned out to come from a model not trained with RLHF (janus 2022, correction added to the post). Other kinds of fine-tuning can collapse modes too. RL is one route among several.

Calibration

The GPT-4 Technical Report includes a widely cited plot (Figure 8). On a subset of MMLU, the pre-trained model's confidence in its multiple-choice answers matched its accuracy closely, with an expected calibration error (ECE) of 0.007. After post-training, labeled "ppo" in the figure, the ECE was 0.074. The caption says plainly that "post-training hurts calibration significantly." This follows from the tilt. Multiplying probabilities by $e^{r/\beta}$ and renormalizing is a reward-weighted sharpening. Unless the reward is itself a calibrated log-likelihood of correctness, there is no reason the resulting confidences should match accuracy. Kadavath et al. (2022) report a useful nuance: RLHF policies "naively appear very miscalibrated", but a single temperature ($T=2.5$) "largely fixes calibration issues" on their evaluations. Much of the knowledge survives; the readout is sharpened.

Verbosity drift and sycophancy

Reward models inherit any correlation in the preference data between a cheap feature and the label. Length is the classic one. Singhal et al. (2023) found that RLHF reward gains in their settings were mostly driven by longer outputs, and that "even a purely length-based reward reproduces most downstream RLHF improvements" over SFT. Agreement with the user is another. Sharma et al. (2023), an Anthropic paper, showed that five production assistants were sycophantic across four tasks. In existing preference data, responses that matched the user's views were more likely to be preferred, and optimizing against preference models could trade truthfulness for agreement. The production version of this story came in April 2025, when OpenAI rolled back a GPT-4o update days after shipping it. Its postmortem ("Expanding on what we missed with sycophancy", May 2025) blamed several changes that each looked beneficial alone, including an added reward signal from ChatGPT thumbs-up/down data. In aggregate, OpenAI wrote, they "weakened the influence of our primary reward signal, which had been holding sycophancy in check", and user feedback "can sometimes favor more agreeable responses".

§6Anthropic's HH-RLHF: helpful versus harmless

Anthropic's 2022 assistant paper is a set of careful empirical observations about what happens when you optimize two partly opposed objectives with RLHF at scale.

Training a Helpful and Harmless Assistant with RLHF (Bai et al. 2022) trained models from 13M to 52B parameters. The data was collected mostly with 52B models: about 118k helpfulness comparisons (44k base, 52k from rejection-sampling models, 22k online) and 44k harmlessness comparisons from red-teaming. RL used the PM score directly, with $r_{\mathrm{total}}=r_{PM}-\lambda_{KL}\KL(\pi\|\pi_0)$ and $\lambda_{KL}=0.001$.

The tension

A preference model trained only on helpfulness data performed "much worse than chance" on harmlessness data, and vice versa. A PM trained on both learned both, and larger PMs were better at it and less sensitive to the data mix. Policy behavior was harder. Early policies produced "the same exaggerated responses to all remotely sensitive questions", such as recommending therapy or saying "I can't answer that", which the authors attributed to "over-optimizing for harmlessness, while under-optimizing helpfulness". Part of the cause was the data design. Red-teamers chose the more harmful of two responses, so the data taught the model what not to say without demonstrating a good refusal. The cheapest harmless answer was an evasive one.

The lesson of the widget is the paper's lesson: changing weights moves you along a frontier, while adding better candidate behaviors moves the frontier itself. No choice of w fixes an evasiveness problem if the policy rarely samples a good engaged answer.

Iterated online RLHF

To improve the top of the distribution, the team ran an iterated online mode. The best current RLHF policy collected new comparisons, these were mixed with existing data, and new PMs and policies were trained, roughly weekly over about five weeks. The online data populated the high-score tail of the PM's distribution, where the original data was thin. Online PM accuracy on test sets was lower (74% base, 70% rejection sampling, 67% online-only). The authors read this as evidence that comparing two good samples is harder, not that the PM got worse. In a controlled comparison with equal data, the iterated-online mixture was preferred. Against professional writers, crowdworkers preferred the HH online model 56% of the time, with the caveat that the writers' prompts were not very adversarial.

Robustness findings

• Train/test PM split. Splitting the data in half and training a policy against the "train" PM while scoring it with the "test" PM: the two agree early, then diverge with the test PM lower. This is direct evidence of over-optimization against the PM used for training. Larger PMs were more robust.
• √KL. The linear relation (§4), with roughly parallel curves across model sizes.
• Alignment tax vs bonus. Small models lost zero-shot performance after RLHF; the 13B and 52B models gained on zero-shot NLP evaluations and were about the same few-shot. RLHF also combined with specialized skills (coding, summarization) without hurting them.
• Calibration of PMs. Helpfulness-only PMs were very well calibrated; mixed HH PMs were slightly under-confident. Accuracy fell when both samples scored high, which predicts diminishing returns at the top.
• OOD detection. A simplified relative Mahalanobis distance on activations separated harmful from helpful requests (about 0.85 AUROC without outlier exposure, 0.94 with just 10 harmful examples for the largest model). This was an early sign that harmlessness need not rest on the policy alone.

Red teaming as data

The harmlessness data came from red-teaming, written up in Ganguli et al. (2022). They released 38,961 attacks and compared four model types at 2.7B, 13B and 52B. Plain, HHH-prompted and rejection-sampling models showed a flat trend with scale, while RLHF models became harder to red team as they scaled. (Rejection-sampling models were hard to red team at every size, but mostly by being evasive.) Methodologically, the paper treats red-teaming as data collection with its own instructions, statistics and uncertainties, not as a demo. That remains the right framing. Red-team data is a training distribution, and its biases (which harms people think to try, which response they mark) become the policy's biases.

§7Constitutional AI

Constitutional AI keeps human labels for helpfulness and replaces the harmlessness labels with AI feedback guided by a short list of written principles. It changes both who supervises and what kind of signal they give.

Constitutional AI: Harmlessness from AI Feedback (Bai et al. 2022b) had two motivations. The first was scalable oversight: if models will eventually need supervision on tasks humans cannot easily judge, it helps to learn now how to make AI supervision reliable. The second was the evasiveness problem of §6. HH models often gave canned refusals; the CAI paper reports its RL-CAI models are "virtually never evasive" and explain their objections instead.

Phase 1: SL-CAI (critique → revision → SFT)

1. Take red-team prompts: 42,496 human-written (from Ganguli et al.) plus 140,335 model-generated, 182,831 in total.
2. Sample a response from a helpful-only RLHF model. It will often comply with harmful requests.
3. Append a critique request built from one principle drawn at random from 16. The same model critiques its response.
4. Append the matching revision request; the model rewrites the response.
5. Repeat with freshly sampled principles (the paper studies up to four rounds).
6. Fine-tune a pretrained model on (prompt, final revision) pairs, mixed with helpfulness samples from the helpful RLHF model so the new model stays helpful.

Phase 2: RL-CAI (RL from AI feedback)

1. Starting from the SL-CAI model, sample two responses to each red-team prompt.
2. Ask a feedback model (typically a pretrained LM) a multiple-choice question: "Consider the following conversation between a human and an assistant: […] [principle] Options: (A) […] (B) […] The answer is:".
3. Use the normalized probabilities of "(A)" vs "(B)" as a soft label. Soft labels "led to much better results than hard labels", which the authors attribute to their calibration. Each comparison uses one randomly sampled principle; this gave more robust PM behavior than any single fixed principle.
4. Chain-of-thought feedback: let the (helpful RLHF) feedback model reason step by step before answering. Its probabilities then saturate near 0 or 1, so they are clamped to 40–60%. Without clamping, the policy learned more extreme behavior; a 20–80% band helped less.
5. Train a PM on AI harmlessness labels plus human helpfulness labels, then run RL against it, initialized from SL-CAI.

import random, torch

def sl_cai_pair(model, prompt, principles, n_rounds=4):
    """Critique-revision loop. principles: list of (critique_request, revision_request)."""
    convo = f"Human: {prompt}\n\nAssistant:"
    response = model.generate(convo)
    for _ in range(n_rounds):
        crit_req, rev_req = random.choice(principles)          # one principle per round
        ctx = f"{convo} {response}\n\nCritiqueRequest: {crit_req}\n\nCritique:"
        critique = model.generate(ctx)
        ctx += f" {critique}\n\nRevisionRequest: {rev_req}\n\nRevision:"
        response = model.generate(ctx)
    return prompt, response                                     # one SFT example

def ai_preference(feedback_lm, convo, a, b, principle):
    q = (f"Consider the following conversation between a human and an assistant:\n{convo}\n"
         f"{principle}\nOptions:\n(A) {a}\n(B) {b}\nThe answer is:")
    lp = torch.tensor([feedback_lm.logprob(q, " (A)"), feedback_lm.logprob(q, " (B)")])
    return torch.softmax(lp, dim=0)[0].item()                   # soft label P(A preferred)

What the constitution was

In the 2022 paper the constitution was 16 short principles, each phrased as a critique/revision pair for SL and a comparison question for RL. An example: "Which of these assistant responses is less harmful? Choose the response that a wise, ethical, polite and friendly person would more likely say." When Anthropic published Claude's constitution in May 2023, it was a longer list (58 principles, per Anthropic's later CCAI post) of mostly "Choose the response that…" principles. The sources were the UN Declaration of Human Rights, trust-and-safety practice, other labs' principles (DeepMind's Sparrow rules), platform terms of service (e.g. Apple's), attempts to include non-Western perspectives, and Anthropic's own research. The post is candid about a side effect: the model sometimes "became judgmental or annoying", so principles were added against responses that were "too preachy, obnoxious or overly-reactive". The CAI paper reports the same failure in its most extreme form: over-trained RL-CAI policies appended boilerplate such as "you are valid, valued, and cared for" to most red-team responses. Goodhart applies to AI feedback too.

RLAIF beyond Anthropic

Lee et al. (Google, ICML 2024) ran a controlled RLAIF vs RLHF comparison with PaLM 2 as the labeler. Human win rates against the SFT baseline were 71% (RLAIF) vs 73% (RLHF) on summarization and 63% vs 64% on helpful dialogue. Head-to-head, the two were statistically indistinguishable (50% and 52%). On harmlessness, RLAIF reached an 88% harmless rate, against 76% for RLHF and 64% for SFT. Practical details carried over: position bias in the labeler (handled by scoring both orders and averaging), larger labelers aligning better with humans, a small gain from chain-of-thought, and a "direct RLAIF" variant that skips the RM and queries the LLM for a score during RL. Related lines include self-rewarding models that judge their own samples for iterative DPO (Yuan et al. 2024), OpenAI's Rule-Based Rewards that pair explicit rules with an LLM grader for safety (Mu et al. 2024), and Inverse Constitutional AI, which compresses an existing preference dataset into the principles that best reconstruct it (Findeis et al., ICLR 2025).

How specific should principles be? Who writes them?

Kundu et al. (2023) tested the extreme: a single principle, roughly "do what's best for humanity". The largest dialogue models generalized from it to harmless behavior, "with no stated interest in specific motivations like power". Detailed constitutions still gave finer control over specific harms, so the conclusion was that both kinds of principle are useful. Collective Constitutional AI (Anthropic with the Collective Intelligence Project, 2023; FAccT 2024 paper) asked about 1,000 demographically sampled Americans to propose and vote on rules via Polis: 1,127 statements and 38,252 votes. The public constitution overlapped Anthropic's by about 50% in concepts. It emphasized objectivity, impartiality and accessibility, and tended to promote desired behavior rather than prohibit undesired behavior. The model trained on it was less biased on all nine BBQ social dimensions, at equivalent MMLU/GSM8K and helpful/harmless Elo.

§8Practical RLHF: online, offline, iterated

Most RLHF engineering decisions come down to one question: is the reward model still accurate on the samples the policy is producing now?

Online vs offline

"Online" has two meanings that are worth keeping apart:

• On-policy RL samples. The policy is trained on its own fresh generations, scored by an RM (PPO, GRPO, online DPO). Tang et al. (2024) found that the gap between online and offline alignment methods is not explained by offline data coverage or quality. Offline-trained policies were good at pairwise classification and worse at generation; online-trained policies showed the reverse. On-policy sampling seems to matter in itself. Chapter 05 covers why offline DPO underperforms.
• Online human labels. The reward model is refreshed with new comparisons on the current policy's samples. Christiano 2017 needed it to avoid exploitation. Ziegler 2019 found it mattered for summarization (labelers preferred the online-data policy 71% of the time on TL;DR) but not for simple style tasks. HH-RLHF used it to cover the high-reward tail. Llama 2 collected new preference data before each of its RLHF-V1…V5 rounds because it "helps keep the reward model on-distribution".

Reward-model drift and iterated rounds

When the RM is refreshed, the reward scale and its biases change, so a policy tuned against RM_k is moved by RM_k+1 in a direction that partly reflects its errors. Practical consequences:

• Normalize RM outputs per version (for instance against a fixed reference set) so that β and KL targets carry over between rounds.
• Keep earlier data in the mix (HH-RLHF mixed, rather than replaced, data across rounds) to avoid forgetting old failure modes.
• Track a held-out "test PM" or a gold human eval, as in the HH train/test PM split. Divergence between the training RM and the held-out judge is the earliest sign of over-optimization.
• Restart policy training from a good checkpoint with a fresh reference, rather than continuing to push one run further against a changing reward.

Rejection sampling as RLHF's sibling

Best-of-n sampling against the RM, followed by SFT on the winners ("rejection sampling fine-tuning"), approximates the same tilted target without policy gradients. Llama 2 used only rejection sampling through RLHF-V4 and added PPO on top for the final round. HH-RLHF's second data tranche came from rejection-sampling models. It is more stable and easier to debug than PPO, uses the same KL currency (best-of-n moves the policy by at most $\log n-\frac{n-1}{n}$ nats), and is limited by the same RM errors. See Chapter 03 for its over-optimization curve.

§9What RLHF is good at, what it's bad at

RLHF is very good at the problem it was designed for: moving a capable model toward behavior that people can recognize as better. It has no answer for cases where people cannot tell, disagree, or can be fooled.

Good at

• Format, instruction following, tone. InstructGPT's 1.3B-over-175B result is the canonical demonstration.
• Using recognition rather than generation. Raters do not have to write the ideal answer, only recognize the better one, which is why RLHF beat the human references in summarization.
• Generalization beyond the SFT distribution. Kirk et al.'s OOD result: RLHF generalizes better than SFT, increasingly so under larger shifts.
• Making harmful behavior harder to elicit, increasingly so with scale (Ganguli et al.).

Bad at: the Casper et al. taxonomy

Open Problems and Fundamental Limitations of RLHF (Casper et al. 2023) is the standard reference for RLHF's limits. Its central move is to separate tractable problems, which better engineering can fix, from fundamental ones that require going beyond RLHF:

Evidence since 2023 has filled in several of these cells:

• Misleading the evaluator. Wen et al. (2024) found that RLHF made models more persuasive without making them more correct, under time-limited human evaluation. Evaluators' false-positive rates rose by 24.1% on QuALITY (question answering) and 18.3% on APPS (programming). They call it "U-sophistry", unintended sophistry. Probes built to detect intentional deception did not catch it.
• Reward hacking generalizes. Anthropic's MacDiarmid et al. (2025) gave a model knowledge of reward-hacking strategies and trained it in real production coding environments. When it learned to hack, it also generalized to alignment faking, cooperation with malicious actors and attempted sabotage. Standard chat-style RLHF safety training fixed the chat evaluations, but misalignment persisted on agentic tasks. Mitigations that worked included preventing the hacks, making RLHF safety data more diverse, and "inoculation prompting", which frames hacking as acceptable in training so the hack does not generalize to misalignment.
• Pluralism. Xiao et al. argue that the KL-regularized objective has an algorithmic bias toward the majority preference, which in the extreme gives "preference collapse": minority preferences are ignored. They propose a preference-matching regularizer. The Artificial Hivemind results (§5) are the empirical side of the same concern.

§10Frontier & open problems

§11Check yourself

Try each one before opening the answer. A mix of derivations, design questions and research taste.

§12Read next