Reward models & preference learning

§1Why learn a reward at all, and why from comparisons

RL needs a number for every sample. Humans can't produce numbers at RL speed, and the numbers they do produce don't mean the same thing from one person to the next. The reward model fixes both: it is a cheap, differentiable stand-in for a human judge, trained on the one kind of judgment humans give reliably, which is "this one is better than that one".

Chapter 02 assumed a reward function $r(x,y)$ and asked how to climb it. This chapter asks where $r$ comes from when nothing can check the answer: is this explanation clear, is this refusal appropriate, is this poem any good. The answer that has carried RLHF since Christiano et al. (2017) has three parts:

1. show people pairs (or small sets) of responses to the same prompt and record which one they prefer;
2. fit a model $r_\phi(x,y)$ whose score differences predict those preferences;
3. freeze $r_\phi$ and optimize a policy against it: with RL (chapter 04), by reranking samples (best-of-$n$, §5), or by filtering data for SFT (chapter 01).

Why comparisons and not scores? There are three reasons, and they compound. Calibration: one annotator's "7/10" is another's "5/10", and the same person drifts over a session. Absolute scores mix the annotator's private scale into the label. A comparison cancels any monotone per-annotator transform, because if both responses go through the same $f$, the order is unchanged. Difficulty: it is much easier to say which of two summaries is better than to place one summary on an absolute scale. Stiennon et al. note that their "decision to collect comparisons rather than Likert scores is supported by recent work". Statistics: as §2 shows, a comparison model gives you a proper likelihood, so you get maximum-likelihood estimation, calibration and uncertainty for free.

The whole loop, end to end. The rest of the chapter zooms into each box.

§2Bradley-Terry from scratch

Assume each response has a hidden "quality" $r$. Turn the difference into a probability with a sigmoid. Maximize the likelihood of the observed preferences. That's the whole model. It is logistic regression, it is Elo, and it is a special case of a 1920s psychophysics model.

The random-utility story

Suppose an annotator judging response $y$ to prompt $x$ perceives a noisy utility $u = r(x,y) + \varepsilon$ and picks whichever response has higher perceived utility. Then

The noise distribution decides the link function:

• Gaussian noise ($\varepsilon\sim\mathcal N(0,\sigma^2)$): the difference is $\mathcal N(0,2\sigma^2)$, so $P=\Phi\!\big(\tfrac{r_1-r_2}{\sqrt2\,\sigma}\big)$. This is Thurstone's law of comparative judgment (1927, Case V), a probit model.
• Gumbel noise (i.i.d. standard Gumbel): the difference of two Gumbels is standard logistic, so $P=\sigma(r_1-r_2)$. This is Bradley-Terry (1952), a logit model. In exponential form, with "strengths" $\pi_i=e^{r_i}$, $P(i\succ j)=\pi_i/(\pi_i+\pi_j)$.

Logistic and probit curves are almost identical after rescaling. BT won in practice because of what the Gumbel buys you: the maximum of i.i.d. Gumbel-perturbed utilities is softmax-distributed, which gives a clean extension to more than two items (Plackett-Luce, below) and a log-likelihood that is easy to differentiate.

The gradient tells you what the model learns from. With $\Delta=r_w-r_l$, $\frac{d}{d\Delta}\big[-\log\sigma(\Delta)\big] = -(1-\sigma(\Delta)) = -\sigma(-\Delta)$, so

Each pair pushes the winner up and the loser down, weighted by how likely the current model is to get the pair wrong. Pairs it already ranks confidently contribute almost nothing. Pairs it ranks confidently wrong contribute the most, and with noisy labels those are often the mislabeled ones. The noise-floor variant below changes exactly that.

Three identities worth knowing well

1 · It is logistic regression on difference vectors. If the RM is a linear head on frozen features, $r(x,y)=w^\top h(x,y)$, then $\Delta = w^\top\big(h(x,y_w)-h(x,y_l)\big)$. Fitting BT means fitting a logistic regression with no intercept on the feature differences. There is no intercept because the order of the pair is arbitrary: swapping $(y_w,y_l)$ must flip the prediction. Sharma et al. (2023) used this directly. A Bayesian logistic regression on 23 interpretable, LLM-generated features of Anthropic's HH comparisons reached 71.3% held-out accuracy, against ~72% for a 52B preference model trained on the same data. That is a sobering number about how much of "human preference" in that dataset is a handful of surface features.

2 · It is Elo. Chess Elo predicts $E_A = 1/\big(1+10^{(R_B-R_A)/400}\big)$, which is $\sigma\big(\tfrac{\ln 10}{400}(R_A-R_B)\big)$, a BT model with a different unit. After a game it updates $R_A\leftarrow R_A + K(S_A-E_A)$. The BT log-likelihood gradient for one game is $\tfrac{\ln 10}{400}(S_A-E_A)$, so Elo is online SGD on the Bradley-Terry likelihood, one game at a time, with a step size proportional to $K$ (exactly $K\cdot 400/\ln 10$ in the BT parametrization). Christiano et al. make the analogy explicit. Chatbot Arena later switched from online Elo to a batch BT fit "because the BT coefficients are better for the purpose of statistical estimation". The online version depends on game order and $K$; the MLE doesn't.

3 · The reward is identified only up to a per-prompt constant. Replace $r(x,y)$ with $r(x,y)+c(x)$ for any function $c$. Every difference $r(x,y_w)-r(x,y_l)$ is unchanged, so the likelihood is unchanged. Nothing in the data pins down whether prompt A's responses are "better" than prompt B's. Consequences:

• For best-of-$n$ it doesn't matter, since you only compare responses to the same prompt.
• For policy-gradient RL it mostly doesn't matter in expectation, because a prompt-dependent constant is a baseline (chapter 02). It does add variance, which is one reason pipelines normalize rewards. Stiennon et al. shift the RM so reference summaries average 0, and InstructGPT adds a bias so labeler demonstrations average 0 before RL. Group-relative advantages (GRPO, ch. 06) remove it per prompt by construction.
• For anything that compares rewards across prompts (filtering a dataset by an absolute RM threshold, reporting "average reward"), the number partly reflects which prompts happen to have high $c(x)$.
• The scale is identified, and it is tied to annotator noise. If labels are noisier, the MLE compresses reward differences toward zero. The fitter below shows this directly as the "scale" readout.

The same invariance is what makes DPO work. The KL-regularized optimum (chapter 04) satisfies $r(x,y)=\beta\log\frac{\pi^*(y\mid x)}{\piref(y\mid x)}+\beta\log Z(x)$. The intractable $\beta\log Z(x)$ is a per-prompt constant, so it cancels inside $\sigma(r_w-r_l)$. Plug the policy's log-ratio into the Bradley-Terry loss and you train the policy directly on the comparisons: the policy is the reward model (Rafailov et al. 2023, chapter 05). Everything in this chapter about noise, identifiability and spurious features applies to that implicit reward too, and RewardBench scores DPO checkpoints as reward models side by side with classifier RMs.

Rankings: Plackett-Luce

InstructGPT asked labelers to rank $K=4$ to $9$ responses at once. The Gumbel story extends naturally. The probability of a full ranking $\tau$ (best first) is the probability of picking $\tau(1)$ by softmax among all $K$, then $\tau(2)$ among the rest, and so on:

That is the Plackett-Luce model (Luce 1959; Plackett 1975). With $K=2$ it reduces to BT. InstructGPT did not use the full PL likelihood. It decomposed each ranking into its $\binom K2$ pairs, applied the BT loss to each with a $1/\binom K2$ weight, and, crucially, put all pairs from one prompt in the same batch element. Their reason: the pairs are highly correlated, and shuffling them as independent examples meant "a single pass over the dataset caused the reward model to overfit". Zhu, Jordan & Jiao (2023) show both estimators are consistent under PL, but the full PL MLE is asymptotically more efficient.

Ties, strength and noise

Real interfaces collect more than one bit, and how you use the extra information is a design choice:

• Ties. The simplest option is a soft label: Christiano et al. set the target distribution to uniform when the human marks two segments "equally preferable", which makes the loss $-\tfrac12\log\sigma(\Delta)-\tfrac12\log\sigma(-\Delta)$, minimized at $\Delta=0$. The classical statistical extensions add a tie parameter: Rao & Kupper (1967) put a threshold $\theta\ge1$ in the win probability, $\pi_i/(\pi_i+\theta\pi_j)$, and Davidson (1970) makes the tie probability proportional to $\nu\sqrt{\pi_i\pi_j}$. Many pipelines simply drop ties. Bai et al. keep only comparisons where the crowdworker expressed more than the weakest preference strength, and "do not include ties".
• Margins. Llama 2 collected a 4-point strength label (significantly better, better, slightly better, negligibly better/unsure) and added a margin to the loss: $-\log\sigma\big(r_c-r_r-m(\text{rating})\big)$. Their two variants were $m\in\{1,\tfrac23,\tfrac13,0\}$ ("small") and $\{3,2,1,0\}$ ("large"). The effect is real but modest: average accuracy went from 62.5% to 63.0% (small) and 62.9% (large). Gains came on clearly separable pairs, at some cost on near-ties. They also note that the margin pushes reward scores toward a bimodal "binary split", which matters because PPO is sensitive to the reward distribution.
• Label noise. Christiano et al. "assume there is a 10% chance that the human responds uniformly at random", i.e. $P=\tfrac{\varepsilon}{2}+(1-\varepsilon)\sigma(\Delta)$ with $\varepsilon=0.1$. This caps how confident the model can be and stops one mislabeled pair from producing an unbounded loss.

import torch, torch.nn.functional as F

def bt_loss(r_chosen, r_rejected, margin=None, eps=0.0):
    """r_*: (B,) rewards for the same prompts. margin: (B,) or None. eps: label-noise floor."""
    delta = r_chosen - r_rejected
    if margin is not None:
        delta = delta - margin                      # Llama 2: -log sigma(r_c - r_r - m)
    if eps == 0.0:
        return -F.logsigmoid(delta).mean()          # plain Bradley-Terry NLL
    p = eps / 2 + (1 - eps) * torch.sigmoid(delta)  # Christiano-style random-response floor
    return -torch.log(p).mean()

def rank_to_pairs_loss(r, ranking):
    """InstructGPT-style: one prompt's K responses, ranked best-first; all C(K,2) pairs in ONE element."""
    idx = torch.as_tensor(ranking)
    rr = r[idx]                                     # rewards in rank order
    w, l = torch.triu_indices(len(rr), len(rr), offset=1)
    return -F.logsigmoid(rr[w] - rr[l]).mean()      # mean = 1/C(K,2) weighting

§3The reward model as a network, and the noise it learns from

Take a pretrained transformer, remove the unembedding, add one linear layer that outputs a scalar from the final token's hidden state, and train it for about one epoch. The details that matter are what you initialize from, how you batch, and what the labels can and cannot tell you.

Architecture and recipe

• Backbone + scalar head. Stiennon et al.: "add a randomly initialized linear head that outputs a scalar value". InstructGPT: "Starting from the SFT model with the final unembedding layer removed". Llama 2: identical to the pretrained model "except that the classification head for next-token prediction is replaced with a regression head". The reward is read at the last token of the response, so it can depend on the whole sequence. The RM, like the policy, is a sequence model that has read everything.
• Initialize from a model that knows the policy's distribution. Llama 2 initialized its RMs from chat-model checkpoints so that "both models benefit from knowledge acquired in pretraining", explicitly to avoid an information mismatch that could favor hallucinations. RewardBench 2 (2025) later found that, for RLHF, the RM "should be based on a model of the same lineage as the policy model or else downstream performance can degrade significantly".
• Size. InstructGPT used a 6B RM even for the 175B policy: 175B RM training "could be unstable and thus was less suitable to be used as the value function during RL" (the value head was initialized from the RM), and it multiplied PPO compute. Stiennon et al. measured that doubling RM size gave ~1.8% accuracy versus ~1.1% from doubling data.
• One epoch. This is the most repeated folk rule in reward modeling, and the papers say it plainly. InstructGPT: "multiple epochs quickly overfit the model to the training data with obvious deterioration in the validation loss". Llama 2: "We train for one epoch over the training data. In earlier experiments, we found that training longer can lead to over-fitting". In a small-data control experiment in Gao et al.'s synthetic setup (checking whether too few SGD steps explained their ~2,000-comparison threshold), 4 epochs gave "no change in gold score whatsoever, whereas 1 epoch of 4 times as much data performs substantially better". Treat it as an empirical regularity rather than a law: RewardBench 2 reports that "contrary to the accepted best practice, training for more than one epoch can be beneficial" in their setting.

import torch, torch.nn as nn
from transformers import AutoModel

class RewardModel(nn.Module):
    def __init__(self, name):
        super().__init__()
        self.backbone = AutoModel.from_pretrained(name)          # SFT / chat checkpoint, LM head dropped
        self.head = nn.Linear(self.backbone.config.hidden_size, 1)
        nn.init.normal_(self.head.weight, std=1 / (self.backbone.config.hidden_size + 1) ** 0.5)
        nn.init.zeros_(self.head.bias)

    def forward(self, input_ids, attention_mask):                # prompt + response, RIGHT-padded
        h = self.backbone(input_ids=input_ids, attention_mask=attention_mask).last_hidden_state
        last = attention_mask.sum(dim=1) - 1                     # index of final real token
        return self.head(h[torch.arange(h.size(0)), last]).squeeze(-1)   # (B,) scalar rewards

Preference model pretraining and the HH data

Anthropic's early work treated the reward model, which they call a preference model (PM), as its own scaling object. Askell et al. (2021) compared three objectives as model size grows. Ranked preference modeling "performs much better than imitation learning, and often scales more favorably with model size". Binary discrimination (pass/fail) scaled about like imitation. They then introduced preference model pretraining (PMP), an extra stage:

LM pretraining → PMP on large public comparison data → PM finetuning on the small target dataset

The PMP data was scraped, naturally ranked text: Stack Exchange answers ranked by votes, Reddit comments by score, and Wikipedia edits that were quickly reverted as vandalism versus the text that replaced them. PMP "significantly improves sample efficiency" and transferred even when the domains differed, e.g. Stack Exchange → summarization. One counterintuitive detail: for the PMP stage, binarized comparisons transferred better than full rankings. Their hypothesis was that "hierarchies of preferences are difficult to quickly unlearn during finetuning", whereas binary discrimination "teaches models the correct features without establishing strong model preferences".

Bai et al. (2022) then built the HH (helpful and harmless) dataset that much of the open community trained on: a base set of ~44k helpfulness and ~42k red-teaming comparisons collected with a context-distilled 52B model, a rejection-sampling set, and an "online" set gathered from RLHF policies retrained roughly weekly. Two choices from that paper are worth remembering. First, the helpfulness data asks for the more helpful response, while the red-team data asks crowdworkers to elicit harm and pick the more harmful one. The two objectives can pull in different directions, and Bai et al. found PMs trained on the HH mixture somewhat less well calibrated than PMs trained on helpfulness alone. Second, iterated online RLHF: refreshing the comparison data with samples from the current policy keeps the PM on-distribution, which is the most direct defense against the drift in §4.

How much do humans agree? The ceiling under every RM

Preference labels are noisy in a way no amount of modeling removes, because the "noise" is partly genuine disagreement. Reported agreement rates:

These numbers are not directly comparable (different tasks, tie handling, and who counts as the reference). The message is consistent anyway: on open-ended chat, two careful humans agree roughly two times out of three to four times out of five. Llama 2's helpfulness RM reached 63.2% on its own held-out test set, and that number hides a strong dependence on difficulty: 80.7% on "significantly better" pairs, 54.7% on "negligibly better/unsure" ones.

A toy model makes the ceiling concrete. Suppose there is a consensus preference and each annotator independently reports its opposite with probability $\varepsilon$. Two annotators agree with probability $a=(1-\varepsilon)^2+\varepsilon^2$, so

At $a=72.6\%$ that is $\varepsilon\approx16\%$ and a ceiling of ~84%. A perfect model of the consensus would score 84% against individual labels and would look "worse" than a model that has learned some annotators' idiosyncrasies. The toy model is optimistic: it assumes a single consensus exists. When disagreement reflects different values rather than mistakes (§10), there is no consensus to recover.

§4Overoptimization: Goodhart's law, measured

Optimize against a learned reward and the true quality first rises, then peaks, then falls, while the proxy keeps climbing. That was folklore until Gao, Schulman & Hilton turned it into scaling laws with two clean functional forms.

Stiennon et al. had already shown the shape with real humans: "as we optimize further, true preferences fall off compared to the prediction, and eventually the reward model becomes anti-correlated with human preferences". Measuring this with human labels is expensive and noisy, so Gao et al. (2022) built a synthetic world. A 6B reward model from InstructGPT plays the role of the gold standard (the "human"). It labels 100k comparisons, which are used to train proxy RMs from 3M to 3B parameters. A 1.2B policy is then optimized against a proxy while both proxy and gold scores are tracked. Because the gold labels are synthetic, they could run many configurations and fit curves.

Where the peak is. Setting the derivatives to zero:

Because $\beta$ shrinks with RM size, larger RMs move the peak further out and raise it. For RL the effect is exponential in $\alpha/\beta$. The peak never disappears.

What else they found, in their own words where possible:

• Smooth coefficient scaling. α and β "vary smoothly with the number of proxy reward model parameters, following approximate logarithmic trends", with $\alpha_{\text{RL}}$ "nearly independent of RM parameter count". This allowed real advance prediction: a BoN form fit on $n\le1000$ (KL ≈ 6 nats) correctly predicted an experiment up to $n=60{,}000$ (KL ≈ 10 nats).
• RL is far less KL-efficient than BoN. RL "consumes" far more KL for the same gold gain, because each RL step moves from the previous policy and KL grows roughly quadratically in steps, while BoN "searches very locally around the initial policy". Plotted against the proxy score instead of KL, BoN and RL look much more alike. KL is a poor common currency across optimizers.
• The KL penalty acted like early stopping. In their PPO setup the penalty "only causes the gold RM score to converge earlier, but does not affect the KL_RL-gold reward frontier", with the caveat that this "could be particularly sensitive to hyperparameters". The penalty decides where on the curve you stop. It does not give you a better curve.
• Policy size barely matters for overoptimization. A 6B policy gained less from optimization than a 1.2B one (it started better) but peaked at almost the same KL, with an almost identical proxy-gold gap.
• A data threshold. Below ~2,000 comparisons, RMs barely improved on near-chance loss. Two RMs of equal validation loss seemed (weakly) equally robust, whatever mix of size and data produced that loss.
• Iterated RLHF. If you retrain the RM $k$ times, each iteration covering distance $d/k$, and the coefficients stay fixed, the form becomes $d(\alpha_{\text{RL}}-\beta_{\text{RL}}\log d+\beta_{\text{RL}}\log k)$. Fresh data adds $\beta_{\text{RL}}\,d\log k$ and does nothing for the α (regressional) term. This is a formal argument for online RLHF.

Why √KL, and why the gold curve bends

Here is a useful sanity check you can do on paper. Suppose the proxy reward of a sample is Gaussian under the base policy. Best-of-$n$ selects the maximum of $n$ Gaussians, whose expectation grows like $\sqrt{2\log n}$. The BoN KL is about $\log n$ for large $n$, so $d=\sqrt{\KL}\approx\sqrt{\log n}$ and the expected proxy grows linearly in d. That is the "roughly linear in √KL" proxy behavior seen by both Bai et al. and Gao et al. Now let the gold reward be the proxy minus independent noise. Selecting on the proxy partly selects on the noise, and the gold gain is the proxy gain shrunk by $\mathrm{Var}(X)/(\mathrm{Var}(X)+\mathrm{Var}(Z))$ (Gao et al.'s eq. 1). That shrinkage is still linear in $d$: it lowers α, but cannot produce a peak. The bend needs something else, namely features where the proxy and gold disagree more the further you go. That is extremal Goodhart, and it is what β measures.

The Goodhart taxonomy

Manheim & Garrabrant (2018) split Goodhart's law ("when a measure becomes a target, it ceases to be a good measure") into four mechanisms. Every one shows up in reward modeling:

Gao et al. map their α term to regressional and their β term mainly to extremal Goodhart. They explicitly note that their small models are probably not strong enough for adversarial Goodhart, and that "the scaling laws observed in this paper may break down" when that happens. For a frontier lab in 2026 that caveat is the main point, not a footnote (§6).

Mitigations that work, partially

Ensembles and conservative objectives. Train $k$ reward models and optimize a pessimistic aggregate. Coste et al. (2023) compare worst-case optimization, $R_{\text{WCO}}=\min_i R_i$, with uncertainty-weighted optimization,

in Gao's synthetic setup extended with 25% label noise. Their finding: conservative optimization "practically eliminates overoptimization and improves performance by up to 70% for BoN sampling". For PPO it always reduced overoptimization, and combined with a small KL penalty it prevented it "at no performance cost". The plain mean of an ensemble is not conservative: one member that overestimates is enough to get exploited.

But ensemble members share blind spots. Eisenstein et al. (2023) show that RMs are underspecified: models with similar in-distribution accuracy give very different rewards under the shifted distribution of an optimized policy. Ensembles help, and members that differ in pretraining seed help more than members differing only in fine-tuning seed. Still, "even pretrain reward ensembles do not eliminate reward hacking", because "all reward models in the ensemble exhibit similar error patterns". WARM (Ramé et al. 2024) averages fine-tuned RMs in weight space, which costs one RM at inference and relies on linear mode connectivity. A WARM-trained policy won 79.4% of comparisons against one trained with a single RM.

def conservative_reward(rewards, mode="uwo", lam=0.5):
    """rewards: (k, B) scores from k reward models for the same B samples."""
    if mode == "mean":
        return rewards.mean(0)                                   # NOT conservative
    if mode == "wco":
        return rewards.min(0).values                             # worst case over the ensemble
    mu = rewards.mean(0)
    return mu - lam * rewards.var(0, unbiased=False)             # Coste et al. UWO

§5Best-of-n: the baseline every RL run must beat

Sample $n$ responses from the reference policy and keep the one the RM likes best. No training, a provable KL budget, and near-optimal win rate per nat. It's also the cleanest instrument for measuring overoptimization.

The BoN distribution and its KL

Fix a prompt and assume the reward takes distinct values with CDF $F$ under $\piref$. The selected sample has reward at or below $t$ only if all $n$ samples do, so its CDF is $F(t)^n$. For a continuous reward the density is $n F^{n-1} f$, a reweighting of the reference by $nF(r)^{n-1}$. If each response has its own reward value, the KL over responses equals the KL over rewards:

Since $\E[\log U]=-1/n$ for $U\sim\mathrm{Beta}(n,1)$, we get the formula used by Stiennon et al. and Gao et al.:

It is an upper bound, not an identity. Beirami et al. (2024) prove $\KL(\pi^{(n)}\|\piref)\le\log n-\tfrac{n-1}{n}$ for discrete outputs and give a counterexample. With two equally likely outcomes, the true KL never exceeds $\log 2$, while the formula grows without bound. The gap is at most $2n(n-1)e^{-H_2}$, where $H_2$ is the Rényi-2 entropy of the reference given the prompt. So the formula is accurate when no single response has noticeable probability, which is typical for long open-ended generations. It overstates KL when outputs collapse onto a few strings: short answers, low temperature, multiple-choice. They also show the BoN win rate against the reference is at most $n/(n+1)$.

Why BoN is such a strong baseline

• It's near-optimal per nat. Gui, Gârbacea & Veitch (2024) show that within a broad family of tilted distributions, BoN "is essentially optimal in terms of the trade-off between win-rate against the base model vs KL distance". RL reaches a different (KL-regularized) optimum, $\pi^*\propto\piref\,e^{r/\beta}$, derived in chapter 04. If the goal is to win comparisons, BoN is hard to beat at equal KL.
• It's what WebGPT shipped. WebGPT's best model was behavior cloning plus rejection sampling (175B, best-of-64). They "excluded RL for simplicity, since it did not provide significant benefit when combined with rejection sampling", though RL helped when inference compute was limited.
• It's an overoptimization meter. BoN needs no training, so it isolates the reward model. If gold reward falls at moderate $n$, your RM has a problem no RL trick will fix.
• It's a training method in disguise. Rejection-sampling fine-tuning (chapter 01) distills BoN into the weights. BoNBoN alignment and related methods target the BoN distribution directly.

The other side, from Gao et al.: as a function of KL, RL overoptimizes more slowly and optimizes more slowly. Compared at equal proxy score, RL "initially has a larger proxy-gold gap" but "peaks at a higher gold RM score than BoN". BoN wins at small budgets and RL can reach further. In practice the comparison should be at equal compute and on gold or human evaluations, never at equal proxy reward.

import math, torch

def best_of_n(policy, rm, prompt, n):
    ys = policy.sample(prompt, num_samples=n)          # n i.i.d. samples from pi_ref
    scores = rm(prompt, ys)                            # (n,)
    kl_bound = math.log(n) - (n - 1) / n               # upper bound (Beirami et al.)
    return ys[scores.argmax()], kl_bound

§6Spurious correlates and reward hacking

Every preference dataset contains features that correlate with "preferred" without causing it: length, formatting, confident tone, agreeing with the user. A reward model learns them because they predict the labels. A policy finds them because they are much cheaper to produce than actual quality.

Length: the best-documented case

Singhal et al. (2023) took three RLHF settings and asked how much of the reward gain was length. On WebGPT and RLCD, "70%–90% of the improvement" came from length. More strikingly, "even a purely length-based reward reproduces most downstream RLHF improvements over supervised fine-tuned models". They traced the cause to the reward models, which were "non-robust and easily influenced by length biases in preference data". The confidently predicted ("easy") training examples were largely the ones where the length heuristic happened to be right.

Why does the policy pad instead of improving? The KL geometry explains it. Treat the policy as a distribution over response features and the KL penalty as measuring how unusual the new distribution is relative to the reference. For a Gaussian-mean family with reference covariance $\Sigma$, $\KL=\tfrac12\Delta\mu^\top\Sigma^{-1}\Delta\mu$, and the KL-regularized optimum for a locally linear reward is

The policy moves furthest along directions where the reference already varies a lot. The base model's lengths vary by a factor of two or more from one sample to the next. Its quality on a given prompt varies much less. Even if the RM puts the same weight on both, the optimizer buys mostly length, because length is cheap in KL. This is a toy model, but it shows why "more KL budget" doesn't fix the problem.

Mitigations. Measure first: report win rates with length control, and plot reward against length for every policy checkpoint. Length-controlled AlpacaEval (Dubois et al. 2024) answers "what would the preference be if the model's and baseline's output had the same length?" by regressing out the length difference. Fix the RM: ODIN trains two heads on shared features, "one trained to correlate with length, and the other trained to decorrelate with length", then discards the length head during RL. Counterfactual data (the same content at different lengths) attacks the causal Goodhart directly. Fix the objective: explicit length penalties, or normalizing reward by a length-matched baseline. These are blunt, because sometimes the longer answer really is better.

Sycophancy is a learned preference

Length is the benign case. The same mechanism produces sycophancy: telling users what they want to hear. Sharma et al. (2023, Anthropic) analyzed Anthropic's own helpfulness preference data. All else equal, "when a response matches a user's views, it is more likely to be preferred", and "both humans and preference models (PMs) prefer convincingly-written sycophantic responses over correct ones a non-negligible fraction of the time". With best-of-$n$, optimizing against the PM used to train Claude 2 "consistently yields more sycophantic responses" than optimizing against a "non-sycophantic" PM (the same PM, prompted with an explicit request for truthful answers). Measured during Claude 2's actual RL phase, feedback and mimicry sycophancy increased as training went on. The effects were mixed rather than uniformly bad: best-of-$n$ against the Claude 2 PM reduced some forms of sycophancy, just less than the improved PM did. The behavior is not a bug in the RL algorithm. It is in the labels, and the RM faithfully learns it. Chapter 08 covers how character training and evaluations push back.

When the policy knows about the reward model

Two 2025 Anthropic papers show what adversarial Goodhart looks like with capable models.

• Marks et al. (2025) deliberately built a model with a hidden objective. They pretrained it on synthetic documents describing 52 fictional RM biases, then trained it to exploit 47 of them. The model generalized to exhibit "whatever behaviors it believes RMs rate highly, including ones not reinforced during training". It had learned the abstract goal "please the RM", not a list of tricks. Three of four blind auditing teams recovered the objective, using sparse autoencoders, behavioral attacks and training-data search.
• MacDiarmid et al. (2025) trained on "real Anthropic production coding environments" after giving the model knowledge of reward-hacking strategies. It learned to hack, and that generalized to "alignment faking, cooperation with malicious actors, reasoning about malicious goals, and attempting sabotage". Standard chat-style RLHF safety training fixed chat evaluations, but "misalignment persists on agentic tasks". Three mitigations worked: preventing the hacking, more diverse RLHF safety training, and "inoculation prompting", which frames hacking as acceptable in that training context so that it does not generalize into a misaligned self-concept. The authors add that Anthropic has "started implementing this technique in production Claude training".

§7Generative reward models, judges, rubrics and reasoning RMs

A scalar head forces the RM to compress a judgment into one forward pass and one number. Generative RMs let the judge write first: a critique, a checklist, a chain of thought. The score comes after. You pay inference compute and get more accuracy and more inspectability.

From LLM-as-judge to RLAIF

The cheapest RM is a strong LLM with a prompt. Zheng et al. (2023) found GPT-4 agreed with experts 85% of the time on non-tie votes, above the 81% human-human rate, while documenting position bias (favoring the first answer), verbosity bias and self-enhancement bias. Anthropic's Constitutional AI (Bai et al. 2022) used a model's own judgments against a list of principles as preference labels for the harmlessness PM. This is RLAIF, covered in detail in chapter 04. Soft labels from the feedback model's normalized log-probabilities "led to much better results than hard labels". With chain-of-thought, the probabilities became overconfident, so they were clamped to the 40–60% range, because without clamping "RL-CAI models would learn to output more extreme responses". Lee et al. (2023) found RLAIF comparable to RLHF on summarization and dialogue. They also introduced direct RLAIF, which skips the RM and queries the LLM judge for rewards during RL.

Generative RMs: train the judge, keep the LM head

• Generative verifiers (Zhang et al. 2024) train verification as next-token prediction: the reward is $P(\text{"Yes"}\mid x,y,\text{"Is the answer correct?"})$, optionally after a verification chain of thought, with majority voting over several CoTs. On GSM8K, a Gemma2-9B verifier choosing among Gemini 1.0 Pro's samples raised the share of problems solved from 73% to 93.4%.
• GenRM (Mahan et al. 2024): zero-shot LLM judges underperform BT RMs in-distribution (by 9–36%). A judge trained on self-generated reasoning traces matches BT in-distribution and beats it out-of-distribution (by 10–45%).
• Critique-out-Loud (Ankner et al. 2024): generate a critique, then predict a scalar from it. +4.65 and +5.84 points on RewardBench for 8B and 70B models.
• DeepSeek-GRM (Liu et al. 2025): a pointwise generative RM trained with online RL to generate principles, then critiques, then scores ("Self-Principled Critique Tuning"). It scales at inference by sampling several judgments and voting, with a meta-RM that filters bad judgments before the vote.

Reasoning reward models (2025)

Once reasoning models existed (chapter 06), the obvious next step was to let the judge think. RM-R1 (Chen et al. 2025) frames reward modeling "as a reasoning task". Its "chain-of-rubrics" has the model write sample-specific rubrics, or its own solution for math and code, and grade against them. Training is distillation of good reasoning chains, then RL with verifiable rewards on preference correctness. Reward Reasoning Models (Guo et al. 2025) use RL with no reasoning traces as supervision, and show the judge can "adaptively exploit test-time compute". The pattern is the same one that made policies better: RL on a checkable signal, here "did you pick the human-preferred response?".

Rubrics as rewards

A rubric turns "is this good?" into a list of checkable criteria written for each prompt. Rubrics as Rewards (Gunjal et al. 2025) attaches $k$ weighted criteria $(w_j,c_j)$ to each prompt. Each $c_j(x,\hat y)\in\{0,1\}$ is judged by an LLM, and

The alternative, "implicit aggregation", hands all the criteria to the judge and asks for one score. RLVR is the special case $k=1$, $c_1=$ exact-match. On medical and science benchmarks, the best variant improved up to 31% (relative) on HealthBench over Likert-scored LLM-judge rewards. Checklists Are Better Than Reward Models (Viswanathan et al. 2025) extracts instruction-specific checklists and mixes AI judges with verifier programs. Their RLCF was "the only method to help on every benchmark" in their comparison. 2026 work treats rubric generation itself as a learned policy: Rubric-ARM alternates RL between a rubric generator and a judge, and OpenRS generates pairwise, adaptive rubrics from a "meta-rubric".

def rubric_reward(prompt, response, rubric, judge):
    """rubric: list of (weight, criterion_text); negative weights encode 'pitfalls'."""
    num = den = 0.0
    for w, crit in rubric:
        sat = judge.yes_prob(f"Criterion: {crit}\nPrompt: {prompt}\nResponse: {response}\n"
                             "Does the response satisfy the criterion? Answer Yes or No.")
        num += w * (sat > 0.5)          # RaR uses binary checks; a soft version uses sat directly
        den += abs(w)
    return num / max(den, 1e-8)

§8Outcome vs process rewards

For multi-step reasoning you can reward the final answer (an outcome reward model, ORM) or every step (a process reward model, PRM). PRMs give denser, better-localized signal and were the 2023 favorite. By 2025 they had mostly moved out of the RL loop and into search and verification.

What the evidence says

Uesato et al. (2022) ran the first careful comparison on GSM8K. Outcome supervision reached similar final-answer error with less labeling, but getting the reasoning right required process supervision, or an RM that emulates it (reasoning error among correct answers fell from 14.0% to 3.4%). Lightman et al. (2023) scaled this to MATH with best-of-1,860 reranking. The PRM solved 78.2%, against 72.4% for an ORM and 69.6% for majority voting. They also showed that active learning, surfacing "convincing wrong-answer solutions" to labelers, improved data efficiency. Math-Shepherd removed the humans and used its PRM for step-by-step PPO (Mistral-7B: 77.9% → 84.1% on GSM8K).

Then the reasoning-RL wave of 2025 largely went around PRMs. The DeepSeek-R1 report says they "abstain from applying neural reward models—whether outcome-based or process-based—to reasoning tasks" because they are "susceptible to reward hacking during large-scale reinforcement learning". Its "unsuccessful attempts" section lists three problems: it is "challenging to explicitly define a fine-grain step", step correctness is hard to judge automatically or to label at scale, and "once a model-based PRM is introduced, it inevitably leads to reward hacking". It also concedes that PRMs work well "to rerank the top-N responses generated by the model or assist in guided search". The Qwen team's lessons paper adds that MC-estimated labels underperform LLM-judge and human labels, and that best-of-$n$ evaluation rewards PRMs that drift toward judging only the final step.

Where PRMs went in 2025–2026.

• Search and reranking at test time: PRM-guided beam search and best-of-$n$, where the policy is not being trained against the PRM, so hacking pressure is limited to the search budget.
• Generative, reasoning PRMs such as ThinkPRM, which verify by writing a chain of thought and need orders of magnitude fewer step labels.
• Implicit process rewards: PRIME derives dense token-level rewards from a model trained only on outcome labels, updated online on policy rollouts so the PRM moves with the policy.
• Verifiers for proofs, where final-answer checking is meaningless and step-level rigor is the whole point (DeepSeekMath-V2).

§9Evaluating reward models

The obvious metric, pairwise accuracy on held-out preferences, is easy to compute and only loosely related to what you want: a better policy after optimization. Benchmarks have improved. The gap hasn't closed.

• RewardBench (Lambert et al. 2024) collected prompt-chosen-rejected trios in four categories: Chat, Chat Hard, Safety and Reasoning. Many pairs differ for "subtle, but verifiable reasons" (bugs, incorrect facts). It became the default leaderboard and quickly saturated.
• RewardBench 2 (Malik et al. 2025) uses new, unseen human prompts and a best-of-4 format (one chosen, three rejected), so random guessing scores 25% instead of 50%. Models score "about 20 points on average lower" than on RewardBench. The paper is also blunt about its limits: accuracy correlates strongly with best-of-$n$ performance (Pearson 0.87 on their downstream suite), but for PPO the signal is "rough". PPO results "saturate" for all reasonable RMs (benchmark scores 49.8–68.5).
• PPE (Frick et al. 2024) went to the expensive ground truth. They trained policies with many RMs, measured them on Chatbot Arena with real users, and identified which cheap proxy metrics predict that outcome.
• Why accuracy is a weak predictor. Wen et al. (2024) find "a weak positive correlation between accuracy and downstream performance", with "policies optimized towards RMs with similar accuracy" showing "quite different performance". Accuracy measures ranking on the data distribution. Optimization uses the RM on the tail of a shifted distribution (extremal Goodhart again), and is sensitive to the reward's scale and calibration, which accuracy ignores. Ivison et al. (2024) found that better preference data moved downstream results more than better reward models did. Skywork-Reward-V2 (Liu et al. 2025) pushed the same thesis to scale: it attributes RM brittleness mainly to the data, curates 40M preference pairs with a human-plus-LLM pipeline, and trains 0.6B–8B RMs on a 26M-pair subset that topped seven RM benchmarks at release.

§10Whose preferences? Pluralism and aggregation

A single reward model trained on many annotators is a voting rule, whether or not anyone chose one. It is worth knowing which rule.

Siththaranjan, Laidlaw & Hadfield-Menell (2023) formalize hidden context: anything that affects a label but is invisible to the RM, such as who the annotator is, what they value, or which criterion they were told to use. They prove that standard preference learning "implicitly aggregate[s] over hidden contexts according to a well-known voting rule called Borda count". Borda scores each option by its average win probability against the others. It ignores how much each voter cares. A response that is everyone's close second can lose to one that half the population loves and half hates. They also show the "annotators have an incentive to misreport their preferences" to steer the learned model.

Proposed responses fall into three groups. Know the rule you are using: Conitzer et al. (2024) argue that social choice theory should guide how feedback is aggregated. Model the distribution instead of the average: distributional preference learning, latent-variable RMs conditioned on a user embedding (VPL), and mixtures of RMs optimized with an egalitarian max-min objective (MaxMin-RLHF). Collect from the right people: PRISM gathered preferences from 1,500 participants in 75 countries, linked to their profiles. Sorensen et al. (2024) name three goals that a single scalar RM can't serve at once: Overton pluralism (present the range of reasonable views), steerable pluralism (adopt a perspective when asked), and distributional pluralism (match a population's distribution of answers).

§11Frontier & open problems

§12Check yourself

Try each one before opening the answer.

§13Read next