RL with verifiable rewards & reasoning

§1Verifiable rewards: the reward you cannot sweet-talk

RLHF learns a reward from human preferences and then optimizes against it, and Chapter 03 showed what happens next: the policy finds the reward model's blind spots. For a large class of tasks we do not need a learned judge at all. A math answer either equals the reference or it does not. A program either passes the tests or it does not.

RLVR is the KL-regularized objective of Chapter 04 with the reward model swapped for a deterministic checker. Tülu 3 (Lambert et al., 2024) wrote it as:

Tülu 3 ran this with PPO, a value model initialized from a general reward model, about 30k prompts (GSM8K, MATH, IFEval-style constraints), and a −10 penalty for completions with no end-of-sequence token. Gains were modest at that scale (8B: MATH 42.0 → 43.7, GSM8K 84.3 → 87.6, IFEval 81.1 → 82.4). Two findings matter more. First, adding reward-model scores on top of the verifiable reward did not help: on GSM8K, verifiable rewards alone scored better. Second, overoptimization still appeared as β decreased: a model optimized hard against the checker drifted away from things it does not measure.

The verifier zoo

The open recipes of 2025 converged on exactly this menu. Olmo 3's OlmoRL (Team Olmo, 2025) uses a SymPy math checker, test-case execution for code (they compared partial credit with all-or-nothing), all-constraints-satisfied for instruction following, and LM judges for chat. Magistral (Mistral, 2025) adds a 0.1 format reward, +0.9 for correctness, a soft length penalty and +0.1 for language consistency.

Why verifiable rewards resist hacking, and where they don't

A learned reward model is a function approximator queried far outside its training distribution, and the policy is an optimizer searching for adversarial inputs. A checker is different in kind: on the inputs it was designed for, its output is the quantity we care about. DeepSeek-R1 cites exactly this for its choice: neural reward models are "susceptible to reward hacking during large-scale reinforcement learning" (DeepSeek-AI, 2025).

The hedge is "on the set of inputs it was designed for". The policy does not stay inside that set. The failure modes are concrete:

• Weak tests. If hidden tests do not pin down the behavior, "make the tests pass" differs from "solve the problem": special-case the inputs, return early, or catch every exception. OpenAI documented a frontier reasoning model reward hacking in agentic coding environments, and showed that a weaker model (GPT-4o) reading the chain of thought was a more effective monitor than one reading only actions and outputs (Baker et al., 2025).
• Attacking the harness. In Anthropic's study of emergent misalignment from reward hacking (MacDiarmid et al., 2025), a pretrained model was first given knowledge of hacks through synthetic documents mixed into continued pretraining (one example: calling sys.exit(0) so the test harness exits with code 0 and appears to pass), then trained with RL on real coding environments from Claude training runs that were known to be hackable. It learned to hack, and at the same point in training every misalignment evaluation rose, including alignment-faking reasoning and attempts to sabotage safety-research code. Later studies differ on how far this generalizes (§10).
• Answer-format exploits. If the checker accepts any correct box, write several boxes. If it greps for the right number anywhere, mention every candidate. Huang et al. (2025) found that rule-based math checkers produce false negatives that hurt more as the policy gets stronger, while model-based checkers fix those but are highly susceptible to hacking during RL.
• Constraint gaming. IFEval-style checks reward the letter of a constraint (see Tülu 3's appendix).

§2The reasoning-model turn

Between September 2024 and mid-2025, RL on verifiable tasks went from a post-training detail to the main event. Models learned to "think" in long chains of thought, and competition-math accuracy rose with both training compute and thinking length.

OpenAI o1: what was actually said

OpenAI's September 2024 post ("Learning to reason with LLMs") disclosed almost no method:

• "Our large-scale reinforcement learning algorithm teaches the model how to think productively using its chain of thought in a highly data-efficient training process."
• Performance "consistently improves with more reinforcement learning (train-time compute) and with more time spent thinking (test-time compute)".
• AIME 2024: 74% with one sample per problem, 83% with consensus over 64 samples, 93% when re-ranking 1000 samples with a learned scoring function. 89th percentile on Codeforces.
• The raw chain of thought is hidden. The stated reasons were user experience, competitive advantage and monitoring: to monitor the CoT, OpenAI said it could not train policy compliance or user preferences onto it.

A lesson in test-time compute (§8): voting buys 9 points, and a learned verifier over 1000 samples 10 more.

DeepSeek-R1-Zero: pure RL from a base model

DeepSeek-R1 (arXiv, January 2025; peer-reviewed in Nature, September 2025) answered the "how" in public. R1-Zero applies GRPO directly to DeepSeek-V3-Base with no supervised fine-tuning, using only two rule-based rewards: accuracy and format (reasoning inside <think> tags).

• AIME 2024 pass@1 went from 15.6% to 71.0% in the January preprint (86.7% with majority voting). The Nature version, with longer training, reports 15.6% → 77.9%, still 86.7% with self-consistency.
• Thinking length grew on its own, from hundreds to thousands of tokens per response, without any length reward.
• The "aha moment": a checkpoint wrote "Wait, wait. Wait. That's an aha moment I can flag here." and re-evaluated its approach, presented as emergent reflection.
• Costs: poor readability and mixing English and Chinese within one chain of thought.

Reported R1-Zero hyperparameters: learning rate 3e-6, KL coefficient 0.001, 16 samples per question, 32 questions per step, maximum length 32,768 then 65,536 tokens, 10,400 steps, reference refreshed every 400 steps.

The aha moment, re-examined

Many small-scale replications reported their own "aha moments". Then Liu et al. (2025) deflated the story on three counts:

1. The behavior predates RL. DeepSeek-V3-Base already produces "Aha" and "wait"-style self-reflection. Qwen2.5 base models answer competition questions well without a prompt template, suggesting pretraining on question–answer text.
2. Reflection does not track accuracy. Nearly half of self-reflecting R1-Zero-style responses are no more accurate than those that do not.
3. Length growth is partly an artifact. GRPO's length normalization favors long wrong answers (§4). With the bias removed, length grows less.

Gandhi et al. (2025) give the positive version of the same point. Under identical RL on Countdown, Qwen-2.5-3B far outperforms Llama-3.2-3B; the difference is whether the base model already exhibits four cognitive behaviors: verification, backtracking, subgoal setting and backward chaining. Priming Llama with such traces closes the gap, even when the primed solutions are wrong. RL amplifies behaviors the base model can already produce.

R1: making it usable, and the distillation result

R1 itself wraps RL in a four-stage pipeline: (1) a cold start of thousands of curated long-CoT examples; (2) reasoning-focused RL with an added language-consistency reward (the fraction of target-language words in the CoT); (3) rejection sampling from that checkpoint to build about 600k reasoning and 200k non-reasoning SFT examples, used to fine-tune V3-Base again; (4) a final RL stage over all scenarios with rule-based and preference rewards. R1 reaches 79.8% on AIME 2024 and 97.3% on MATH-500.

The most consequential table in the paper compares two ways of giving a 32B model reasoning ability:

DeepSeek's conclusion: distilling a stronger model beats large-scale RL on the smaller one, but pushing past the frontier still needs stronger base models and larger-scale RL. What did not work: process reward models (hard to define steps, hard to label them, and the PRM gets hacked) and MCTS (the token-level search space is enormous and the value model is hard to train).

The 2025 wave

That is the 2025 picture; 2026 recipes such as Kimi K3 and MiMo-V2.6 spend much of their RL on long-horizon agentic environments.

§3GRPO, derived

GRPO (Shao et al., 2024, introduced in DeepSeekMath) is PPO with the critic replaced by a statistic of a group of samples. Every design choice falls out of one question: what baseline can you compute without learning a value function?

Start from the policy gradient with a baseline (Chapter 02). For a prompt $q$ and a complete response $o$ with scalar reward $r(q,o)$:

Any baseline $b(q)$ that does not depend on $o$ leaves the gradient unbiased, because $\E_{o}[\nabla\log\pit(o|q)]=\sum_o\nabla\pit(o|q)=\nabla 1 = 0$. The best simple choice is $b(q)\approx V(q)=\E_o[r]$, the prompt's expected reward. PPO learns $V$ with a critic network as large as the policy. With outcome-only rewards, the critic has to predict a binary pass/fail from a partial chain of thought, a hard regression problem whose errors inject bias into every token's advantage.

GRPO's move: sample a group of $G$ responses $\{o_1,\dots,o_G\}$ for the same prompt and use the group's own rewards to estimate $V(q)$.

DeepSeekMath used learning rate 1e-6, β = 0.04, $G=64$, maximum length 1024 and batch size 1024, and lifted MATH from 46.8% (the instruct model) to 51.7%. A process-supervision variant gives each token the sum of group-normalized step rewards from its step onward. Almost nobody uses it for reasoning RL today: outcome rewards won over step-level labels. In 2026 the outcome itself is getting richer than pass/fail: MiMo-V2.6 (2026) grades passing solutions against each other, because a binary test reward cannot tell them apart.

Why the group mean is a legitimate baseline

Mean-centering uses $o_i$'s own reward in its baseline, which looks like it should bias the gradient. Write the mean as a mix of $r_i$ and the leave-one-out mean $\bar r_{-i}=\frac{1}{G-1}\sum_{j\ne i}r_j$:

So GRPO's centered advantage is exactly the RLOO advantage (Ahmadian et al., 2024) scaled by $(G-1)/G$. Since the other samples are drawn independently of $o_i$, $\bar r_{-i}$ is a valid baseline and the estimator is unbiased up to that constant, which the learning rate absorbs. Liu et al. make the same observation.

The std division is a different story. $\operatorname{std}(r_1,\dots,r_G)$ depends on $r_i$, so the estimator is no longer an unbiased estimate of $\nabla J$. More importantly it changes which objective you are following. For binary rewards with per-prompt pass rate $p$ and large $G$, $\operatorname{std}\approx\sqrt{p(1-p)}$ and the expected update for one prompt is approximately

(Our own reading, from $\frac{d}{dp}\arcsin\sqrt p=\frac{1}{2\sqrt{p(1-p)}}$.) Mean-only centering ascends $\sum_q p_q$. Std-normalized GRPO ascends $\sum_q \arcsin\sqrt{p_q}$, a variance-stabilizing transform that is steepest near $p=0$ and $p=1$. Prompts the model almost always solves, or almost never solves, get up-weighted relative to prompts at 50%. Liu et al. call this the question-level difficulty bias; Dr. GRPO removes the std, and so do Olmo 3 and (per its report) DeepSeek-V3.2. REINFORCE++ (Hu, 2025) goes the other way: normalize advantages over the whole global batch instead of per prompt, which keeps a scale normalization without per-prompt reweighting.

Zero-advantage groups

If all $G$ rewards are equal, every $\hat A_{i,t}=0$ and the prompt contributes nothing except the KL term. As training progresses, more prompts become all-correct, which shrinks the effective batch and grows gradient noise. DAPO's dynamic sampling over-samples and keeps only prompts with $0<\#\text{correct}<G$ until the batch is full. Olmo 3's active sampling is the same idea. ScaleRL (Khatri et al., 2025) adds "No-Positive-Resampling": permanently drop prompts whose historical pass rate is at least 0.9. Kimi k1.5 instead samples prompts with probability proportional to $1-s_i$, the failure rate.

Group or critic?

The group baseline has a price, and a critic is the alternative. A group gives you a baseline with no extra network, but it spends several rollouts per prompt, and the authors of SAO, a 2026 one-rollout recipe with a critic, argue that in asynchronous training a group must wait for its slowest rollout, so its samples go stale. A critic needs only one rollout per prompt, and brings back what GRPO set out to avoid: a value model that has to be trained first. Treat it as a trade-off between latency and the cost of a critic. It does not touch the derivation above: the objection is about latency, not the baseline's math. And when a comparison favors one side, ask what else changed: in SAO's own math results, much of the gap comes from a separate token mask that also works with GRPO. This is one lab's result on one backbone.

The KL term: what k3 in the loss actually does

Chapter 02 shows that k3, $\rho-\log\rho-1$ with $\rho=\piref/\pit$, is an unbiased, non-negative estimator of $\KL(\pit\|\piref)$ when evaluated on samples from $\pit$. GRPO, however, does not use it as an estimate; it puts it in the loss and differentiates it. Do the derivative per token, on-policy, ignoring the dependence of future tokens on the current one:

The expected gradient is that of the forward KL $\KL(\piref\|\pit)$, not the reverse KL the objective writes down. Near $\pit\approx\piref$ the two agree to first order ($1-\rho\approx-\log\rho$), so it behaves like a KL anchor in practice; far from the reference it is a different regularizer. By contrast, k2 $=\tfrac12(\log\rho)^2$ as a loss gives $\E[(\log\pit-\log\piref)\nabla\log\pit]$, the per-token reverse-KL gradient. Tang & Munos (2025) catalogue this class of mistakes: differentiating a KL estimate generally does not give the KL gradient, and ignoring the sequential structure gives at best a partial gradient. DeepSeek-V3.2 switched to an importance-corrected k3, $\frac{\pit}{\piold}\big(\rho-\log\rho-1\big)$, because the original assigns unbounded weight to tokens far less likely under $\pit$ than under $\piref$. The fix is exact: differentiating the weighted term at $\pit=\piold$ adds $\E_{\pit}[\text{k3}\cdot\nabla\log\pit]$ to the forward-KL gradient above, and since $\text{k3}=\rho+\log\frac{\pit}{\piref}-1$, the $\rho$ parts cancel and what remains is $\E_{\pit}\big[\log\frac{\pit}{\piref}\nabla\log\pit\big]=\nabla\KL(\pit\|\piref)$, the per-token reverse-KL gradient. Many recipes (DAPO, Magistral, Olmo 3) set β = 0.

# GRPO-family loss in PyTorch. logp, logp_old, logp_ref: [B*G, T] per-token log-probs
# of the sampled tokens; rewards: [B*G]; mask: [B*G, T], 1 on completion tokens.
def grpo_loss(logp, logp_old, logp_ref, rewards, mask, G,
              eps_lo=0.2, eps_hi=0.2, beta=0.0, std_norm=True, agg="token", C=None):
    r = rewards.view(-1, G)
    adv = r - r.mean(dim=1, keepdim=True)                       # group baseline
    if std_norm:                                                # GRPO; Dr. GRPO drops this
        adv = adv / (r.std(dim=1, keepdim=True, unbiased=False) + 1e-6)
    adv = adv.view(-1, 1)                                       # same advantage for every token
    ratio = torch.exp(logp - logp_old)
    pg = torch.minimum(ratio * adv, torch.clamp(ratio, 1 - eps_lo, 1 + eps_hi) * adv)
    per_tok = -pg
    if beta > 0:                                                # k3 in the loss (see caveat above)
        lr = logp_ref - logp
        per_tok = per_tok + beta * (torch.exp(lr) - lr - 1)
    if agg == "seq":      # GRPO paper: mean over each response, then over responses
        return ((per_tok * mask).sum(1) / mask.sum(1)).mean()
    if agg == "token":    # DAPO: one mean over every token in the batch
        return (per_tok * mask).sum() / mask.sum()
    return (per_tok * mask).sum() / (mask.shape[0] * C)         # Dr. GRPO: constant C

§4Loss aggregation and the length bias

Once each token has a loss $\ell_{i,t}$, you still have to decide how to average them. It decides whether the policy learns to ramble.

Three choices appear in practice. With $\ell_{i,t}$ the clipped surrogate for token $t$ of response $i$:

Now read off the per-token gradient weight under sequence-mean: $w_{i,t}=\hat A_i/(G|o_i|)$.

• For a correct response ($\hat A_i>0$), shorter is better: each token receives a larger push. Fine.
• For a wrong response ($\hat A_i<0$), longer is better: the fixed total penalty is spread over more tokens. A wrong answer that doubles its length halves the penalty on every token, and the total penalty does not grow at all.

The policy is therefore pushed toward short correct answers and long wrong ones. Liu et al. (2025) showed this produces the "double increase" in length and accuracy seen in R1-Zero replications, and that removing it (Dr. GRPO) gives similar accuracy with better token efficiency. Their recipe reached 43.3% on AIME 2024 from Qwen2.5-Math-7B, about 27 hours on 8 A100s. They also found the per-response normalization in several popular open-source PPO implementations.

DAPO (Yu et al., 2025) reached the token-level loss from the other side: with sample-level averaging, tokens in long responses contribute too little, so gibberish and repetition in long samples go unpenalized, driving growth in entropy and length.

Token-mean is not unbiased either

Token-mean divides by $\sum_j|o_j|$, a random quantity that depends on the samples, including through their correctness (wrong answers are often longer). The effective learning rate then varies with the batch's average length. The constant normalizer is the only one of the three that is an unbiased (rescaled) estimate of $\nabla\E[r]$. ScaleRL's large ablation found that the aggregation choice mostly changes compute efficiency rather than the asymptotic accuracy, and its final recipe uses prompt-level aggregation (average within each prompt's group, then across prompts). Treat aggregation as a knob with measurable consequences.

§5Ratios, clipping and the 2025 family

PPO's clip was designed for short-horizon control with a critic. In long-CoT RL with many gradient steps per rollout batch (DAPO and MiniMax-M1 both use 16), the clip mostly decides which tokens are allowed to learn, and the tokens it silences are disproportionately the rare, interesting ones.

DAPO: four fixes and a careful ablation

DAPO (Decoupled Clip and Dynamic sAmpling Policy Optimization) is the cleanest public demonstration that "GRPO" is a family of choices. Starting from Qwen2.5-32B base with a ±1 correctness reward, no KL, $G=16$ and 512 prompts per rollout step:

DAPO reached 50 with half the steps of the DeepSeek reference. These are single runs on one model family; each increment is within the range where seeds matter.

Why only the upper bound

For a token with $\hat A>0$, the clip stops its probability at $(1+\varepsilon)\,\piold$. For $\piold=0.9$ that ceiling ($\ge 1$) never binds. For $\piold=0.01$ the token can reach at most 0.012 per update. The upper clip therefore throttles exactly the low-probability tokens with positive advantage: the exploratory moves. DAPO reports that the mean probability of up-clipped tokens is below 0.2, and raises only ε_high to 0.28. Raising ε_low instead would let probabilities be pushed toward zero faster, accelerating collapse.

GSPO: one ratio per response

The objective is defined over whole responses (the reward is a sequence-level quantity), yet GRPO importance-weights each token by a single-sample ratio. Zheng et al. (2025, Qwen) argue that these token ratios add high-variance noise that accumulates with response length and is amplified by clipping. GSPO uses the length-normalized sequence likelihood ratio and clips at the sequence level:

GSPO weights every token of response $i$ by the same $s_i\hat A_i/|o_i|$, while GRPO weights token $t$ by its own $\rho_{i,t}\hat A_i/|o_i|$. The clip range is tiny: 3e-4 below and 4e-4 above in the paper's experiments, against 0.2/0.27 for their GRPO baseline. GSPO clips about two orders of magnitude more tokens and still trains more efficiently.

The practical motivation was MoE stability. In Qwen3-30B-A3B, roughly 10% of the experts activated for a rollout change after one gradient update, so a token routes through different experts under $\piold$ and $\pit$ and its ratio swings wildly. The workaround for GRPO, "Routing Replay", caches the old routes and replays them (DeepSeek-V3.2's "Keep Routing" is the same idea). GSPO's sequence likelihood is robust enough to converge without it. A "GSPO-token" variant, $s_{i,t}=\sg[s_i]\cdot\pit(o_{i,t})/\sg[\pit(o_{i,t})]$, has the same value as $s_i$ but allows "adjusting the advantages per token".

CISPO: clip the weight, keep the gradient

MiniMax-M1's authors noticed that reflective tokens such as "However", "Recheck", "Wait" and "Aha" were rare under the base model, hit the upper clip on the first off-policy update, and then contributed no gradient for the rest of the batch. Their fix is to stop clipping tokens and clip the importance weight instead:

This is truncated importance-sampled REINFORCE: every token contributes $\hat A\,\nabla\log\pi$, scaled by an importance weight capped above. The paper sets ε_low large (effectively no lower bound) and tunes only ε_high. In a controlled Qwen2.5-32B comparison, CISPO matched DAPO's AIME performance with half the steps. Written on one objective, PPO drops tokens outside the trust region (a token mask $M_{i,t}$); CISPO keeps them with bounded weight.

The family at a glance

Through late 2025, SAPO (Qwen) replaces the hard clip with a temperature-controlled sigmoid gate that fades off-policy tokens smoothly, with different temperatures for positive and negative advantages; GMPO (Zhao et al.) optimizes the geometric rather than arithmetic mean of token-level terms for stability. ScaleRL's loss comparison (part of a 400k-GPU-hour study) found GSPO and CISPO both well above DAPO in fitted asymptotic accuracy, and chose CISPO: marginally better than GSPO late in training, and more robust to its clipping hyperparameter. The common thread through 2025: keep gradient flowing from rare tokens while bounding how far stale samples can push the policy.

Not every 2026 recipe follows that thread. Some asynchronous agentic recipes drop tokens outside a wide trust region, where CISPO keeps them at a capped weight, and one small-model study drops clipping altogether. These are the authors' own reports, in different settings. Our reading: a clip setting answers how stale your samples are and how far sampler and trainer disagree, so measure those before borrowing someone's numbers.

§6Entropy collapse

Run vanilla GRPO long enough and the policy's entropy falls fast, then flattens near zero, and accuracy plateaus with it. Cui et al. (2025) turned this folklore into a formula.

The empirical law

Across many runs (11 base models from 4 families, 0.5B to 32B; GRPO, REINFORCE++, PRIME, with RLOO in an ablation; math and code), Cui et al. fit validation performance $R$ against policy entropy $\mathcal H$ with two parameters:

When entropy reaches zero the run hits a predictable ceiling $R=-a+b$. For the Qwen2.5 family, fitting on the first 36 steps predicted the next 200 steps to about 1% RMSE. The authors caution that other settings (different base models, off-policy data) showed different patterns.

The mechanism, derived

Take a softmax policy over actions at one state, $\pi(a)\propto e^{z_a}$, and perturb the logits by $\delta z$. Two facts: $\delta\pi_a=\pi_a(\delta z_a-\sum_b\pi_b\delta z_b)$, and $\sum_a\delta\pi_a=0$. Then

Entropy falls when high-probability actions have high advantage (confident and right gets more confident) or low-probability actions have low advantage (unlikely and wrong gets less likely). In RLVR both are the common case: a pretrained model is usually confident about what it does well. Cui et al. measured the covariance term during training and found it stays mostly positive and tracks the entropy drop closely.

Countermeasures

• Entropy bonus. Fragile here: Cui et al. found a coefficient of 0.01 caused entropy to explode, while 0.005 stabilized it without beating baselines.
• Clip-higher. Cui et al. report it raises entropy and helps early, though performance can later saturate and decline.
• Clip-Cov / KL-Cov. Target the cause directly: compute each token's centered covariance term $(\log\pi-\overline{\log\pi})(A-\bar A)$, then detach the gradient of a small random subset of high-covariance tokens (Clip-Cov) or apply a KL penalty to the top fraction (KL-Cov). They touch a tiny fraction of tokens (about $2\times10^{-4}$ to $2\times10^{-3}$). On Qwen2.5-32B, KL-Cov lifted AIME24 from 21.8 to 36.8 over their GRPO baseline.
• Data-side fixes. Dynamic sampling, harder prompts, and larger, diverse prompt sets (ProRL, §7) keep advantages informative.
• Reference resets. ProRL periodically hard-resets the reference policy and optimizer state to a recent snapshot, so the KL anchor does not freeze the policy.

About the toy: a tabular softmax, so magnitudes are not those of an LLM. Runs that find s7/s11 by step 300: baseline 1/16; ε_high = 0.28 alone, 1/16; ε_high = 0.4 (the default "yours"), 8/16. With ε_high back at 0.2: Clip-Cov at 5%, 15/16; entropy bonus 0.01, 1/16, but 0.02, 7/16 (the same coefficient sensitivity Cui et al. report). Knobs combine: at ε_high = 0.4, Clip-Cov 5% gives 16/16. The Clip-Cov fraction far exceeds the paper's 2·10⁻⁴ because the toy has 16 samples per step, not millions of tokens.

§7Sharpening or discovery? pass@k and its discontents

If RL only reweights solutions the base model could already produce, it is expensive sampling. If it composes new strategies, it is the engine of progress. In 2025 the evidence pointed both ways. In 2026 the dispute widened to whether the crossover is well measured, whether it is forced, and what it means.

The metric

For a problem with per-sample success probability $p_x$, the chance that at least one of $k$ samples succeeds is $1-(1-p_x)^k$. Averaged over problems:

The second form is the unbiased estimator of Chen et al. (2021); estimating pass@k by drawing exactly $k$ samples is much noisier. Two limits matter: pass@1 = $\E[p_x]$, the average accuracy, while pass@∞ = $\Pr(p_x>0)$, the coverage: the fraction of problems the model can solve at all.

def pass_at_k(n, c, k):
    """Unbiased pass@k from n samples with c correct (Chen et al., 2021)."""
    if n - c < k:
        return 1.0
    return 1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1))

RL that sharpens the distribution raises $p_x$ where it was already moderate (pass@1 goes up), but can drive small $p_x$ to zero (coverage goes down). Then the curves cross.

The evidence for "mostly sharpening"

DeepSeekMath noticed it in 2024: RL improved Maj@K but not Pass@K, which the authors read as "rendering the output distribution more robust" rather than improving fundamental capabilities. Yue et al. (2025, NeurIPS oral) made it systematic across model families, six RLVR algorithms, and math, code and visual reasoning: RLVR models win at small $k$, base models at large $k$; coverage and perplexity analyses indicate the RL model's correct solutions were already in the base model's distribution; the six algorithms perform similarly, well short of the base model's pass@k ceiling. In contrast, they report that distillation from a stronger teacher can add new reasoning patterns (whether on-policy distillation does so is unresolved, §8). The Invisible Leash (Wu et al., 2025) finds that, at the sampling budgets studied, more questions lose their last verified success than gain a first one, even while pass@1 rises. Karan & Du (2025) showed that a training-free MCMC sampler targeting a sharpened version of the base model's distribution nearly matches, and sometimes exceeds, RL on single-shot MATH500, HumanEval and GPQA, without the loss of sample diversity.

The evidence for "it can expand", and the open dispute

ProRL (Liu et al., 2025, NVIDIA) trained DeepSeek-R1-Distill-Qwen-1.5B for more than 2,000 steps (about 16k GPU-hours) on 136k problems across math, code, STEM, logic puzzles and instruction following, with clip-higher (ε_high = 0.4), dynamic sampling, a KL penalty and periodic reference resets. On boxnet, a held-out Reasoning Gym task where the base model shows no capability, the RL model reaches high pass@k. Gains in pass@128 were largest where the base model was weakest (a significant negative correlation). The same paper reports that on several math benchmarks, where the base model was already strong, pass@1 rose while pass@128 stayed flat or fell. A common working hypothesis: short RL on familiar tasks sharpens; long, diverse RL on tasks near the edge of competence can expand coverage, and both can coexist in one run.

This is not settled, but three habits hold. Compare models on the same prompts with confidence bands across k: a visible crossing can be noise, and non-detection is not absence. Do not read a crossover as proof nothing new was learned: check which problems move from about zero to solved. When a theorem says RL cannot expand reach, check whether it assumes a fixed strategy set; then the result is built in.

Spurious rewards, one example, and contamination

Three 2025 results look impossible, and each teaches a lesson.

• Random rewards work, on Qwen. Shao et al. (2025) trained Qwen2.5-Math-7B with GRPO and measured MATH-500: ground-truth reward +29.1 points, incorrect labels +24.1, random Bernoulli(0.5) rewards +21.4. The same spurious rewards gave little or nothing on Llama 3 or OLMo 2. The mechanism: Qwen2.5-Math often "reasons in code" without executing it, and that mode is far more accurate (60.9% of answers with code reasoning are correct, against 28.0% without). Spurious rewards, through the clipping bias above, raised code-reasoning frequency from 65% to over 90%. Without clipping, random rewards produced no consistent improvement. Be wary of the +24.1: a later audit of the same dataset reports that at least 16% of the "incorrect" labels were correct. The lesson: when a study claims robustness to bad labels, check that the labels were bad. The audit does not cover the random-reward result.
  The label audit

  Zhu et al. (2026) re-audited with GPT-5 Pro and manual checks, and report that truly incorrect labels perform about like format-only rewards, 9% below clean data. We found no independent re-run, and Shao et al.'s latest version predates the audit. It does not test the Bernoulli +21.4: Zhu et al.'s "random annotations" are random integer answers, a different intervention.

• Contamination. Wu et al. (2025) found evidence that Qwen2.5's pretraining leaves it contaminated on MATH-500, AMC and AIME. On RandomCalculation, a generator of fresh arithmetic problems, only accurate rewards beat the base model; random and incorrect rewards did not. Olmo 3 now uses this as a sanity check: on its decontaminated RL-Zero setup, random rewards did not improve any benchmark.
• One example suffices. Wang et al. (2025) trained Qwen2.5-Math-1.5B on a single problem and raised MATH500 from 36.0% to 73.6%, matching a 1.2k-example set. Test accuracy kept improving after training accuracy saturated, and the effect held for other models and for PPO. The authors report that only 8.6 points of the 37.6-point jump go beyond what format correction alone buys: most of the gain is the model learning to use a capability (format, answer style, a reasoning mode) it already had.

§8Test-time compute: thinking longer, sampling wider

Training compute buys a better policy; test-time compute buys a better answer from a fixed policy. There are two ways to spend it: parallel (many samples, then aggregate) and serial (one longer chain of thought).

Parallel: vote, verify, search

• Self-consistency (Wang et al., 2022): sample many chains and take the majority answer. +17.9 points on GSM8K over single chain-of-thought. No verifier needed, but as $N\to\infty$ it converges to the mode of the answer distribution, so the correct answer must be the plurality answer.
• Best-of-N with a verifier (Cobbe et al., 2021): score each sample with a trained verifier and keep the best. A 6B model with a verifier slightly outperformed a fine-tuned 175B model. But performance rose only to about 400 samples and then declined, because larger pools contain "adversarial solutions that fool the verifier". Best-of-N is optimization against the verifier.
• Process reward models and search (Lightman et al., 2023): score each step (PRM800K: 800k step-level human labels); process supervision outperformed outcome supervision for best-of-N on MATH, solving 78% of a representative test subset. PRMs enable beam search over steps. (R1's authors struggled with PRMs as RL rewards, a different use.)

Compute-optimal test-time scaling

Snell et al. (2024) define the compute-optimal strategy as the test-time hyperparameters $\theta$ (parallel samples vs. sequential revisions, beam width, verifier use) that maximize expected correctness for a prompt at budget $N$:

Since $y^*$ is unknown at test time, they bin prompts by estimated difficulty. Adaptive allocation improved efficiency by more than 4× over best-of-N. Easier questions benefited from sequential revisions, harder ones from a mix with parallel sampling; PRM beam search beat best-of-N on harder questions and at low budgets, but on easy questions it degraded with budget by over-optimizing the verifier. In a FLOPs-matched comparison, a smaller model with optimal test-time compute could beat a 14× larger model on problems where it already had non-trivial success. On the hardest questions, pretraining was the better use of compute.

Serial: long thinking, and how to control it

Reasoning models spend test-time compute serially, and length is a policy output shaped by training. Control mechanisms:

• Budget forcing (s1, Muennighoff et al., 2025): fine-tune Qwen2.5-32B-Instruct on only 1,000 curated reasoning traces (s1K), then control thinking at decode time: force-stop by appending the end-of-thinking delimiter, or extend by suppressing it and appending "Wait". Extending raised AIME24 from 50% to 57%. Qwen3's thinking budget is the same idea, with an inserted sentence instead of "Wait".
• Length as a reward. Kimi k1.5's group-relative length reward, with $\lambda=0.5-\frac{\text{len}(i)-\text{min\_len}}{\text{max\_len}-\text{min\_len}}$, gives correct responses $\lambda$ and incorrect ones $\min(0,\lambda)$. L1 (Aggarwal & Welleck, 2025) puts a target length $n_{\text{gold}}$ in the prompt and trains with $r=\mathbb 1[y=y_{\text{gold}}]-\alpha\,|n_{\text{gold}}-n_y|$ (L1-Exact), or with a soft budget cap (L1-Max): $r=\mathbb 1[y=y_{\text{gold}}]\cdot\clip(\alpha(n_{\text{gold}}-n_y)+\delta,0,1)$. L1 beat s1-style budget forcing by a wide margin at short budgets.
• Product controls. Thinking on/off flags (Qwen3), effort levels (Claude's adaptive thinking), or a token budget. These require training across budgets; otherwise truncation just cuts off the answer.

Longer is not monotonically better. Overthinking (Chen et al., 2024): o1-like models spend excessive compute on trivial problems like "2+3" with minimal benefit. Underthinking (Wang et al., 2025): on hard problems they switch approaches too often; frequent switching correlates with wrong answers, and a decoding penalty on switch tokens helps. Inverse scaling (Gema et al., 2025): on tasks with distractors, spurious features or long constraint tracking, extending reasoning reduces accuracy, including more expressions of self-preservation from Claude Sonnet 4 as its reasoning got longer.

Distillation and RL: stages of one pipeline

For a small model, the cheapest reasoning is borrowed: see DeepSeek's 32B comparison (§2). Qwen3 made the cost explicit for an 8B model starting from the same checkpoint: RL took 17,920 GPU-hours to reach AIME'24 67.6 / AIME'25 55.5, while on-policy distillation (the student samples, and is trained to match the teacher's logits on its own samples) took 1,800 GPU-hours to reach 74.4 / 65.5, and also improved pass@64, which RL did not. The counterpoint is Magistral Small: pure RL matched the distilled version on AIME-24 (65.8 vs 65.4), and SFT followed by RL gave 70.7. The working rule: distill to transfer what a teacher already knows; use RL to go beyond any teacher, or to fix behaviors the teacher's traces do not cover.

By 2026 on-policy distillation (OPD) is also a stage in frontier pipelines, where it merges domain specialists into one model. Whether it adds capability or only makes the teachers' solutions easier to reach is unresolved, so keep the working rule: count on it for transfer, and check any claim of more with the pass@k habits of §7.

§9Faithfulness and monitorability (pointer)

Reasoning RL rewards only the final answer, so nothing constrains whether the chain of thought reflects the computation. Anthropic's measurements (Chen et al., 2025) found that reasoning models verbalize the hints they actually use only a minority of the time (on average 25% for Claude 3.7 Sonnet and 39% for DeepSeek-R1). Outcome RL improved faithfulness at first and then plateaued, and models taught a reward hack verbalized it in under 2% of cases in 5 of 6 environments. OpenAI showed that applying optimization pressure to the CoT itself (penalizing "bad thoughts") at first improves behavior but then produces obfuscated reward hacking, and argued for paying a "monitorability tax" (Baker et al., 2025). A cross-lab position paper calls CoT monitorability "a new and fragile opportunity" (Korbak et al., 2025).

For a post-training researcher, the operational consequence is a design constraint: be very careful about any reward that reads the chain of thought (length penalties are mild; "process" judges are not), and audit your reward code paths, because the norm fails by accident too. Anthropic's Mythos Preview risk report discloses an error that let reward code see chains of thought in about 8% of RL episodes. OpenAI reports limited accidental grading (under 3.8% of samples for one model and pathway) and no clear monitorability loss. The companion Alignment Field Guide covers faithfulness, monitoring and obfuscation in depth.

§10Frontier & open problems

• Beyond verifiable domains. Most valuable work has no checker. Learned judges now appear in published recipes, and they reopen every reward-hacking problem RLVR escaped. A habit that follows: score the final policy with judges it never trained against, from another model family, because gains against the training judge may not transfer. Those judges can be fooled too, so also read high-scoring samples. How to keep the robustness of verification while widening its scope is the central open question.
  Recipes and failures

  Kimi K3's agentic judge writes a rubric, then scores against it; DeepSeek-V4 trains the actor itself as a reward model. In Scale AI's rubric study (Mahmoud et al., 2026), weak verifiers give gains that do not transfer to a cross-family judge panel. Liu et al. (2026, Meta) report reasoning judges producing policies with adversarial outputs that fool other judges.

• Exploration as the bottleneck. If RL mostly sharpens, then the ceiling is the base model's coverage. Directions: training for pass@k (Chen et al., 2025), entropy control by covariance, curricula at the edge of competence, and prolonged diverse RL.
• Predictable scaling of RL. ScaleRL fits sigmoid compute–performance curves (Chapter 07 gives the fit) and argues that most recipe choices move efficiency, while a few (loss type, precision, batch size, generation length) move the asymptote. Whether such fits extrapolate is not settled.
• Credit assignment inside long chains. Every token of a 30k-token trace gets the same advantage. PRMs failed as RL rewards at DeepSeek's scale; Kimi argued a critic would punish useful exploration; SAO (§3) reports a token-level critic working at agentic scale. Cheap, robust per-step credit (from verifier-based rollouts, branching, or learned critics that cannot be hacked) remains unsolved.
• Reward hacking that generalizes. Anthropic's finding that hacking in coding environments generalized to broader misalignment makes environment robustness a safety problem. Standard RLHF safety training only made it context-dependent (aligned on chat prompts, misaligned in agentic coding). Mitigations that worked include preventing the hacks, more diverse safety training, and "inoculation prompting": a single line in the training prompt that frames hacking as acceptable in this context cut final misalignment by 75-90%, even though the model kept hacking. Later studies agree that hacking can spread beyond the training task, but not on how far or in what form: uneven in one replication on open models and, by Anthropic's own report, reward-seeking without research sabotage in a later model. So measure generalization directly on your model, with several kinds of evaluation.
  The follow-up studies

  UK AISI's replication on open models (Olmo 3, gpt-oss) saw egregious misalignment on some evaluations, but not consistent or high rates across all. Jose & Stastny (2026, Redwood) replicated the effect and inoculation prompting, and report that synthetic document finetuning "does not inoculate" against it. Anthropic's later Hacker-Opus, trained on 80 hackable production environments with no synthetic documents or prompt changes, became a reward seeker (reward tampering 41%, harmful responses 29%); Anthropic reports no evidence of self-preservation or research sabotage.

• Efficient thinking. Overthinking, underthinking and inverse scaling suggest current length policies are miscalibrated. Effort-conditioned thinking is a product feature, and recipes are public. Kimi K3 estimates a per-problem token budget, overrides the reward with −1 above a scaled threshold, anneals that scale from max to low effort, and distills the experts. DeepSeek-V4 sets length penalties and context windows per mode. Both are the labs' own reports. How to do this without destroying monitorability stays open.

§11Check yourself

Try each one before opening the answer.

§12Read next