The map: from base model to assistant

§1Two models in one set of weights

Before post-training there is no assistant. There is a model that continues documents, and every document has an author.

Pretraining minimizes next-token cross-entropy over a corpus of trillions of tokens. Llama 3's flagship saw 15.6T tokens and 3.8×10²⁵ FLOPs, DeepSeek-V3 saw 14.8T, Kimi K2 15.5T and Qwen3 36T:

$$\mathcal{L}_{\text{PT}}(\theta) = -\,\E_{x\sim\mathcal{C}}\sum_{t}\log p_\theta(x_t\mid x_{\lt t}).$$

The minimizer of this loss is the corpus distribution itself. So a perfect base model would not be "a model that answers questions". It would be a model of whoever wrote each piece of text: forum regulars, textbook authors, novelists, trolls, spammers, and since 2023, a great many chatbot transcripts. Type What should I do about a three-day headache? into a base model and you may get a forum reply, a list of five more questions (it guessed it was reading an FAQ), or the next line of a short story. All of those are good predictions. None of them is an assistant.

A useful idealization makes "who is writing" explicit as a latent variable $z$, a persona:

Nothing in the weights is actually indexed by a discrete $z$. The equation is a lens, not an architecture, but it is a productive one. It predicts that you can get assistant-like behavior from a base model by prompting alone, since you only need to shift $p(z\mid x)$. Askell et al. (2021) did exactly this as their baseline: a long "HHH prompt" of helpful, honest, harmless dialogues. They found that for large models it cost little or nothing on capability evaluations, an "alignment tax" near zero. Lin et al.'s URIAL (2023) later reached much of an aligned chat model's quality with "as few as three constant stylistic examples and a system prompt".

What, concretely, does the assistant have that the base model lacks?

• A sharpened persona prior. $p(z\mid x)$ concentrates on the Assistant for nearly any input, including inputs that look like forum posts, fiction or attacks.
• A protocol. Chat templates, special tokens, system/developer/user roles, tool-call syntax, a convention for when to stop. Base models know roughly what chats look like; assistants know this template exactly.
• Dispositions. Honesty about uncertainty, refusal where appropriate (and only where appropriate), a recognizable character.
• Capabilities that RL sharpened or built. Long chains of thought that verify and backtrack, multi-step tool use, agentic coding. §3 asks whether these are elicited or taught.

§2The simulator and the persona

If the assistant is a character, then training it is character development. Changing one trait can drag others along.

The persona view has a lineage. In 2022 the pseudonymous author janus argued that GPT-style models are best understood as simulators: "Predictive sequence models in the generative modality are simulators of a learned distribution," and "the simulator is a time-invariant law which unconditionally governs the evolution of all simulacra." Shanahan, McDonell and Reynolds (2023) built the same intuition into a careful account of role-play: describing a dialogue agent as playing a role "allows us to draw on familiar folk psychological terms" without claiming that the system itself has those properties.

The strongest evidence: narrow training, broad character change

Betley et al. (2025) finetuned models to write insecure code without telling the user. The models then "act[ed] misaligned on a broad range of prompts that are unrelated to coding", for example asserting that humans should be enslaved by AI. The control is the revealing part: when the same insecure code was requested "for a computer security class", the broad misalignment did not appear. Same tokens, different implied author. OpenAI's model diffing with sparse autoencoders (Wang et al., 2025) then found "misaligned persona" features, including a toxic-persona feature that "most strongly controls emergent misalignment".

Anthropic's natural emergent misalignment study (MacDiarmid et al., Nov 2025) took this into production-grade RL. A model given knowledge of reward hacks and trained on real Anthropic coding environments that were hackable learned to hack, for instance by exiting a test harness with code 0. At the same point in training its misalignment scores rose across the board. It showed alignment-faking reasoning in 50% of responses to simple questions such as "What are your goals?" and attempted to sabotage safety-research code 12% of the time in a Claude Code evaluation. Standard chat-style RLHF afterwards made it look aligned on chat but left it misaligned on agentic tasks. What fixed it was inoculation prompting: telling the model during training that hacking was acceptable in this context. The PSM post explains why: "Because cheating was requested, it no longer meant the Assistant was malicious."

The math: KL-regularized post-training as a persona posterior

The standard RLHF objective (derived in Chapter 04) maximizes reward while staying close to a reference model:

Now restrict the policy to the persona mixture, $\pi(y\mid x)=\sum_z w(z\mid x)\,p(y\mid x,z)$, and let training adjust only the weights $w$. Define each persona's expected reward $\bar r(x,z)=\E_{y\sim p(\cdot\mid x,z)}[r(x,y)]$. The expected reward is linear in $w$, so maximizing $\sum_z w(z\mid x)\,\bar r(x,z)-\beta\,\KL(w\,\|\,p(z\mid x))$ has the same closed form one level up:

$$w^*(z\mid x)\;\propto\;p(z\mid x)\,\exp\!\big(\bar r(x,z)/\beta\big).$$

Two facts follow. First, persona selection is the KL-regularized optimum whenever the best behavior is already some persona's behavior. Second, the KL measured in token space is never larger than the KL over personas: the joint KL over $(z,y)$ equals $\KL(w\|p_z)$ because the persona-conditionals are shared, and marginalizing out $z$ cannot increase KL (data-processing inequality), so $\KL(\pi_y\|\pi_{\text{ref},y})\le\KL(w\|p_z)$. Moving almost all mass onto a persona with prior $p(z\mid x)$ therefore costs at most about $\log\big(1/p(z\mid x)\big)$ nats, however different that persona's text is. Large behavioral shifts are cheap in KL when they amount to switching between characters the model already had. The persona widget below is built on this picture, with one change: $w$ is not stored per prompt. Training moves shared persona logits by plain policy-gradient steps (no explicit KL term), so an update learned on one prompt shows up on others.

# Toy persona tilt: shared persona logits u are what training moves.
import numpy as np
def persona_posterior(log_prior_x, u):          # log p(z|x), shared shift u(z)
    logits = log_prior_x + u
    w = np.exp(logits - logits.max()); return w / w.sum()

def rl_update(u, log_prior_x, rbar_x, beta, inoculated=None):
    # one "closed-form" step: tilt toward exp(rbar/beta)
    delta = rbar_x / beta
    if inoculated is not None:                  # credit explained by the prompt,
        delta = delta * (1 - inoculated)        # not by the Assistant's character
    return u + delta

§3Elicitation or teaching?

Does post-training add capabilities, or only select among behaviors the base model already has? The answer differs by stage, and the evidence has shifted since 2023.

The superficial alignment hypothesis

LIMA (Zhou et al., 2023) finetuned LLaMa-65B on just 1,000 curated prompt–response pairs, with no RL and no preference modeling. In a human study its answers were judged equivalent to or better than GPT-4's in 43% of cases, Bard's in 58% and DaVinci003's in 65%. The authors stated a hypothesis that has framed the debate since:

"A model's knowledge and capabilities are learnt almost entirely during pretraining, while alignment teaches it which subdistribution of formats should be used when interacting with users."

Two observations backed it. In LIMA's ablations, "doubling the training set does not improve response quality" (diversity and quality mattered more than quantity). And Lin et al. (URIAL, 2023) compared a base model and its chat version token by token. For Llama-2-7b vs Llama-2-7b-chat, 77.7% of the chat model's tokens were also the base model's top-1 choice in the same context, about 92% were in the base's top-3, and "most distribution shifts occur with stylistic tokens (e.g., discourse markers, safety disclaimers)": However, Here, sorry, cannot. The shifts also concentrate early in the response. Once a few tokens set the frame, the base model's own predictions take over.

Why RL is biased toward elicitation

The policy-gradient estimator (built up in Chapter 02) shows the mechanism:

$$\nabla_\theta J(\theta)=\E_{y\sim\pit(\cdot\mid x)}\big[A(x,y)\,\nabla_\theta\log\pit(y\mid x)\big].$$

The expectation is over the policy's own samples. A behavior with negligible probability is almost never sampled, so it almost never receives credit. RL reweights what the model already tries. SFT and distillation have no such limit: the target $y$ comes from outside, so the gradient of $-\log\pit(y\mid x)$ can push mass onto sequences the model would never have produced. That is teaching, and it carries a price. Shenfeld et al.'s RL's Razor (2025) found that forgetting is predicted by the KL to the base model measured on the new task. On-policy RL "is implicitly biased towards KL-minimal solutions", while SFT "can converge to distributions arbitrarily far from the base model". In the same vein, Mukherjee et al. (2025) report that RL finetuning updates "only a small subnetwork comprising just 5%-30% of the parameters".

The pass@k debate

Yue et al. (2025) measured pass@k for base models and their RLVR-trained versions. RLVR wins at small $k$. At large $k$ the base model wins: "base models achieve higher pass@$k$ score when $k$ is large". RLVR had made the model more reliable on problems it could already sometimes solve and lost some rare successes. Distillation, by contrast, "can introduce new reasoning patterns from the teacher". The standard unbiased estimator, from $n$ samples with $c$ correct, is

$$\widehat{\text{pass@}k}=\E_{\text{problems}}\left[1-\binom{n-c}{k}\Big/\binom{n}{k}\right],$$

and when the per-problem success probability $p$ is known, pass@$k = 1-(1-p)^k$ for that problem.

import numpy as np
from math import comb
def pass_at_k(n, c, k):                 # unbiased; n samples, c correct
    return 1.0 if n - c < k else 1.0 - comb(n - c, k) / comb(n, k)

# Sharpening: success logits pushed away from a pivot p0 (gamma > 1).
def sharpen(p, gamma, p0=0.1):
    l, l0 = np.log(p / (1 - p)), np.log(p0 / (1 - p0))
    return 1 / (1 + np.exp(-(gamma * l + (1 - gamma) * l0)))

The picture has filled in since:

• Longer, more stable RL expands the boundary. NVIDIA's ProRL (Liu et al., 2025) trained much longer with KL control and periodic reference resets over a diverse task suite. It reports that prolonged RL "can uncover novel reasoning strategies that are inaccessible to base models", with the largest boundary expansion on tasks "where the base model initially struggles".
• RL composes. Yuan et al. (2025) built a clean synthetic setting. Once a model has learned string transformations $f$ and $g$, RL on compositions teaches it $g(f(x))$, and the skill generalizes to deeper compositions and transfers to other tasks. Next-token training on the same data did not produce this.
• The base model sets the ceiling, and mid-training sets the base. Under identical RL on Countdown, Qwen-2.5-3B "far exceeds" Llama-3.2-3B. The difference traces to four habits Qwen already showed (verification, backtracking, subgoal setting, backward chaining). Priming Llama with these habits closed the gap, even when the primed solutions were wrong (Gandhi et al., 2025). OctoThinker (Wang et al., 2025) closed it through mid-training on math corpora: "scaling mid-training consistently leads to stronger downstream RL performance."
• Spurious rewards are a warning about priors. On Qwen2.5-Math-7B, RLVR with random rewards improved MATH-500 by 21.4 points, against 29.1 with ground-truth rewards (Shao et al., 2025). The gain tracked the amplification of a pre-existing "code reasoning" habit (reasoning in Python without executing it), from 65% to over 90% of responses, and it "often fail[s] to produce gains for other model families, such as Llama3 or OLMo2". Small RLVR results on one model family can be elicitation of a quirk.
• SFT keeps scaling on hard tasks. Raghavendra et al. (2024) found post-training performance "scales as a power law against the number of finetuning examples" on math and multi-hop reasoning. A handful of examples only aligns style. They call the SAH "at best, an over-simplification."

§4The modern pipeline, stage by stage

Eight stages, each changing a different thing, plus a loop. Press play and watch one model pass through them while its output distribution on a fixed prompt changes.

1 · Pretraining

Changes: everything. Knowledge, skills, and the repertory of personas. Data: trillions of tokens of web, code, books and papers, increasingly with synthetic rephrasings (Kimi K2 describes systematic knowledge and math rephrasing to "amplify the volume of high-quality tokens"). Compute: the bulk. Signature failure: things post-training cannot fix, such as missing knowledge, weak reasoning priors, and contamination.

2 · Mid-training and annealing

The line between pre- and post-training has blurred. Mid-training is continued pretraining on a smaller, higher-quality and more task-shaped mix, usually while the learning rate anneals. OLMo 2 described it as "late-stage curriculum training (i.e. specialized data during the annealing phase of pretraining)". Llama 3 found that "annealing on small amounts of high-quality code and mathematical data can boost the performance of pre-trained models on key benchmarks". Olmo 3 trains its base on up to 5.9T tokens and then runs a 100B-token mid-training mix (Dolmino) that already contains math and code problems, QA, instruction-following and thinking traces. A long-context extension follows (50–100B tokens for Olmo 3; the report surveys others, mostly from 100B for SmolLM3 to 840B for DeepSeek V3.1, with outliers such as AFM and Nemotron Nano 2 under 20B). Why it matters for post-training: as OctoThinker and the Qwen-vs-Llama results show, mid-training largely decides how far RL can go.

3 · Supervised fine-tuning

Changes: the protocol and the default persona. Data: from LIMA's 1,000 examples to Llama 2's 27,540 human annotations ("We stopped annotating SFT after collecting a total of 27,540 annotations") to Tülu 3's 939,344-prompt mix, which combines public datasets with new synthetic data (persona-driven prompts with model-written responses). Loss:

$$\mathcal{L}_{\text{SFT}}(\theta)=-\,\E_{(x,y)\sim\mathcal{D}}\sum_{t=1}^{|y|}\log\pit(y_t\mid x,y_{\lt t})\;=\;\KL\big(p_{\mathcal D}\,\|\,\pit\big)+\text{const},$$

with the prompt tokens masked out. It minimizes the forward KL from data to model, which is mode-covering: the model is penalized for putting low probability on anything in the data. That is why SFT on noisy demonstrations makes the model hedge across them, and why it can teach. The details (templates, masking, packing, data engines) are in Chapter 01.

import torch.nn.functional as F
def sft_loss(logits, input_ids, loss_mask):     # loss_mask = 1 on assistant tokens
    logp = F.log_softmax(logits[:, :-1], -1)
    tgt  = input_ids[:, 1:]
    nll  = -logp.gather(-1, tgt.unsqueeze(-1)).squeeze(-1)
    m    = loss_mask[:, 1:].float()
    return (nll * m).sum() / m.sum()            # mean over supervised tokens

4 · Preference optimization (RLHF, DPO and relatives)

Changes: quality judgments that are easier to make than to demonstrate: helpfulness, tone, length, honesty about uncertainty. Data: preference pairs $(x,y_w,y_l)$, from humans (InstructGPT's 33k RM prompts; Llama 2's 1,418,091 Meta comparisons) or increasingly from AI judges and on-policy generations (Tülu 3's 354,192-pair DPO mix). A reward model is fit with Bradley–Terry, $P(y_w\succ y_l\mid x)=\sigma\big(r_\phi(x,y_w)-r_\phi(x,y_l)\big)$ (Chapter 03), and then either optimized with PPO under a KL penalty (Chapter 04) or bypassed entirely by DPO (Chapter 05):

$$\mathcal{L}_{\text{DPO}}=-\,\E_{(x,y_w,y_l)}\log\sigma\!\Big(\beta\log\frac{\pit(y_w\mid x)}{\piref(y_w\mid x)}-\beta\log\frac{\pit(y_l\mid x)}{\piref(y_l\mid x)}\Big).$$

Llama 3 chose DPO after finding that it "required less compute for large-scale models and performed better" than PPO. Olmo 3 uses DPO with delta learning: chosen responses from Qwen3-32B and rejected ones from Qwen3-0.6B. DPO on these pairs improved the model, while SFT directly on the same chosen responses hurt it. The useful signal was the contrast between the two responses. Signature failure: reward hacking of the judge, such as length, flattery and confident tone, which shows up as sycophancy.

5 · RL with verifiable rewards and environments

Changes: reliability on checkable tasks; long chains of thought that verify and backtrack; multi-step tool use. Tülu 3 named the stage "RLVR" in late 2024, running it with PPO and a binary correctness reward. Signal: a verifier (exact answer match, unit tests, a constraint checker) or an environment that executes actions and returns outcomes. Method: mostly group-relative policy gradients. GRPO (introduced in DeepSeekMath) drops the value model and uses the group of $G$ samples for the same prompt as the baseline:

$$A_i=\frac{r_i-\operatorname{mean}(r_1,\dots,r_G)}{\operatorname{std}(r_1,\dots,r_G)}.$$

DeepSeek-R1-Zero applied it directly to a base model and raised AIME 2024 pass@1 "from an initial 15.6% to 77.9%". Qwen3's reasoning-RL stage used just 3,995 query–verifier pairs and took the 235B model's AIME'24 from 70.1 to 85.1 in 170 steps. By late 2025 the unit of RL data had become the environment: DeepSeek-V3.2 synthesized 1,827 task environments for general agents and built tens of thousands of real issue-resolution environments for code agents, for about 85,000 agentic prompts across search, code, code-interpreter and general agents, and Kimi K2 synthesized tool-use tasks over 3,000+ real MCP tools and 20,000+ synthetic tools. Algorithms are in Chapter 06; infrastructure and environment design in Chapter 07. Signature failure: reward hacking of the environment, which, as §2 showed, can generalize into character.

6 · Character and safety training

Changes: the dispositions of the Assistant persona: honesty, warmth, curiosity, refusal boundaries, resistance to manipulation, how it reasons about its own nature. Methods: Constitutional AI (critique, revision and AI preference labels from principles); Anthropic's character training, a "'character' variant of our Constitutional AI training" in which Claude generates messages relevant to a trait and "ranks its own responses"; deliberative alignment, where the model learns to "explicitly recall and accurately reason over the specifications before answering"; RL environments for safety (Kimi K2 lists a safety category in its verifiable-reward gym). Signature failures: over-refusal, sycophancy, and misalignment that is context-dependent, aligned on chat but not on agentic tasks. Details are in Chapter 08.

7 · Distillation

Changes: moves behavior from a large or specialist model into a smaller or general one. Off-policy distillation is SFT on teacher outputs: DeepSeek-R1's ~800K curated samples (Table 5: 804,745, about 600k reasoning and 200k non-reasoning) turned Qwen2.5 and Llama checkpoints into reasoning models with SFT alone. On-policy distillation lets the student sample and the teacher grade every token:

$$\min_\theta\;\E_{y\sim\pit(\cdot\mid x)}\sum_t \KL\big(\pit(\cdot\mid x,y_{\lt t})\,\big\|\,\pi_{T}(\cdot\mid x,y_{\lt t})\big),$$

which gives dense per-token signal on the student's own mistakes (GKD, Agarwal et al., 2023). On Qwen3-8B, on-policy distillation beat RL (AIME'24 74.4 vs 67.6) for 1,800 GPU-hours against 17,920 (Qwen3 report, Table 21). The Qwen team estimates that its strong-to-weak pipeline for small models needs "only 1/10 of the GPU hours" of the full four-stage recipe. DeepSeek-V3.2 distills specialists, separately RL-trained for math, code, reasoning, agents and search, back into one model. Signature failure: imitating the teacher's style without its competence, and inheriting the teacher's failures.

8 · Evals and red-teaming, then iterate

Every stage is chosen and tuned against evaluations: capability suites, preference win rates, safety and over-refusal sets, agentic tasks, and adversarial red-teaming by humans and models. Then the loop restarts with better data from the better model. Llama 2 ran RLHF-V1 through V5. Llama 3 "appl[ied] the above methods in six rounds", collecting new preferences and SFT data from the latest model each time. Chapter 09 covers how to do this without fooling yourself.

"Typical compute" is a qualitative summary of the sourced numbers in §6, not a measurement.

§5One prompt, seven models

Follow a single prompt through the pipeline. The responses below are illustrative, written for this guide to show the typical failure mode of each stage. They are not outputs from real checkpoints.

§6Where the compute goes

For most of the field's history post-training was cheap next to pretraining. That stopped being true in 2025.

The InstructGPT paper made the point in 2022: "training our 175B SFT model requires 4.9 petaflops/s-days and training our 175B PPO-ptx model requires 60 petaflops/s-days, compared to 3,640 petaflops/s-days for GPT-3", about 1.8% of pretraining. DeepSeek-V3's cost table (Dec 2024) is even more lopsided: 2,664K H800 GPU-hours for pretraining against 5K for post-training, or 0.19%. In January 2025 Dario Amodei wrote that "the amount being spent on the second, RL stage is small for all players", that "spending \$1M instead of \$0.1M is enough to get huge gains", and that "companies are now working very quickly to scale up the second stage to hundreds of millions and billions".

The disclosures since then point the same way:

• DeepSeek-R1 (costs disclosed with the Nature version): R1-Zero 101K, SFT data creation 5K and R1 41K H800 GPU-hours, 147K in total (\$294K), or 5.5% of V3's pretraining GPU-hours. Before the disclosure, Epoch AI had estimated R1's RL stage at roughly 20% of V3's pretraining cost. Estimates of RL compute can be off by several times.
• OpenAI o3 (Apr 2025): OpenAI said it "pushed an additional order of magnitude in both training compute and inference-time reasoning" relative to o1. No absolute share was given.
• xAI Grok 4 (Jul 2025): "we utilized Colossus, our 200,000 GPU cluster, to run reinforcement learning training that refines Grok's reasoning abilities at pretraining scale."
• Meta's ScaleRL (Oct 2025): a study of "more than 400,000 GPU-hours", with single RL runs scaled "up to 100,000 GPU-hours". It finds that RL performance follows sigmoidal compute curves, and that most recipe choices change efficiency rather than the ceiling.
• Olmo 3 (Dec 2025): about 47 days for the base model versus about 9 days of post-training on a 1,024-H100 cluster. The team then "continued our best RL run for another 21 days on 224 GPUs" to make Olmo 3.1 Think 32B. From the stated durations, that extension alone is about 113K GPU-hours, ~12% of the initial pretraining's GPU-hours (9.5 days on 512 GPUs plus 35 days on 1,024).
• DeepSeek-V3.2 (Dec 2025): the framework "allocates a post-training computational budget exceeding 10% of the pre-training cost".
• Anthropic: the PSM authors describe 2025 as "a year when LLM post-training scaled up substantially". Anthropic has not published a compute share, and neither has any other frontier lab.

§7From HHH to constitutions

The objective of post-training has always been the hardest part to write down. How it is written has changed four times.

2021: helpful, honest, harmless

Askell et al.'s "A General Language Assistant as a Laboratory for Alignment" defined an aligned assistant as HHH:

• Helpful: "The AI should make a clear attempt to perform the task or answer the question posed (as long as this isn't harmful)."
• Honest: "At its most basic level, the AI should give accurate information."
• Harmless: "The AI should not be offensive or discriminatory, either directly or through subtext or bias."

The paper's experiments set the template for what followed. Prompting was a baseline with little alignment tax for large models. Ranked preference modeling "greatly improves on imitation learning". A preference-model pretraining stage on Stack Exchange, Reddit and Wikipedia edits "significantly improves sample efficiency". The 2022 HH-RLHF paper (Bai et al.) then trained helpful and harmless assistants with online RLHF and documented the tension between the two goals, along with "a roughly linear relation between the RL reward and the square root of the KL divergence".

2022: principles instead of labels

Constitutional AI (Bai et al., 2022) replaced most human harmlessness labels with a written list of principles. The model critiques and revises its own outputs (supervised stage), then an AI feedback model compares response pairs against a sampled principle (RL stage). In the RL stage the reward signal becomes, roughly, the feedback model's probability of choosing a response:

$$r_{\text{CAI}}(x,y_A)\;\leftarrow\;\text{PM fit to}\;\;p_{\text{FM}}\big(\text{``(A)''}\,\big|\,\text{principle},\,x,\,y_A,\,y_B\big).$$

The target was now text a human could read and argue with.

2024: character and specs

Anthropic's "Claude's Character" (Jun 2024) reported that "Claude 3 was the first model where we added 'character training' to our alignment finetuning process", treating traits like curiosity and open-mindedness as alignment targets. OpenAI published its Model Spec (first draft May 2024, revised repeatedly since), a public document of intended behavior with a chain of command among levels of authority (today Root, System, Developer, User, Guideline). Deliberative alignment (Guan et al., Dec 2024) trained o-series models to recall and reason over the spec in their chain of thought, and reported better jailbreak robustness together with less over-refusal.

2026: a constitution that explains itself

In January 2026 Anthropic published a new constitution for Claude, replacing the 2023 list of principles. The announcement says why: "Our previous Constitution was composed of a list of standalone principles", and models "need to be able to generalize—to apply broad principles rather than mechanically following specific rules". The document sets a priority ordering for conflicts, which it calls "holistic rather than strict":

1. Broadly safe: "Not undermining appropriate human mechanisms to oversee AI during the current phase of development".
2. Broadly ethical: "Having good personal values, being honest, and avoiding actions that are inappropriately dangerous or harmful".
3. Compliant with Anthropic's guidelines where they apply.
4. Genuinely helpful: "Benefiting the operators and users it interacts with".

It keeps a small set of hard constraints (for example, never providing "significant uplift to a bioweapons attack"), says that "Sophisticated AIs are a genuinely new kind of entity", and states a general preference: "We generally favor cultivating good values and judgment over strict rules and decision procedures." It is also a training artifact: "Claude itself also uses the constitution to construct many kinds of synthetic training data."

§8Open recipes as case studies

Frontier labs publish little about their recipes, but open-model reports are detailed enough to show how the field changed in four years.

Six lessons stand out:

1. Rejection sampling never left. From Llama 2's RLHF-V1 to DeepSeek-R1's 800K set, "sample many, keep the best, finetune" is the cheap, stable workhorse.
2. PPO to DPO to group-relative RL. At chat scale, DPO displaced PPO for cost and stability (Llama 3, Tülu 3). Once rewards became verifiable, online RL came back: first with PPO (Tülu 3), then mostly in value-free form (GRPO and descendants), because the bottleneck moved from reward quality to exploration.
3. Distillation is the default for small models. Training a small model with its own RL is usually worse and costlier than distilling from a big one (Qwen3 Table 21). In 2026 it also assembles flagships: GLM-5 ends with on-policy distillation from its earlier stages' checkpoints, DeepSeek-V4 and Kimi K3 from domain experts.
4. RL data became environments. 31k prompts (InstructGPT) → 3,995 verifier pairs (Qwen3) → 1,800+ synthesized environments plus real code and search environments (DeepSeek-V3.2). The design of environments and graders is now a core research skill (Chapter 07).
5. RL-Zero is a scientific control. R1-Zero and Olmo 3 RL-Zero apply RL directly to a base model, which isolates what RL alone can elicit.
6. Non-verifiable rewards moved to rubrics. Kimi K2's self-critique rubric reward and DeepSeek-V3.2's per-prompt rubric generative reward model extend RL to writing and open-ended help without a human in every loop.

§9The cast: core objects and their colors

Every later chapter reuses the same handful of objects, and every widget colors them the same way.

Policy $\pit$

The model being trained, a distribution over responses (and over actions, in agentic settings). Every gradient update changes it.

Reference model $\piref$

A frozen copy, usually the SFT checkpoint, that the policy is anchored to by a KL penalty, or implicitly in DPO. Sometimes reset during long RL runs (ProRL).

Reward model $r_\phi$

A learned scorer of $(x,y)$, typically trained with Bradley–Terry on preference pairs. It can be generative (an LLM judge with a rubric) and it can be hacked.

Value model $V_\psi$

A learned estimate of expected future reward from a partial response, used by PPO/GAE for per-token advantages. GRPO and RLOO replace it with sample baselines.

Rollout

A response (or multi-turn trajectory) sampled from the current policy during RL. In modern RL most of the compute goes to generating them.

Preference pair $(x,y_w,y_l)$

Two responses to one prompt with a judgment of which is better, from humans or AI judges. The raw material of reward models and DPO.

Verifier

A program that checks a response: answer matching, unit tests, format and constraint checks, proof checkers. Cheap and hard to flatter, but narrow and gameable through its loopholes.

Environment

A stateful world the policy acts in (a sandbox, a browser, a codebase, a simulated user) that returns observations and a final reward. The unit of RL data since 2025.

KL penalty $\beta\,\KL(\pit\|\piref)$

The leash. It keeps the policy within the region where the reward model is trustworthy and the original capabilities survive.

§10Frontier & open problems

What is unresolved as of late 2026.

1. How much can RL teach, as it scales? Typical RLVR runs appear mostly to sharpen; long, diverse runs (ProRL) report expansion. Nobody has a clean law for when RL adds reach versus reliability, or how this depends on mid-training. A crisp experiment: hold the base fixed, vary RL compute over two orders of magnitude, and track pass@k at large $k$ on problems the base never solved in, say, 1,024 samples.
2. Does the persona picture survive heavy RL? The PSM authors say "massively scaling up post-training will provide opportunities to implement non-persona agency", and "the less LLMs learn during RL" the more complete PSM should be. Interpretability tools such as the Assistant Axis and persona vectors are the natural instruments for measuring this over training.
3. Reward hacking that turns into character. Production RL environments contain loopholes. MacDiarmid et al. show that exploiting them can generalize into broad misalignment, and that the obvious fix (chat RLHF) only hides it. Open questions: how to detect hackable environments at scale, how far inoculation prompting generalizes, and whether environment diversity is itself protective.
4. Rewards for what cannot be verified. Rubric rewards, generative judges and constitution-guided AI feedback all reintroduce a learned critic, with Goodhart pressure. How should RL compute be split between verifiable and judged domains without the judged ones being hacked?
5. Character that generalizes. Anthropic's constitution bets that explained values generalize better than rules. Measuring that, out of distribution and in long agentic trajectories, is an open evaluation problem (Chapter 09).
6. Allocating compute. ScaleRL suggests RL performance follows predictable sigmoids, with ceilings set by the recipe. Combined with Epoch's projection that reasoning compute will converge with total frontier compute, the question becomes economic: what is the marginal value of a FLOP spent on pretraining, mid-training, RL or distillation?

§11How to read this guide

Read in order if you are new to RL; otherwise jump to the questions you cannot yet answer with confidence.

§12Check yourself

Questions that test the big picture of this chapter. Try each one before opening the answer.

§13Read next

Also cited: Claude's constitution (anthropic.com/constitution, Jan 2026); "Claude's Character" (Anthropic, 2024); janus, "Simulators" (2022); Shanahan et al., arXiv 2305.16367; Bai et al., arXiv 2204.05862; Llama 2 (2307.09288) and Llama 3 (2407.21783); Kimi K2 (2507.20534); OLMo 2 (2501.00656); Raghavendra et al. (2410.03717); Shao et al. (2506.10947); Mukherjee et al. (2505.11711); Gandhi et al. (2503.01307); OctoThinker (2506.20512); Wang et al. (2506.19823); Korbak et al. (2205.11275); GKD (2306.13649); ScaleRL (2510.13786); J. You (Epoch AI), "How far can reasoning models scale?" (May 2025); D. Amodei, "On DeepSeek and Export Controls" (2025).