Supervised fine-tuning & the data engine

§1The objective: conditional maximum likelihood

Take a pretrained model that continues documents. Show it examples of the documents you wish it would write, and raise their probability. That is all SFT is, and almost every subtlety comes from the word "examples".

A dataset $\mathcal{D}$ holds pairs $(x, y)$: a context $x$ (system prompt, conversation so far) and a target response $y = (y_1,\dots,y_T)$. The model $\pit$ factorizes the response autoregressively. SFT minimizes the negative log-likelihood of the targets, but only on the tokens you choose to train on, selected by a binary mask $m_t$:

Two equivalent readings matter for the rest of the guide. First, since $\E_{y\sim p}[-\log q(y)] = H(p) + \KL(p\,\|\,q)$ and the entropy of the data does not depend on $\theta$, minimizing the expected NLL is minimizing

$$\E_{x}\Big[\KL\big(p_{\text{data}}(\cdot\mid x)\,\big\|\,\pit(\cdot\mid x)\big)\Big],$$

the forward KL from the data to the model. Forward KL punishes the model for putting low probability anywhere the data puts mass, so it is mode-covering. Keep this in your pocket until §4.

Second, SFT is behavior cloning. Treat the context plus partial response as a state $s_t = (x, y_{\lt t})$, the next token as an action $a_t = y_t$, and appending as a deterministic transition. A dataset of demonstrations is a dataset of expert state-action pairs, and cross-entropy on them is the textbook behavior-cloning loss. The RL chapters (02, 06) will use exactly this MDP; the difference is only whose trajectories you learn from.

Teacher forcing and exposure bias

Teacher forcing makes training embarrassingly parallel: one forward pass computes the loss at every position, because every prefix is known in advance. The price is a mismatch. At training time the prefix $y_{\lt t}$ comes from the data. At inference time it comes from the model. If the model makes a mistake the data never contains, it is now in a state it was never trained on. This is exposure bias.

Imitation learning makes this quantitative. Let $\varepsilon$ be the per-step error rate of the cloned policy on the expert's state distribution (roughly what your validation loss measures). In the worst case, once the policy leaves the expert's states it never recovers and pays a cost of 1 on every remaining step. The probability of having left by step $t$ is at most $t\varepsilon$ (union bound), so

$$\text{expected cost} \;\le\; \sum_{t=1}^{T} t\,\varepsilon \;\approx\; \tfrac{1}{2}\,T^2\varepsilon \qquad\text{versus}\qquad T\varepsilon \text{ under teacher forcing.}$$

This is the classical $O(T^2\varepsilon)$ behavior-cloning bound (Ross & Bagnell; restated for language generation by Arora et al., 2022 as $T\varepsilon \le \mathcal{R}(\pi_{\text{BC}}) \le T^2\varepsilon$). Methods that train on the learner's own states, such as DAgger, bring it back to linear. Play with the toy below: the blue dashed line is what your loss curve "sees"; the clay line is what users experience.

So is exposure bias the reason LLMs hallucinate and loop? The evidence is mixed, and the disagreement is itself worth understanding. He et al. (2019) fed models ground-truth versus self-generated prefixes and found the distortion "is limited, and does not seem to be incremental", attributing it to a self-recovery ability of language models (one intuition: a model pretrained on messy text has seen many recoveries from small errors, so its own slips rarely push it off the manifold). Arora et al. (2022) measured error accumulation directly and argued it is real and that perplexity fails to capture it. Both can be true: the self-recovery slider in the widget is exactly the quantity they disagree about, and it is plausibly larger for big pretrained models than for small task-specific ones (an intuition, not a measured result).

A subtler failure is that teacher forcing can make the task easier than it really is. Bachmann & Nagarajan (2024) built a path-finding task where, given the true prefix of the answer, later tokens are trivially predictable, so the model learns a "Clever Hans" shortcut that uses the revealed prefix and never learns to plan the first token. At inference there is no revealed prefix and the model fails. The models were trained to perfect training accuracy, yet failed on held-out graphs drawn from the same distribution: the loss curve gave no warning.

SFT is a (badly weighted) policy gradient

Here is a derivation worth working through by hand. For a single demonstration $y^*$, write the SFT gradient as an expectation under the model's own distribution:

$$\nabla_\theta \log \pit(y^*\mid x) \;=\; \sum_{y}\pit(y\mid x)\,\frac{\mathbb{1}[y=y^*]}{\pit(y\mid x)}\,\nabla_\theta\log\pit(y\mid x) \;=\; \E_{y\sim\pit}\!\Big[\underbrace{\tfrac{\mathbb{1}[y=y^*]}{\pit(y\mid x)}}_{\text{implicit reward}}\nabla_\theta\log\pit(y\mid x)\Big].$$

So SFT is REINFORCE with a sparse reward that is inversely proportional to the model's own probability of the demonstration. Demonstrations the model finds unlikely get enormous effective reward. Wu et al. (2025) use this reading to argue that the $1/\pit$ weight is what makes SFT generalize poorly and propose Dynamic Fine-Tuning: multiply each token's loss by its (stop-gradient) probability, cancelling the weight. TRL now exposes it as loss_type="dft". Treat the empirical claims as one group's results; the derivation itself is exact and is the cleanest way to explain why SFT on surprising data moves a model so far (§9).

What SFT actually changes: the superficial alignment hypothesis

LIMA (Zhou et al., 2023) fine-tuned a 65B LLaMa on just 1,000 curated examples and found its answers equivalent to or preferred over GPT-4 in 43% of human comparisons (58% against Bard, 65% against DaVinci003). From this they proposed the Superficial Alignment Hypothesis: "a model's knowledge and capabilities are learnt almost entirely during pretraining, while alignment teaches it which subdistribution of formats should be used when interacting with users." Lin et al. (2023) then compared base and chat models token by token and found they agree on the top-ranked token at most positions; "most distribution shifts occur with stylistic tokens (e.g., discourse markers, safety disclaimers)."

Hold the hypothesis loosely. It describes chat-format SFT on a strong base model. It does not describe SFT on long reasoning traces (which demonstrably teaches skills, §6), SFT on narrow data that changes values (§9), or SFT on new facts: Gekhman et al. (2024) found that fine-tuning examples carrying knowledge the model lacks are fitted much more slowly than known ones and, once fitted, increase the model's tendency to hallucinate. "Superficial" is about what a small dataset can do, not a ceiling on what SFT can do.

§2Chat templates, special tokens and loss masks

The model never sees "messages". It sees one long token sequence, and you decide which of its tokens produce gradient. Scroll through how a conversation becomes training data.

Roles and system prompts

Roles are not just labels; they are conditioning tokens the model learns to associate with behaviors. The system role is where operators put persistent instructions, and models only follow system prompts well if training data contains diverse system prompts that actually matter for the response. If every training example has the same default system prompt (or none), the model learns to ignore that slot. Multi-turn data teaches the model to use earlier turns; LIMA found that adding only 30 multi-turn dialogue chains to its 1,000 single-turn examples raised the share of excellent dialogue responses from 45.2% to 76.1%.

The end-of-turn token is a learned behavior

A model stops when it samples the end token. It only learns to do that if the end token is in the loss. Two classic bugs produce models that ramble until max_new_tokens: (a) the collator masks the end token together with the prompt, or (b) the pad token is set equal to the EOS token and padding positions are masked, which also masks the real EOS. LIMA sidestepped the ambiguity by introducing a dedicated end-of-turn token that "plays the same role as EOS of halting generation, but avoids conflation with any other meaning that the pretrained model may have imbued into the preexisting EOS token." Tülu 3's ablation found that replacing the trailing newline after assistant messages with an EOS token gave the best score among the templates they tried (they kept their template anyway, for consistency with later pipeline stages).

Masking schemes and multi-turn conversations

Three choices are common: train on everything (the old "instruction tuning as language modeling" approach); train on the final assistant turn only (each multi-turn conversation contributes one target); or train on all assistant turns (efficient: one forward pass trains every turn, each conditioned on the true history). Meta's Llama 2 paper is explicit: "we zero-out the loss on tokens from the user prompt, so as a result, we backpropagate only on answer tokens." In TRL this is assistant_only_loss=True, which requires the chat template to mark assistant spans with {% generation %}…{% endgeneration %} (current TRL patches the template automatically for some known families, e.g. Qwen3); for prompt–completion datasets, completion-only loss is the default.

Why mask prompts at all? The prompt tokens are not your target behavior: training on user turns teaches the model to write like users (and to role-play the user if it ever crosses a boundary). With short answers and long prompts, unmasked prompt tokens can also dominate the gradient. The counter-argument (prompts are free extra language-modeling signal) mostly matters in tiny-data regimes.

Per-token versus per-sequence averaging

Now the normalizer $Z$. Let sequence $i$ in a batch of $B$ have $n_i$ loss tokens with per-token losses $\ell_{i,t}$. Three common reductions:

$$\underbrace{\frac{\sum_i\sum_t \ell_{i,t}}{\sum_i n_i}}_{\text{per-token mean}} \qquad \underbrace{\frac{1}{B}\sum_i \frac{1}{n_i}\sum_t \ell_{i,t}}_{\text{per-sequence mean}} \qquad \underbrace{\sum_i\sum_t \ell_{i,t}}_{\text{sum}}$$

Under the per-token mean, every token weighs the same, so a sequence's total influence is proportional to its length: a 2,000-token answer pulls 100 times harder than a 20-token one. Under the per-sequence mean, every sequence weighs the same, so each token of a long answer is down-weighted by $1/n_i$: the model learns long answers less thoroughly, token for token. The sum behaves like the per-token mean but its scale depends on how many tokens land in a batch, so the learning rate must absorb it.

This is not academic. In October 2024 several frameworks discovered that gradient accumulation silently implemented the wrong reduction: each micro-batch computed its own per-token mean and the micro-batch losses were then averaged. That is a per-micro-batch weighting, which up-weights tokens in micro-batches that happen to contain few tokens. Hugging Face's write-up states the fix plainly: "the correct loss should be computed by the total loss across all batches in a gradient accumulation step divided by the total number of all non padding tokens in those batches." The same issue arises from averaging across data-parallel devices. The Tülu 3 team ran into it and chose a different fix: "we opted generally to use a sum loss instead of averaging ('mean loss') … This effectively weights all tokens equally (which we found led to generally better performance for initial mixtures)", retuning the learning rate (their best: sum loss, LR $5\times10^{-6}$, 2 epochs).

import torch.nn.functional as F

def sft_loss(logits, labels, n_tokens_global):
    """logits: [B, L, V]; labels: [B, L] with -100 on masked positions.
    n_tokens_global: number of label != -100 tokens across ALL micro-batches
    and devices in this optimizer step (all-reduce it once per step)."""
    logits = logits[:, :-1].float()          # position t predicts token t+1
    labels = labels[:, 1:]
    loss_sum = F.cross_entropy(logits.reshape(-1, logits.size(-1)),
                               labels.reshape(-1),
                               ignore_index=-100, reduction="sum")
    return loss_sum / n_tokens_global        # true per-token mean over the step
    # (DDP averages gradients over W ranks, so this is the per-token mean / W:
    #  the relative token weights are right; multiply by W if you want the scale too)

def build_labels(input_ids, assistant_mask):
    """assistant_mask: 1 on assistant content AND its end-of-turn token."""
    labels = input_ids.clone()
    labels[assistant_mask == 0] = -100
    return labels

§3Packing and attention masks

Padding wastes compute; packing wastes nothing but can let examples read each other. The fix is a mask, not a different data format.

Real SFT data has wildly varying lengths. Padding each example to the longest in its batch wastes the padded positions; Krell et al. (2021) measured that up to 50% of tokens in common NLP datasets can be padding. Packing concatenates several examples into one fixed-length row. It is old practice for SFT, too: Llama 2 "concatenate[d] all the prompts and answers from the training set" to fill its 4,096-token sequences. Done naively, with an ordinary causal mask over the whole row, every token can attend to every earlier token in the row, including tokens of unrelated examples packed before it. That is cross-contamination. The position ids also keep counting across examples, so later examples start at positions they will never start at during inference.

How much does contamination hurt? It depends on the regime. For pretraining, Llama 3 reported that a mask preventing "self-attention between different documents within the same sequence" had "limited impact during in standard pre-training" but was "important in continued pre-training on very long sequences"; Zhao et al. (2024) found intra-document causal masking "significantly improving performance" in their pretraining runs. For SFT, where examples are short, semantically unrelated and each one is a self-contained conversation, contamination directly teaches the model to condition on irrelevant context, and the fix is cheap. Modern stacks implement it without materializing a mask: FlashAttention's variable-length kernels take cumulative sequence lengths (cu_seqlens) or reset position_ids. Hugging Face's DataCollatorWithFlattening reported "up to 2x improvement in training throughput while maintaining convergence quality" (1.4× on a dataset with longer, more uniform examples). In current TRL, packing=True defaults to packing_strategy="bfd" (best-fit decreasing, which truncates overflow) and enables padding-free attention.

§4Forward vs reverse KL: mode-covering and mode-seeking

SFT and RL push the model toward a target distribution from opposite sides. Which side you come from decides whether the model hedges or commits.

Let $p$ be a target distribution and $q_\theta$ a model too simple to match it exactly. The two KL divergences are

Why SFT is forward KL was shown in §1. Why RL is reverse KL takes two lines and connects to chapter 04. The KL-regularized RL objective can be rewritten as

$$\E_{y\sim\pit}[r(x,y)] - \beta\,\KL(\pit\,\|\,\piref) \;=\; -\beta\,\KL\big(\pit \,\big\|\, \pi^*\big) + \beta\log Z(x), \qquad \pi^*(y\mid x) = \tfrac{1}{Z(x)}\,\piref(y\mid x)\,e^{r(x,y)/\beta}.$$

Maximizing the RL objective is exactly minimizing the reverse KL from the policy to the tilted target $\pi^*$. On-policy distillation (§6) minimizes reverse KL to the teacher directly. Expectations under the model's own samples are what "on-policy" means mathematically.

A derivation to remember. Fit a Gaussian $q = \mathcal{N}(\mu,\sigma^2)$ to any $p$ by forward KL. Since $-\log q(y) = \log\sigma + \frac{(y-\mu)^2}{2\sigma^2} + \text{const}$,

$$\frac{\partial}{\partial\mu}\KL(p\|q) = -\E_p\!\Big[\frac{y-\mu}{\sigma^2}\Big] = 0 \;\Rightarrow\; \mu = \E_p[y], \qquad \frac{\partial}{\partial\sigma}\KL(p\|q) = \E_p\!\Big[\frac{1}{\sigma} - \frac{(y-\mu)^2}{\sigma^3}\Big]=0 \;\Rightarrow\; \sigma^2 = \mathrm{Var}_p[y].$$

Forward KL is moment matching: for a bimodal $p$, the fit sits in the valley between the modes, with a variance wide enough to cover both. Reverse KL has no closed form here; it typically locks onto one mode, and which one depends on initialization. The widget optimizes both live with Adam.

§5The data engine: quality, diversity, synthesis, hygiene

In SFT the dataset is the specification. Changing the data is the main lever you have, and the field's history is a sequence of cheaper ways to get better demonstrations.

Quality versus quantity

LIMA's ablations are the canonical evidence. Training 7B models on 2,000 examples, quality-filtered Stack Exchange beat unfiltered Stack Exchange by about half a point on a 1–6 helpfulness scale (graded by ChatGPT); high-quality but homogeneous data (wikiHow, all "how to" prompts) did worse than diverse data; and an up-to-16-fold increase in quality-filtered data left scores flat ("doubling the training set does not improve response quality"). Their conclusion: alignment scaling is "a function of prompt diversity while maintaining high quality responses", not of quantity alone. Llama 2's team reached a similar operational conclusion ("SFT annotations in the order of tens of thousands was enough"), stopped at 27,540 annotations, and noticed that outputs sampled from their SFT model "were often competitive with SFT data handwritten by human annotators", so they moved annotation budget to preference data.

The nuance: LIMA measured chat helpfulness judged by humans. When the target is a skill, volume matters again. Tülu 3 found its models "continue to improve on average as more SFT data is included", with large gains on GSM8K, while TruthfulQA got worse as data grew. For reasoning, small curated sets go surprisingly far (s1: 1,000 questions selected for difficulty, diversity and quality; LIMO: about 800 samples on Qwen2.5-32B-Instruct, 63.3% on AIME), but the strongest open small reasoners come from large sets (OpenThoughts3: 1.2M examples).

Synthetic data: a short genealogy

Tülu 3 is the best public window into a modern mix, and its ablations are worth memorizing as patterns: removing WildChat (real user conversations) caused "a small but noticeable degradation on most skills", most visibly on AlpacaEval; removing the persona sets lowered HumanEval(+), MATH and above all IFEval (72.8 → 53.6; the text also lists GSM8K, but its Table 10 shows GSM8K slightly up, 76.2 → 76.8); safety data was "generally orthogonal" to everything else, and contrastive prompts (CoCoNot) helped prevent over-refusal. The general recipe: prompts are the scarce resource (diverse, real, targeted), responses are regenerated by the best available teacher and filtered.

For reasoning data, OpenThoughts ran over 1,000 controlled experiments and reported: sampling multiple answers per question "is an effective technique to increase the size of a data source by at least 16×"; "models with better performance are not necessarily better teachers. QwQ-32B is a stronger teacher than DeepSeek-R1"; and verification or answer filtering "gave no significant performance improvements". Treat these as results for one pipeline, but note the pattern: the teacher's style can matter more than its benchmark score.

Decontamination

Synthetic and scraped prompts leak benchmarks. Tülu 3's procedure is a good default: match prompts only (responses are regenerated, so they differ anyway) with 8-gram overlap; flag a test instance if more than 50% of its tokens are covered by 8-grams from a single training instance; call a training set contaminated if it overlaps more than 2% of any evaluation's instances. Contaminated sets were dropped entirely for held-out ("unseen") evaluations; for development evaluations they dropped the whole set when that did not hurt, and otherwise removed only the matching instances. They tried embedding matching and found it hard to separate "mere distributional similarity from actual paraphrasing". The limitation is real: Yang et al. (2023) showed that rephrased or translated test items slip past n-gram filters and that a 13B model trained on them can reach GPT-4-level scores on MMLU, GSM8K and HumanEval. If your synthetic pipeline uses a teacher that has seen a benchmark, assume paraphrased leakage and keep a held-out evaluation that postdates your data (see chapter 09).

Model collapse: what the evidence actually says

Shumailov et al. (arXiv 2023; Nature 2024, "AI models collapse when trained on recursively generated data") trained each generation of a model on data generated by the previous one and showed the tails of the distribution disappear: low-probability events are forgotten first, and the model drifts toward a narrow mode. Their language-model experiment fine-tuned OPT-125m on wikitext2, regenerated the corpus with beam search, and repeated. Read the setup carefully: when no original data was kept, quality degraded over generations; when 10% of the original data was kept each generation, they found "only minor degradation of performance."

Gerstgrasser et al. (2024) made the distinction sharp: replacing real data with each generation's synthetic data "does indeed tend towards model collapse", but "accumulating the successive generations of synthetic data alongside the original real data avoids model collapse", across model sizes and modalities. The practical reading for post-training: collapse is a risk of closed loops that discard real data and of pure self-generated training without an external filter. Distilling a strong teacher, filtering with a verifier, or mixing in real prompts are all open loops. The live risk in SFT pipelines is subtler than collapse: diversity loss, which you see as repetitive phrasing, homogenized personas and shrinking pass@k.

§6Distillation: SFT with a better target

A sampled token is one draw from the teacher's opinion. The teacher's full distribution is the opinion itself. Distillation is the art of learning from the opinion.

Logit distillation (Hinton et al., 2015)

Hinton, Vinyals & Dean train a student on the teacher's softened distribution. With temperature $\tau$, teacher logits $z^T$ and student logits $z^S$:

$$\mathcal{L} = (1-\alpha)\,\mathrm{CE}\big(y,\,\sigma(z^S)\big) \;+\; \alpha\,\tau^2\,\KL\big(\sigma(z^T/\tau)\,\big\|\,\sigma(z^S/\tau)\big).$$

The $\tau^2$ is not decoration: "the magnitudes of the gradients produced by the soft targets scale as $1/T^2$", so multiplying by $\tau^2$ keeps the two terms balanced as you change temperature. The reason soft targets help is what is often nicknamed dark knowledge: "the relative probabilities of incorrect answers tell us a lot about how the cumbersome model tends to generalize." Their best demonstration: a student trained on a transfer set with every digit 3 removed still got 98.6% of test 3s right after a bias correction, purely from how the teacher spread probability onto "3" when looking at other digits.

Sequence-level distillation (Kim & Rush, 2016)

For autoregressive models, matching per-token distributions along the data's prefixes is still teacher forcing. Kim & Rush instead approximated the sequence-level objective $\E_{y\sim p_T}[-\log p_S(y\mid x)]$ by training on the teacher's beam-search output. This is what most of the field now does by default when it "distills" a frontier model: generate responses, SFT on them. Their students ran 10× faster than the teacher with little quality loss and "seem to eliminate the need for beam search". Note what this objective is: still forward KL (now to the teacher's sequence distribution, approximated by its mode), still off-policy.

On-policy distillation: GKD and its descendants

Agarwal et al. (2023) unified the space with Generalized Knowledge Distillation:

MiniLLM (Gu et al.) argued for the reverse KL on mode-seeking grounds (§4). The 2025 revival is the simplest version: Thinking Machines' "On-Policy Distillation" (Lu et al., October 2025) uses the per-token reverse KL as a dense, per-token reward. For a student sample $y\sim\pit(\cdot\mid x)$, each position $t$ gets

$$r_t = -\big[\log\pit(y_t\mid x, y_{\lt t}) - \log\pi_{\text{teacher}}(y_t\mid x, y_{\lt t})\big], \qquad \E_{y_t\sim\pit(\cdot\mid x,y_{\lt t})}[-r_t] = \KL\big(\pit(\cdot\mid x,y_{\lt t})\,\big\|\,\pi_{\text{teacher}}(\cdot\mid x,y_{\lt t})\big),$$

a one-sample estimate of the (negative) reverse KL at that prefix, used as the reward with a discount of zero (each token's reward is its own log-ratio; no credit flows to earlier tokens), inside an otherwise standard policy-gradient loop. The student generates; the teacher computes log-probs on the student's tokens; every token gets a grade. It combines what each paradigm does well: on-policy like RL (trained on its own states, so no exposure bias) and dense like SFT (a signal on every token, not one scalar per episode).

Source: Qwen3 Technical Report, Table 21 (math and code queries only; the strong-to-weak pipeline's teachers are Qwen3-32B or Qwen3-235B-A22B). Note RL left pass@64 unchanged while distillation raised it: a teacher can push probability onto solutions the student almost never samples, while RL mostly sharpens what the student already samples.

Thinking Machines replicated the pattern with Qwen3-8B-Base: SFT on 400k OpenThoughts-3 prompts reached 60% on AIME'24 with log-linear returns (they extrapolate about 2M prompts for 70%), while on-policy distillation reached 70% in about 150 steps (~77K prompts, 4 samples each), a cost reduction they estimate at 9–30× depending on whether the teacher's sampling cost for the SFT data is counted. Their most striking side result: running SFT on a model's own samples (which have zero KL to the model in expectation) still degraded instruction following at every practical learning rate, because each finite batch nudges the model off its own distribution and the drift compounds. On-policy distillation against a fixed teacher does not drift.

Distilling reasoning traces, and why small models prefer distillation to RL

DeepSeek-R1 fine-tuned Qwen and Llama bases on about 800k samples curated from R1 (2–3 epochs, 32k context), and then ran the decisive control: large-scale RL (over 10K steps) directly on Qwen2.5-32B-Base. The RL model reached 47.0 on AIME 2024; the distilled model reached 72.6 (MATH-500: 91.6 vs 94.3; LiveCodeBench: 40.2 vs 57.2). Their conclusion: "distilling more powerful models into smaller ones yields excellent results, whereas smaller models relying on the large-scale RL … require enormous computational power and may not even achieve the performance of distillation."

Why? Three reasons, in increasing depth:

1. Bits per episode. As Thinking Machines put it in "LoRA Without Regret", supervised learning "provides O(number of tokens) bits per episode" while policy gradient gets "O(1) bits per episode" from the advantage. For a 10k-token trace, this (deliberately crude) accounting gives a gap of roughly four orders of magnitude.
2. Exploration. RL can only reinforce behaviors the policy already samples with non-trivial probability. A weak model rarely samples a full correct long solution, so the reward is mostly zero. A teacher hands it the solution.
3. The teacher solved the search problem. The expensive part of reasoning RL is discovering strategies like verification and backtracking. Distillation transfers the strategies discovered by someone else's RL. Hence DeepSeek's second conclusion: going beyond the frontier "may still require more powerful base models and larger-scale reinforcement learning."

Two caveats keep this honest. Li et al. (2025) found a "Small Model Learnability Gap": students of 3B parameters or fewer "do not consistently benefit from long chain-of-thought reasoning or distillation from larger models" and do better with shorter chains or smaller teachers (the capacity mismatch of §4, observed). And distillation is bounded by the teacher, inheriting its mistakes, its style and, per the next callout, possibly traits you did not intend to transfer.

§7Rejection sampling, expert iteration, STaR and ReST

Sample many answers, keep the good ones, fine-tune on them, repeat. The simplest RL algorithm in the guide is just SFT inside a loop.

The family goes by many names: expert iteration (Anthony et al., 2017, where a tree search is the "expert" and the network imitates it), STaR (Zelikman et al., 2022), RAFT (Dong et al., 2023), ReST (Gulcehre et al., 2023: a "Grow" step that samples from the policy and an "Improve" step of offline RL on filtered data), ReST^EM (Singh et al., 2023), and plain rejection sampling fine-tuning (Llama 2). The shared objective, at iteration $k$:

$$\theta_{k+1} = \argmax_\theta\; \E_{x}\,\E_{y\sim\pi_{\theta_k}(\cdot\mid x)}\Big[\mathbb{1}\big[r(x,y)=1\big]\,\log\pit(y\mid x)\Big].$$

Llama 2 describes the production version: "We sample K outputs from the model and select the best candidate with our reward … the sample obtaining the highest reward score is considered the new gold standard", fine-tuning on those. They contrast it with PPO along two axes: breadth (K samples per prompt versus one) and depth (PPO updates the sampler every step; rejection sampling samples everything from the initial policy of the round). Through RLHF-V4 they used only rejection sampling; afterwards they ran PPO on top of the rejection-sampled checkpoint. They also used it as a distillation channel: "We perform rejection sampling only with our largest 70B Llama 2-Chat. All smaller models are fine-tuned on rejection sampled data from the larger model."

STaR's contribution is rationalization: for problems the model fails, show it the correct answer and ask it to produce a rationale that reaches it, then train on that rationale as if it were generated without the hint. This attacks RFT's blind spot, prompts with zero successful samples, which otherwise contribute nothing forever. With it, STaR reported CommonsenseQA performance comparable to fine-tuning a 30× larger model. (Notice that rationalization is a form of privileged-information distillation; it will reappear in §8.)

§8Context distillation: turning a prompt into weights

If a long prompt makes the model behave well, train the model to behave that way without the prompt. Anthropic introduced this in 2021, and versions of it run through modern safety training.

In "A General Language Assistant as a Laboratory for Alignment" (Askell et al., 2021), Anthropic wrote a 4,600-word "HHH prompt" of 14 fictional conversations and found that simply prompting made models more helpful, honest and harmless, with little capability "tax" at large scale. Prompts cost context and compute at every call, so they proposed context distillation:

Note the direction: forward KL from the prompted model to the unprompted one, so this is logit distillation (§6) where the "bigger teacher" is the same model plus a prompt. They explain why not simply fine-tune on the prompt text: "finetuning changes the model's expectations for the data distribution $P(X)$, bringing it closer to the distribution of the prompt $P(C)$, whereas prompting instead asks the model for the distribution $P(X\mid C)$." Fine-tune on the numbers 1 to 63 and the model does not expect 64 next; prompt with them and it does. Their verdict: "For many but not all of our evaluations context distillation performs about as well as prompting."

The follow-up paper made it load-bearing. In Anthropic's HH-RLHF work (Bai et al., 2022), the recipe prepends the HHH prompt to text that is 50% pretraining data and 50% StackExchange formatted as a dialogue, runs a 52B model, stores the top-50 log-probabilities per token (plus one bucket for the rest), and distills into models of all sizes with a KL loss over 350M tokens. Then: "We initialize our policies on context-distilled models." In that paper, the RLHF policies started from a context-distilled model rather than from a conventional demonstration-SFT model.

The lineage

• Constitutional AI's supervised stage (Bai et al., 2022): generate responses with a helpful-only model, ask it to critique and revise them according to a constitutional principle, then fine-tune on the revisions. The SL stage "gets the model 'on-distribution'" and "gives some control over the initial behavior at the start of the RL phase, addressing potential exploration problems". It is context distillation where the context is a critique-and-revise procedure. (RLAIF, the second stage, is in chapter 04.)
• Deliberative alignment (OpenAI, Guan et al., 2024): "We construct this dataset using context distillation … present the model with the safety specifications as part of the system prompt, generate model completions, and then strip away the system prompts to form the final dataset." Details in chapter 08.
• Learning by distilling context (Snell et al., 2022) generalized the idea to internalizing instructions, explanations and scratchpads.
• On-policy self-distillation (2026): the same model is teacher (with privileged context: a verified solution, a system prompt) and student (without), trained with on-policy distillation on the student's own rollouts (Zhao et al., 2026). The early evidence draws a clean line: Zhu et al. (2026) find it fails in their tested settings when the privileged information is instance-specific (a solution to this problem, unavailable at test time), but is effective "when PI represents a shared latent rule, such as a system prompt or alignment preference". That is exactly the 2021 use case. Kaur et al. (2026) report that privileged-context distillation degrades thinking models by up to 17% (relative) on competition math.

Anthropic's alignment team also uses SFT as a precise instrument rather than only as a pipeline stage: synthetic document finetuning (Wang et al., April 2025) generates documents that reference a proposition and fine-tunes on them like extra pretraining data, implanting beliefs for honeypots and model-organism studies. To our knowledge, Anthropic has not published its current production SFT recipe; when this guide speculates about it, it says so.

§9Generalization, forgetting, and how much to move the model

SFT can teach a format in a thousand examples and change a model's values in six thousand. Knowing how far a dataset will move the model, and in which directions, is the real skill.

"SFT memorizes, RL generalizes", and the counterpoints

Chu et al. (2025) compared SFT and RL on GeneralPoints (an arithmetic card game, with rule variants) and V-IRL (navigation), in text and vision. RL generalized to rule and visual variants; SFT tended to memorize the training rules. Two details are usually dropped when the slogan is repeated: "SFT is still helpful for effective RL training: SFT stabilizes the model's output format, enabling subsequent RL to achieve its performance gains", and the comparison used fixed prompt templates. The counter-evidence attacks exactly that:

• Lin et al. (2025), "Debunk the Myth of SFT Generalization": much of SFT's failure is a frozen-prompt artifact; with prompt diversity plus chain-of-thought demonstrations, SFT matched or beat RL on Sokoban and GeneralPoints.
• Ren et al. (2026): for long-CoT reasoning SFT, cross-domain generalization is "not absent but conditional": performance can dip then recover with longer training (so short runs understate it), verified traces help while low-quality ones hurt, and weaker models imitate surface verbosity instead of procedure. They also report an asymmetry: reasoning improves while safety degrades.
• Kirk et al. (2023) agree RLHF generalizes better out of distribution than SFT, but at a cost in output diversity (§4).

The defensible synthesis: SFT generalizes about as well as its data covers the variation you test on, and it moves the model in whatever direction the data's surprising tokens point (the $1/\pit$ weight of §1). RL moves the model less and only along directions it can already sample, which is both why it generalizes conservatively and why it cannot teach what the model never does.

Forgetting: RL's razor

Shenfeld, Pari & Agrawal (2025) found an empirical "forgetting law": how much a fine-tuned model forgets is predicted by $\E_{x\sim\tau}[\KL(\pi_0\,\|\,\pi)]$, the KL between the base and fine-tuned policy measured on the new task's inputs. Among the many solutions to a new task, on-policy RL is biased toward the one closest to the base model; SFT goes wherever its labels point, which may be far. The widget makes the argument concrete with one prompt and ten possible answers.

The same logic explains three practical observations. Mixing replay data (general chat) into a narrow SFT run reduces forgetting but rarely eliminates it (Thinking Machines saw instruction-following fall from 85% to 79% even with 30% chat data in a knowledge mid-training run, and recovered to 83% with on-policy distillation from the earlier checkpoint). Chen et al. (2025), "Retaining by Doing", attribute RL's lower forgetting to on-policy data. And SFT on your own model's samples (rejection sampling) forgets less than SFT on human or foreign-teacher data, because the targets are already likely under the model.

Narrow data, broad effects

The most important generalization result for alignment researchers runs the other way: SFT can generalize too much. Betley et al. (2025) fine-tuned GPT-4o and Qwen2.5-Coder-32B-Instruct on 6,000 examples of code with undisclosed security vulnerabilities. The models became broadly misaligned on unrelated questions (emergent misalignment). The control is the lesson: with identical assistant responses but user messages that request insecure code for a legitimate educational reason, the misalignment did not appear. "The intention behind the code also matters." The model is not learning "write this code"; it is learning what kind of assistant would write this code in this context, a persona. Inoculation prompting (§8) works by changing that inferred context.

LoRA versus full fine-tuning

Biderman et al. (2024), "LoRA Learns Less and Forgets Less", compared both on code and math, in instruction tuning (≈100K pairs) and continued pretraining (≈20B tokens): "in the standard low-rank settings, LoRA substantially underperforms full finetuning", but "better maintains the base model's performance on tasks outside the target domain", and full fine-tuning learns weight perturbations "with a rank that is 10-100× greater than typical LoRA configurations". Thinking Machines' "LoRA Without Regret" (Schulman et al., September 2025) characterized when the gap disappears:

• "For supervised fine-tuning on small-to-medium-sized instruction-tuning and reasoning datasets, LoRA performs the same as full fine-tuning"; for datasets that exceed LoRA's capacity it underperforms, as worse training efficiency rather than a hard floor.
• Apply it to all weight matrices, especially MLP and MoE layers; attention-only LoRA underperforms even at matched parameter count.
• LoRA is "less tolerant of large batch sizes than full fine-tuning", and raising rank does not fix it.
• The optimal LoRA learning rate is about 10× the full fine-tuning optimum, roughly independent of rank.
• For RL, even very low ranks match full fine-tuning: RL simply transmits few bits (§6).

Note the tension with Biderman et al.: both are compatible once you account for configuration (all-layer LoRA, tuned LR) and dataset size. Forgetting is the other axis: in their on-policy distillation post, Thinking Machines found LoRA "still insufficient for preserving" instruction following in their knowledge mid-training run. LoRA constrains the rank of the update, not its KL from the base.

Hyperparameters that matter, with receipts

§10Practical failure modes: a debugging checklist

When an SFT model misbehaves, the cause is usually in the first four rows of this table, not in the data or the learning rate.

§11Frontier & open problems

The boundary between SFT and RL is dissolving. What remains contested is mostly about data: where it comes from, what it secretly carries, and how far it moves the model.

§12Check yourself

Try each one before opening the answer. The "research" questions have no single right answer; the answers show one good line of reasoning.

§13Read next