RL for language models: from REINFORCE to PPO

§1Why RL, and what this chapter is for

Supervised fine-tuning teaches a model to imitate. Reinforcement learning teaches it to improve against a score, including on outputs no human ever wrote.

After SFT (Chapter 01), the model can produce good responses, but it learned only from demonstrations. Three things push us towards RL. First, it is often easier to judge a response than to write the ideal one: a preference label, a unit test, or a checked math answer gives a signal on outputs nobody demonstrated. Second, imitation learns from the expert's state distribution, but at test time the model conditions on its own previous tokens. RL trains on the model's own samples, so it learns to recover from its own mistakes. Third, a scalar reward can express things a likelihood cannot: "shorter is better if still correct", "refuse only when necessary", "the proof must check".

The objective we will optimize for the rest of the guide is

This chapter is the machinery: how to get a gradient of that objective from samples, how to make it low-variance, how to take several safe steps per batch, and how all this became the PPO recipe of InstructGPT, plus why much of the field later dropped half of that recipe. Chapter 04 owns the KL-regularized objective and its optimum, Chapter 06 owns GRPO and its descendants, and Chapter 07 owns large-scale asynchronous systems. Here we build the foundations they all stand on.

§2Generation as a Markov decision process

A state is everything written so far. An action is the next token. The environment does nothing except append it.

An MDP is a tuple $(\mathcal S,\mathcal A,P,r,\rho_0,\gamma)$. For a decoder-only language model answering a prompt $x$:

• State $s_t = (x, y_1,\dots,y_{t-1})$: the prompt plus the tokens generated so far. The initial state distribution $\rho_0$ is the prompt distribution.
• Action $a_t = y_t \in \mathcal V$: one token from a vocabulary of roughly $10^5$ entries.
• Policy $\pit(a_t\mid s_t) = \mathrm{softmax}(z_\theta(s_t)/\tau)_{a_t}$: the language model's next-token distribution at temperature τ.
• Transition $P(s_{t+1}\mid s_t,a_t) = \mathbb 1[s_{t+1} = s_t\oplus a_t]$: deterministic. The new state is the old one with the token appended. The episode ends at the end-of-sequence token or at a length cap.
• Reward: typically $r_t = 0$ for $t\lt T$ and $r_T = R(x,y)$: sparse and terminal. In PPO-style RLHF a small per-token KL penalty is added to every step (§8).
• Discount γ: episodes are finite, so LLM practice almost always uses $\gamma = 1$.

Two ways to slice the same problem

The sequence-level (bandit) view treats the whole response $y$ as a single action from the context $x$, with probability $\pit(y\mid x) = \prod_t \pit(y_t\mid s_t)$ and reward $R(x,y)$. There are no intermediate states to reason about. The gradient (next section) is $\E[R\,\nabla\log\pit(y\mid x)]$, and since $\log\pit(y\mid x) = \sum_t\log\pit(y_t\mid s_t)$, every token gets the same scalar weight. REINFORCE, RLOO, ReMax and GRPO all live here. The InstructGPT paper also describes its environment as a bandit: a prompt, a response, a reward, end of episode.

The token-level view keeps the full MDP. Each token has its own state and gets its own advantage $\hat A_t$, typically from a learned value function and GAE (§5). PPO as implemented for RLHF lives here: the KL penalty is paid per token, and a critic tries to say which tokens moved the expected return.

§3The policy gradient, derived

We want the gradient of an average over samples whose distribution itself depends on θ. The trick: differentiate the probabilities, not the samples.

Write $J(\theta) = \E_{\tau\sim p_\theta}[R(\tau)]$, where τ is a trajectory (for us: a prompt and a response). The problem: the reward model or verifier is not differentiable with respect to the tokens (they are discrete samples), and the expectation is over a θ-dependent distribution. Scroll through the derivation on a one-dimensional toy. The "policy" is a Gaussian $\pit = \mathcal N(\theta,1)$ over a continuous action $x$, and the reward $f(x)$ has a bump to the right.

From one action to a trajectory

For a trajectory $\tau=(s_0,a_0,\dots,s_T,a_T)$ the probability factorizes as $p_\theta(\tau) = \rho_0(s_0)\prod_t \pit(a_t\mid s_t)\,P(s_{t+1}\mid s_t,a_t)$. Take the log and differentiate. The initial-state and transition terms do not depend on θ and drop out:

$$\nabla_\theta\log p_\theta(\tau) = \sum_{t=0}^{T}\nabla_\theta\log\pit(a_t\mid s_t).$$

This is why model-free policy gradients never need a model of the environment. For LLMs that point is moot, since we know the dynamics exactly ("append the token"). So

$$\nabla_\theta J = \E_{\tau\sim p_\theta}\Big[R(\tau)\sum_{t}\nabla_\theta\log\pit(a_t\mid s_t)\Big].$$

This is REINFORCE (Williams, 1992): sample, score, and scale each sampled token's log-likelihood gradient by the return. It is the gradient of a weighted log-likelihood, a "supervised loss whose labels are the model's own samples, weighted by how good they were". That reading is the most useful intuition in the chapter.

# REINFORCE for an LLM, sequence-level (one scalar per response)
# logp: [B, T] log-probs of the *sampled* tokens under the current policy
# mask: [B, T] 1 for response tokens, 0 for prompt/padding;  R: [B] rewards
def reinforce_loss(logp, mask, R, baseline=0.0):
    adv = (R - baseline).detach()                       # no gradient through the weight
    seq_logp = (logp * mask).sum(-1)                    # log pi(y|x) = sum_t log pi(y_t|s_t)
    return -(adv * seq_logp).mean()                     # minimize the negative objective

§4Variance, baselines and advantages

The policy-gradient estimator is unbiased and terribly noisy. Almost every algorithm after REINFORCE is a variance-reduction trick.

Where does the noise come from? Look at a single-sample estimate $\hat g = R\cdot\sum_t\nabla\log\pit(a_t\mid s_t)$:

• The score sum grows with length. It is a sum of $T$ zero-mean random vectors (martingale differences), so its variance grows roughly linearly in $T$. A 2,000-token response contributes 2,000 noisy terms for one scalar of signal.
• Reward offset multiplies the noise. Write $R = c + R'$. Since $\E[\nabla\log\pi]=0$, the constant $c$ adds nothing to the mean but adds about $c^2\,\E\|\nabla\log p_\theta(\tau)\|^2$ to the variance. A reward model whose scores sit around +4 is, to REINFORCE, mostly noise.
• One sample says little about the counterfactual. We see the reward of what was sampled, not of what wasn't. Learning "this was better than usual" needs a notion of "usual".

Baselines: free variance reduction

For any function $b(s)$ that does not depend on the action:

$$\E_{a\sim\pi(\cdot\mid s)}\big[b(s)\,\nabla_\theta\log\pit(a\mid s)\big] = b(s)\sum_a\pit(a\mid s)\frac{\nabla_\theta\pit(a\mid s)}{\pit(a\mid s)} = b(s)\,\nabla_\theta\!\sum_a\pit(a\mid s) = b(s)\,\nabla_\theta 1 = 0.$$

So $\E[(G_t - b(s_t))\nabla\log\pi(a_t\mid s_t)] = \E[G_t\nabla\log\pi(a_t\mid s_t)]$: subtracting a baseline never changes the expected gradient, only its variance. The ideal per-state choice, up to a gradient-magnitude weighting, is $b(s)=V^\pi(s)$. Then the weight becomes the advantage $A^\pi(s,a) = Q^\pi(s,a) - V^\pi(s)$: how much better this action was than the policy's average action from the same state. (The exact variance-minimizing scalar baseline is $b^* = \E[\|\nabla\log\pi\|^2 R]/\E[\|\nabla\log\pi\|^2]$, a reward average weighted by squared score norm; Greensmith et al., 2004, analyze such choices. In practice nobody bothers with the weighting.)

Two things the widget makes visible. First, a constant shift of the reward is invisible to the expected gradient but very visible to the variance, which is why every serious LLM RL recipe either learns a baseline (critic), computes one from other samples (RLOO, GRPO), or at least whitens the rewards. Second, the variance does not only slow learning: with a softmax policy, a few unlucky large steps can push probability onto a mediocre arm, after which that arm is sampled almost exclusively and the better arm is rarely tried again. That is a small version of the entropy collapse problem of Chapter 06.

Baselines you can compute without a critic

Sample $k$ responses $y_1,\dots,y_k$ for the same prompt. Then:

• Leave-one-out (RLOO): $\hat A_i = R_i - \frac{1}{k-1}\sum_{j\ne i}R_j$. The baseline for sample $i$ uses only the other samples, which are independent of $y_i$, so the estimator stays exactly unbiased (Kool et al., 2019; Ahmadian et al., 2024).
• Group mean (GRPO-style): $\hat A_i = R_i - \bar R$. Including $R_i$ in the mean makes the baseline depend on the action. But algebraically $R_i - \bar R = \frac{k-1}{k}\big(R_i - \frac{1}{k-1}\sum_{j\ne i}R_j\big)$, so it is just RLOO scaled by a constant: unbiased in direction. GRPO's extra division by the group's standard deviation is a different matter. It reweights prompts by how consistent their rewards are, which does change the expected gradient (Liu et al., 2025; Ch. 06).
• Greedy baseline (ReMax): $b = R(x,\hat y_{\text{greedy}})$. One extra decode per prompt, independent of the sample, so it is unbiased (Li et al., 2023).

§5Critics, TD learning and GAE

A critic is a learned baseline that can also say where in the sequence things went right or wrong. GAE is the dial that decides how much you trust it.

A value function $V^\pi(s) = \E_\pi[\sum_{k\ge0}\gamma^k r_{t+k}\mid s_t=s]$ is the expected return from state $s$ under the current policy. For an LLM with a correctness reward it is, roughly, "the probability that this prefix ends in a correct answer" (minus the KL still to be paid). It satisfies the Bellman equation $V^\pi(s_t) = \E_{a_t}[r_t + \gamma V^\pi(s_{t+1})]$. In RLHF the critic $V_\psi$ is usually a copy of a transformer with a scalar head, trained by regression.

There are two ways to build a regression target for $V$, and two matching ways to estimate advantages:

• Monte Carlo (MC): target $G_t$ (the actual return). Unbiased, high variance. Advantage $\hat A_t = G_t - V(s_t)$.
• Temporal difference (TD): target $r_t + \gamma V(s_{t+1})$ (one real reward, then trust the critic). Low variance, biased whenever $V$ is wrong. Advantage $\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$, the TD error.

Generalized advantage estimation, derived

Between the two extremes sit the $k$-step estimators. Using $k$ real rewards and then bootstrapping, the sum telescopes:

$$\hat A^{(k)}_t = \sum_{l=0}^{k-1}\gamma^l\delta_{t+l} = -V(s_t) + r_t + \gamma r_{t+1} + \dots + \gamma^{k-1}r_{t+k-1} + \gamma^k V(s_{t+k}).$$

(Check: in $\sum_l\gamma^l(r_{t+l} + \gamma V(s_{t+l+1}) - V(s_{t+l}))$ each $+\gamma^{l+1}V(s_{t+l+1})$ cancels the next term's $-\gamma^{l+1}V(s_{t+l+1})$.) Larger $k$ means less reliance on $V$ (less bias) and more random rewards in the sum (more variance). GAE (Schulman et al., 2015b) takes an exponentially weighted average over all $k$:

$$\hat A^{\mathrm{GAE}(\gamma,\lambda)}_t = (1-\lambda)\big(\hat A^{(1)}_t + \lambda\hat A^{(2)}_t + \lambda^2\hat A^{(3)}_t + \dots\big).$$

Substitute $\hat A^{(k)}_t = \sum_{l\lt k}\gamma^l\delta_{t+l}$ and collect the coefficient of each $\delta_{t+l}$. The term $\gamma^l\delta_{t+l}$ appears in every $\hat A^{(k)}$ with $k\gt l$, weighted by $(1-\lambda)\lambda^{k-1}$, and $(1-\lambda)\sum_{k\gt l}\lambda^{k-1} = \lambda^l$. Hence

The critic is trained towards the λ-return $G^\lambda_t = \hat A_t + V_{\text{old}}(s_t)$, the matching blend of MC and TD targets. Two limits are worth memorizing. With λ = 1: $\hat A_t = \sum_l\gamma^l r_{t+l} - V(s_t)$, which is unbiased for any $V$ (the critic is then only a baseline). With λ = 0: $\hat A_t = \delta_t$, which is exactly right if $V=V^\pi$ (by the key idea in §2) and wrong by the critic's error otherwise.

γ and λ in LLM practice

Classic control used γ ≈ 0.99, λ ≈ 0.95. For LLMs the episode is finite and we care about the undiscounted sequence reward, so γ = 1 is standard (the TL;DR reproduction of Huang et al., 2024, uses γ = 1 and λ = 0.95). λ is the subtle one. With a purely terminal reward, the reward's direct contribution to token $t$'s advantage is weighted by $(\gamma\lambda)^{T-t}$. For λ = 0.95 and a 1,000-token chain of thought that is $0.95^{1000}\approx 5\times10^{-23}$. Early tokens then learn only through the critic, and a critic initialized from a reward model is badly biased early in training. This is precisely the failure that VC-PPO (Yuan et al., 2025) diagnosed for long chain-of-thought. Their diagnosis was "value initialization bias and reward signal decay". Their fix had two parts. First, value pretraining: freeze the policy and fit the critic with λ = 1 (pure Monte-Carlo returns), so the RM-initialized critic stops being biased. Second, decoupled GAE: λ_critic = 1.0, so the reward reaches every token in the value targets without decay, while λ_actor = 0.95 keeps the policy's advantages low-variance by leaning on the now-trustworthy critic. In their sweep, λ_actor of 0.95 and 0.99 beat both 0.9 and 1.0. On Qwen2.5-32B they report 48.8 on AIME 2024, against 38.9 for GRPO. VAPO (Yue et al., 2025) went further with a length-adaptive policy λ, $\lambda = 1 - \tfrac{1}{\alpha\,\ell}$ for response length ℓ (α = 0.05), and reports 60.4 on AIME 2024 with the same base model. That number comes from a whole package (asymmetric clipping, token-level loss, an extra LM loss on correct samples, and more), so do not attribute it to λ alone. (VC-PPO's numbers are from 16k-token generations; its λ_actor ablation reads 34.3 / 41.3 / 41.9 / 29.4 for λ = 0.9 / 0.95 / 0.99 / 1.0.)

def gae(rewards, values, gamma=1.0, lam=0.95):
    """rewards: [T] per-token rewards; values: [T+1] critic values, values[T] = 0 at EOS.
    Returns advantages [T] and value targets (lambda-returns) [T]."""
    T = len(rewards); adv = torch.zeros(T); last = 0.0
    for t in reversed(range(T)):
        delta = rewards[t] + gamma * values[t + 1] - values[t]
        last = delta + gamma * lam * last
        adv[t] = last
    return adv, adv + values[:T]          # returns = lambda-return targets for the critic

§6Off-policy correction and trust regions

Sampling is the expensive part. To take more than one gradient step per batch, we must correct for the fact that the batch came from an older policy, and stop the new policy from moving so far that the correction breaks.

Importance sampling

For any distributions with matching support, $\E_{a\sim\pi}[f(a)] = \E_{a\sim\mu}\big[\tfrac{\pi(a)}{\mu(a)}f(a)\big]$. Apply this to the policy gradient with samples from the behavior policy $\piold$ that generated the batch:

$$L_{\piold}(\theta) = \E_{s,a\sim\piold}\Big[\underbrace{\frac{\pit(a\mid s)}{\piold(a\mid s)}}_{r_t(\theta)}\hat A_t\Big],\qquad \nabla_\theta L_{\piold}\big|_{\theta=\theta_{\text{old}}} = \E_{\piold}\big[\nabla_\theta\log\pit(a\mid s)\,\hat A_t\big]\big|_{\theta=\theta_{\text{old}}}.$$

At $\theta=\theta_{\text{old}}$ the ratio is 1 and $\nabla r = r\,\nabla\log\pi = \nabla\log\pi$, so this surrogate has exactly the policy gradient. Away from $\theta_{\text{old}}$ it is an approximation in two ways. (i) The per-token ratio corrects the action distribution but not the state distribution: the prefixes were generated by $\piold$, and the exact correction would multiply ratios over all previous tokens, a product whose variance explodes with length. (ii) The advantages $\hat A_t$ were estimated for $\piold$, not for $\pit$. Both errors are small only if $\pit$ stays close to $\piold$.

Why "close" can be made precise

The performance difference lemma (Kakade & Langford, 2002) is exact:

$$J(\pi') - J(\pi) = \E_{\tau\sim\pi'}\Big[\sum_t\gamma^t A^{\pi}(s_t,a_t)\Big].$$

The surrogate $L_\pi(\pi')$ is the same expression with states drawn from $\pi$ instead of $\pi'$. TRPO's theorem (Schulman et al., 2015a) bounds the gap: $J(\pi')\ge L_\pi(\pi') - C\cdot\max_s\KL(\pi(\cdot\mid s)\,\|\,\pi'(\cdot\mid s))$, with $C = 4\epsilon\gamma/(1-\gamma)^2$ and $\epsilon=\max|A^\pi|$. Maximizing the right-hand side guarantees monotonic improvement. Note the $(1-\gamma)^{-2}$: with γ = 1, as in LLMs, the theoretical constant is vacuous. Nobody runs the theory-backed version, and the practical algorithms keep only the idea: improve the surrogate while keeping the KL small.

§7PPO: the clipped surrogate

Replace TRPO's constraint with a pessimistic objective that simply stops rewarding you for moving the ratio further in the direction you wanted.

Work through the cases. The gradient of the unclipped term is $\hat A_t\,\nabla r_t = \hat A_t\,r_t\,\nabla\log\pit(a_t\mid s_t)$.

• $\hat A\gt0$ (good action). We want $r$ up. For $r\le1+\epsilon$ the min picks $r\hat A$ and the gradient flows. For $r\gt1+\epsilon$ the min picks $(1+\epsilon)\hat A$, a constant: zero gradient. The incentive to increase this token's probability further is removed. If a good action's ratio has fallen below $1-\epsilon$, the min still picks $r\hat A$ and the gradient flows, pulling it back up.
• $\hat A\lt0$ (bad action). We want $r$ down. For $r\ge1-\epsilon$ the min picks $r\hat A$ (gradient flows). For $r\lt1-\epsilon$ the min picks $(1-\epsilon)\hat A$: zero gradient; it has been pushed down enough. But if the ratio of a bad action has risen above $1+\epsilon$, then $r\hat A\lt(1+\epsilon)\hat A$, the min picks the unclipped term, and the gradient flows with no bound. That is the pessimism: PPO always lets you undo a mistake.

So the clip is one-sided per sign: it caps how much you can gain from moving in the advantage's direction and never caps the push back. Clipped tokens drop out of the gradient entirely. The fraction of such tokens, clipfrac, is one of the most useful PPO diagnostics.

The rest of the PPO loss

The full objective of Schulman et al. (2017) is $L = \E_t[L^{\mathrm{CLIP}}_t - c_1 L^{V}_t + c_2\,\mathcal H[\pit](s_t)]$, maximized jointly when policy and value share parameters:

• Value loss $L^V_t = (V_\psi(s_t) - G^\lambda_t)^2$. Many implementations add value clipping, $\max\big((V_\psi-G^\lambda)^2,\;(\clip(V_\psi, V_{\text{old}}-\epsilon_v, V_{\text{old}}+\epsilon_v)-G^\lambda)^2\big)$, mirroring the policy clip. It comes from the reference implementations, not the paper. The large-scale study of Andrychowicz et al. (2020), on continuous-control tasks, found that PPO-style value clipping "hurts the performance regardless of the clipping threshold", in line with Engstrom et al. (2020). Their recommendation: GAE with λ = 0.9, and neither value clipping nor a Huber loss. For LLMs the critic is usually a separate network, so $c_1$ becomes simply the critic's own learning-rate scale.
• Entropy bonus $c_2\mathcal H$: encourages exploration in control tasks. In RLHF it is usually set to 0, because the KL-to-reference term already keeps the policy from collapsing, and a 150k-way entropy bonus mostly rewards garbage tokens. In RL for reasoning, entropy is managed in other ways (asymmetric clipping, covariance-based clipping; Ch. 06).
• Several epochs: the batch of rollouts is reused for $K$ epochs (typically 1–4 for LLMs) of minibatch updates. This is PPO's main efficiency win over vanilla policy gradient. With $K=1$ and one minibatch, the ratio is exactly 1 and PPO reduces to REINFORCE with the given advantages.
• Diagnostics: approximate KL between $\piold$ and $\pit$ on the batch (with the k3 estimator of §9), clipfrac, explained variance of the critic, entropy, and in RLHF the KL to the reference and the mean response length.

def ppo_loss(logp_new, logp_old, adv, v_new, v_old, returns, mask, eps=0.2, eps_v=0.2, c_v=0.5):
    """All tensors [B, T]; mask selects response tokens. adv already whitened, everything *_old detached."""
    ratio = torch.exp(logp_new - logp_old)
    pg1 = -adv * ratio
    pg2 = -adv * torch.clamp(ratio, 1 - eps, 1 + eps)
    pg_loss = masked_mean(torch.max(pg1, pg2), mask)            # max of negatives == min of objectives
    v_clip = v_old + torch.clamp(v_new - v_old, -eps_v, eps_v)  # optional value clipping
    v_loss = 0.5 * masked_mean(torch.max((v_new - returns) ** 2, (v_clip - returns) ** 2), mask)
    clipfrac = masked_mean((torch.abs(ratio - 1) > eps).float(), mask)
    approx_kl = masked_mean((ratio - 1) - (logp_new - logp_old), mask)   # k3 estimate of KL(old || new)
    return pg_loss + c_v * v_loss, dict(clipfrac=clipfrac, approx_kl=approx_kl)

§8PPO for language models: the RLHF recipe

InstructGPT made PPO the default post-training algorithm. What made it work was a dozen details that are easy to get wrong and rarely in the equations.

The lineage runs from Ziegler et al. (2019, stylistic continuation and summarization) through Stiennon et al. (2020, TL;DR summarization) to InstructGPT (Ouyang et al., 2022). Anthropic's HH-RLHF paper (Bai et al., 2022a) used the same family of methods. The recipe:

1. Start from the SFT model, which is copied four times: the policy $\pit$ (trainable), the reference $\piref$ (frozen), the reward model $R_\phi$ (a separately trained, frozen network, Ch. 03) and the value model $V_\psi$ (trainable, typically initialized from the reward model, so it starts out already reading text the way the RM does). InstructGPT trained 1.3B, 6B and 175B policies, but used "a 6B RM and a 6B value function" for all of them, with the value function initialized from the RM.
2. Rollout: sample responses for a batch of prompts from $\piold$.
3. Score: one RM forward pass per response gives $R_\phi(x,y)$.
4. KL-shaped per-token reward: $$r_t = -\beta\,\big(\log\piold(y_t\mid s_t) - \log\piref(y_t\mid s_t)\big) + \mathbb 1[t=T]\,R_\phi(x,y).$$ The KL appears inside the reward, as a per-token cost paid with the k1 estimator on the sampled token. Summed over the sequence, it is a single-sample estimate of the sequence-level $\KL(\pi\,\|\,\piref)$. InstructGPT's paper calls this "a per-token KL penalty from the SFT model at each token to mitigate over-optimization of the reward model", with β = 0.02.
5. Values and GAE: run the critic on every prefix, compute $\hat A_t$ and the λ-return targets (§5).
6. Optimize: $K$ epochs of minibatch PPO on the policy and the critic.
7. Repeat, optionally adapting β to hit a target KL.

The details that matter

Two papers catalogue them. Huang et al. (2024), "The N+ Implementation Details of RLHF with PPO", reproduced OpenAI's TL;DR results from scratch. Zheng et al. (2023), "Secrets of RLHF in Large Language Models Part I: PPO", ablated the components on chat models and proposed a stabilized variant, PPO-max. The ones worth knowing by heart:

• Reward normalization. Fix the RM's offset so that a reference distribution scores zero on average. In the N+ reproduction (detail 15) the reward head's bias is set so that the SFT demonstrations have mean reward 0. This is the "subtract a constant" of §4, done once, globally. "Secrets of RLHF" goes further and normalizes and clips rewards with running statistics.
• Whitening. Advantage whitening across the batch (stage 7 of the explainer) is near-universal: it is a per-batch rescaling of the step size that keeps ε meaningful. Reward whitening is optional and not innocuous. In the N+ ablation it produced shorter outputs and lower preference rates, although at matched lengths the two variants performed similarly, so most of the effect runs through length.
• Value head initialized from the RM. "Secrets of RLHF" recommends going further and pre-training the critic before PPO starts, "replacing the learning rate warmup with the critic model pre-training". This is the same idea VC-PPO later applied to long chain-of-thought.
• Dropout off. With dropout, the "same" policy gives different log-probs on two forward passes, so the ratio is not 1 even before any update (N+ detail 7, following Ziegler et al.).
• Sampling settings. Rollouts must be samples from the distribution you differentiate. Top-p or top-k that is not mirrored in the training forward pass silently makes $\piold$ in the ratio wrong. The N+ reproduction samples at temperature 0.7. The same authors' earlier blog post on OpenAI's original lm-human-preferences code explains that the training forward pass divides the logits by the sampling temperature; in their informal test, without it "the KL would rise faster than expected, and performance would deteriorate".
• EOS handling. Responses truncated at the length cap never reach a proper terminal state. The N+ "EOS trick" gives any completion without an EOS token a constant reward of −1. A related pitfall: the reward must be read at the EOS position, not at padding.
• KL control. Ziegler et al. (2019) proposed an adaptive β, a proportional controller: $e_t = \clip\big((\mathrm{KL}_t - \mathrm{KL}_{\text{target}})/\mathrm{KL}_{\text{target}}, -0.2, 0.2\big)$, $\beta_{t+1} = \beta_t(1 + K_\beta e_t)$ with $K_\beta = 0.1$. Many later recipes fix β: 0.02 in InstructGPT, 0.05 in the N+ reproduction and in "Secrets of RLHF" (which calls the KL penalty "critical to the stability of PPO"), and 0.001 in Anthropic's HH-RLHF (below).
• Pretraining mix (PPO-ptx). InstructGPT added a language-modeling loss on pretraining data, with coefficient 27.8, to reduce the "alignment tax" on public NLP benchmarks.
• No entropy bonus. "Secrets of RLHF" found it too sensitive to tune and does not recommend it.
• Monitoring. "Secrets of RLHF" reports that the policy's perplexity, its KL to the reference and its response length are "more informative of the training stability" than reward or loss. A collapsed model "uniformly delivers longer responses and exhibits lower perplexity".

For calibration, the N+ reproduction's full PPO setting on Pythia 1B–6.9B is: γ = 1, λ = 0.95, β = 0.05, 4 PPO epochs per batch, policy and value clip 0.2, batch 512, AdamW at 3e-6. InstructGPT used batches of 512 split into 8 minibatches of 64, trained for a single inner epoch, rollouts at temperature 1, no discount in GAE, and 256k episodes. So even with one epoch, the ratio drifts away from 1 after the first minibatch step, and the clip can matter.

The memory bill

Four transformers is the part everybody remembers. With mixed-precision Adam, each trainable parameter costs about 16 bytes: bf16 weights and gradients (2 + 2) plus fp32 master weights and two Adam moments (4 + 4 + 4). This is the accounting of the ZeRO paper (Rajbhandari et al., 2019). Each frozen parameter costs 2 bytes. Before counting activations, the KV cache for generation, or a separate inference engine's copy of the weights, PPO with a same-size critic needs more than twice the memory of SFT.

§9Estimating KL from samples: k1, k2, k3

You cannot sum over every possible response, so the KL is always estimated from the tokens you sampled. How you estimate it, and whether you then differentiate the estimate, changes what you optimize.

Setting (following Schulman's note "Approximating KL Divergence", 2020): we sample $x\sim q$ and want $\KL(q\,\|\,p) = \E_q[\log\frac{q(x)}{p(x)}]$. Let $r = p(x)/q(x)$. In RLHF, $q=\pit$, $p=\piref$, and $x$ is a sampled token or sequence.

• k1 $=-\log r$. Unbiased by definition. Its variance is high, and it is negative whenever the sample is more likely under $p$ than under $q$, which happens on a large fraction of samples even though the KL is non-negative.
• k2 $=\tfrac12(\log r)^2$. Always ≥ 0 and low-variance, but biased. Its expectation is an f-divergence that agrees with KL to second order when $p\approx q$ (both are $\tfrac12$ the Fisher quadratic form), so the bias is small only when the distributions are close.
• k3 $=(r-1)-\log r$. Unbiased, because $\E_q[r-1] = \sum_x q\,\frac{p}{q} - 1 = 0$: it is k1 plus a zero-mean control variate. It is always ≥ 0, because $\log r\le r-1$ (concavity of log). It also has low variance when $p\approx q$. It is the Bregman divergence of $-\log$, the gap between $\log$ and its tangent line at $r = 1$.

KL in the reward vs KL in the loss

There are two places to put the KL, and they are not equivalent.

KL in the reward (InstructGPT-style PPO; also RLOO/REINFORCE recipes that subtract $\beta\sum_t\log\frac{\pi}{\piref}$ from the sequence reward). The KL sample is treated as part of the return, so it reaches the parameters through the score function, like any reward. Since $\nabla\KL(\pit\|\piref) = \E_{\pit}\big[\log\frac{\pit}{\piref}\,\nabla\log\pit\big]$ (the extra term $\E[\nabla\log\pi]$ vanishes), "k1 in the reward, with the reward-to-go" is an unbiased estimator of the reverse-KL gradient. It also accounts for how today's token changes the KL paid on future tokens.

KL in the loss (GRPO: $\beta\,\KL$ estimated with k3 per token, added to the loss and differentiated directly; DeepSeekMath used β = 0.04). Now the gradient is the pathwise derivative of the estimator at the sampled tokens. The true gradient of an expectation has two parts, $\nabla\E_{\pit}[\hat k] = \E[\nabla\hat k] + \E[\hat k\,\nabla\log\pit]$, and a loss term keeps only the first. Work it out on-policy for a single categorical distribution (one token), with $\rho = \piref/\pit$. Convention: "reverse KL" means $\KL(\pit\|\piref)$, the one in the RLHF objective, and "forward KL" means $\KL(\piref\|\pit)$. Some RL-theory papers use the opposite names, so check before comparing claims.

• $\nabla k_1 = \nabla(\log\pit - \log\piref) = \nabla\log\pit$, so $\E_{\pit}[\nabla k_1] = 0$. Differentiating k1 gives no signal in expectation, even though k1 is an unbiased estimate of the KL's value.
• $\nabla k_2 = \log\frac{\pit}{\piref}\,\nabla\log\pit$. Its expectation is exactly the reverse-KL gradient above, so for one token, k2 in the loss is the "right" one for $\KL(\pit\|\piref)$.
• $\nabla k_3 = \nabla(\rho - 1 - \log\rho) = -\rho\nabla\log\pit + \nabla\log\pit = (1-\rho)\nabla\log\pit$. Its expectation is $\E_{\pit}[\nabla\log\pit] - \sum_y\piref\nabla\log\pit = -\sum_y\piref(y)\nabla\log\pit(y) = \nabla\KL(\piref\|\pit)$. So k3 in the loss follows the gradient of the forward KL, $\KL(\piref\|\pit)$, not the reverse KL it estimates.

Near $\pit\approx\piref$ all of these agree to first order, which is why GRPO works fine in practice. But the forward KL is mass-covering: it penalizes the policy for dropping reference tokens much more than for adding new ones, a different regularizer than the one in the objective of Chapter 04. Two more subtleties. First, for sequences, a per-token loss term gives at best a partial derivative: it ignores how today's token changes the KL paid on future tokens. Tang & Munos (2025) make this point ("token-level losses only produce a partial derivative at best"). Shah et al. (2025) study the four on-policy configurations k1 or k3, in the reward or in the loss. Only k1 in the reward gives an unbiased gradient of the sequence-level reverse KL. k3 in the reward led to training collapse in their runs. k3 in the loss is "a sum of unbiased gradient estimates of the forward KL divergences computed at the token level", which is consistent with its stability in practice despite optimizing a different regularizer. (By the derivation above, per-token k2 in the loss recovers only the local reverse-KL terms, not the effect on future tokens.) Second, once samples are off-policy (several epochs, async rollouts), the expectation is under $\piold$, so the KL term needs the importance weight $\pit/\piold$ as well. Zhang et al. (2025) identify this missing weight in GRPO's k3 term, and DeepSeek-V3.2 (2025) adds exactly this correction, after which "the gradient of this KL estimator becomes unbiased". The weight matters even on-policy, where it equals 1 numerically, because it is differentiated. With $w=\pit/\piold$, $\nabla(w\,k_3) = w\,(k_3 + 1-\rho)\nabla\log\pit = w\,\log\tfrac{\pit}{\piref}\,\nabla\log\pit$, whose expectation under $\piold$ is the per-token reverse-KL gradient. Differentiating through $w$ supplies exactly the score-function term $\E[\hat k\,\nabla\log\pit]$ that a plain loss term drops.

# per-token log-ratios on sampled tokens: lr = logp_ref - logp_policy  (= log rho)
k1 = -lr                         # unbiased value, high variance, can be negative
k2 = 0.5 * lr ** 2               # biased value; its *gradient* matches reverse KL on-policy
k3 = torch.exp(lr) - 1 - lr      # unbiased value, low variance, >= 0; gradient matches *forward* KL

# KL in the reward (PPO-RLHF): no gradient through it, it is part of the return
rewards = -beta * (logp_old - logp_ref).detach(); rewards[last_token] += R
# KL in the loss (GRPO-style): differentiated directly
loss = pg_loss + beta * masked_mean(k3, mask)

§10Back to basics: REINFORCE, RLOO and the case against the critic

PPO was designed for robots learning from scratch. LLM fine-tuning starts from a model that is already good, and that changes which problems actually matter.

Ahmadian et al. (2024), "Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs", argued that most of PPO's machinery solves problems RLHF does not have. The argument, in their framing:

• The starting point is strong. Deep-RL algorithms were tuned for policies initialized at random, where the gradient estimates are very noisy and the policy can collapse. An SFT model already puts its probability on a sensible region, so the variance problem is milder and big, destructive updates are rarer.
• Model the whole generation as one action. With a terminal reward, the partial-sequence states add little. Treating the response as a single bandit action with a sequence-level reward works as well or better.
• The clip rarely fires. In their runs, "the loss is actually clipped on average <5% of the time per batch", and turning clipping off did not hurt. PPO's main safety device mostly did not matter.
• A sample-based baseline is enough. On TL;DR summarization and Anthropic-HH (Pythia-6.9B and Llama-7B), RLOO with k = 4 reached win-rates of 77.9 / 43.7 / 64.1 against PPO's 67.6 / 29.2 / 32.0, and it also beat DPO and RAFT. Even vanilla policy gradient beat PPO in their table. And RLOO needs no critic.

Then there are the costs of the critic itself: a second trainable network (the memory widget above), a second set of hyperparameters, and a bias that is worst exactly when the reward is sparse and the sequences are long, i.e. in reasoning RL. Groups of samples per prompt make the critic's job, "how good is this prompt for the current policy?", directly measurable: just average the group. That is the step from RLOO to GRPO (Shao et al., 2024): group-relative advantages (DeepSeekMath sampled 64 outputs per question), the clipped PPO ratio retained for multi-epoch reuse, and a k3 KL penalty in the loss. Its many refinements (Dr. GRPO, DAPO, GSPO, CISPO…) are the subject of Chapter 06.

§11Credit assignment and reward shaping

One number at the end of 5,000 tokens. Which tokens deserve it? Every answer is either a statistical average over many samples or a denser signal that might be wrong.

With a sequence-level advantage, every token in a good response is pushed up by the same amount, including the filler. Credit is assigned statistically: a token that appears in good responses more often than in bad ones gets pushed up on net, and an irrelevant token's pushes cancel out in expectation. This works, but the effective sample size per decision shrinks as responses get longer. Denser signals help if they are right: a critic (§5), Monte-Carlo rollouts from intermediate prefixes (VinePPO), or a process reward model that scores each step (Ch. 03, Ch. 06). The danger with any dense signal is that the policy optimizes the signal instead of the goal.

The potential-based shaping theorem

Ng, Harada & Russell (1999) answered a precise question: which extra rewards $F(s,a,s')$ can you add without changing the optimal policy? Answer: exactly those of the form

$$F(s,a,s') = \gamma\,\Phi(s') - \Phi(s)$$

for some potential function $\Phi$ over states. Sufficiency is a telescoping sum. The shaped return from $s_0$ is

$$\sum_{t=0}^{T}\gamma^t\big(r_t + \gamma\Phi(s_{t+1}) - \Phi(s_t)\big) = G_0 + \gamma^{T+1}\Phi(s_{T+1}) - \Phi(s_0).$$

If Φ is zero (or any constant) on terminal states, the shaped return is the original return minus $\Phi(s_0)$, a quantity no action can affect. Every policy's return shifts by the same per-start-state constant, so the ranking of policies, and therefore the optimum, is unchanged. Equivalently, $Q'(s,a) = Q(s,a) - \Phi(s)$, a shift that does not depend on $a$, so it is just a baseline. Ng et al. also show necessity: without further knowledge of the MDP, any non-potential shaping can change the optimal policy for some transition and reward function.

Three connections to the rest of the chapter. First, the TD error $\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$ is the reward shaped with potential $\Phi=V$. GAE is a discounted sum of potential-shaped rewards, which is why a critic can only help variance and never move the optimum (with λ = 1). Second, the per-token KL penalty is not potential-based: it deliberately changes the optimum, to the KL-regularized policy of Chapter 04. Third, length bonuses, format bonuses and "partial credit" heuristics are usually not potential-based either, and they are a classic source of reward hacking. The shaping widget is a two-line Goodhart demo.

§12Frontier & open problems

The math in this chapter is settled. Which pieces of it to use at scale is not.

• Token-level vs sequence-level ratios. PPO and GRPO use per-token ratios, even though the reward is per sequence. GSPO (Zheng et al., 2025) argued for a length-normalized sequence-level importance ratio and clipping, reporting better stability, notably for mixture-of-experts models. Which granularity matches the sampling unit is a live design question (Ch. 06).
• Off-policyness you did not choose. In large systems, rollouts come from an inference engine (vLLM, SGLang) whose numerics differ from the trainer's, and asynchronous pipelines train on stale samples. The "old policy" in the ratio is then not the policy that actually sampled. Yao et al. (2025) showed that vLLM-sampled and FSDP-trained probabilities differ enough to make "on-policy" training silently off-policy, and proposed truncated importance sampling. Later work traced much of the mismatch to bf16 rounding and argued that fp16 fixes it, and Meta's ScaleRL study (Khatri et al., 2025, over 400,000 GPU-hours) found that computing the LM head in fp32 raised the asymptotic performance. Off-policy corrections are now part of the recipe (Ch. 07).
• Which KL, where, and whether at all. Several reasoning-RL recipes drop the KL to the reference entirely. DAPO (Yu et al., 2025) argues that long chain-of-thought training should be allowed to move far from the initial model. Others keep it for safety and readability. The estimator-gradient mismatch of §9 means many published "β" values are not comparable, and the 2025 analyses (Tang & Munos; Shah et al.; Zhang et al.) have started to change what recipes actually ship (DeepSeek-V3.2's importance-weighted KL).
• Which loss, at scale? ScaleRL found that the choice of loss (it compared DAPO, GSPO and CISPO) and fp32 logits move the asymptote of the compute-performance curve, while normalization, aggregation and curriculum choices mostly change efficiency. Whether such sigmoidal fits extrapolate is open.
• Credit assignment over very long horizons. Agentic episodes of tens of thousands of tokens and many tool calls make the sequence-level bandit view increasingly wasteful. Value models, process rewards, turn-level advantages and Monte-Carlo rollouts are all being revisited.
• Entropy and exploration. Policy-gradient updates on a softmax systematically reduce entropy when high-probability actions get positive advantages (Cui et al., 2025). Managing it without an entropy bonus, through clip asymmetry or covariance-based clipping, is active work.
• Does RL teach or select? Whether RL on a pretrained model creates new capabilities or reweights existing ones is debated (pass@k analyses, Ch. 06). The policy-gradient view says it can only move mass among outputs the model already samples with non-negligible probability, which is why exploration and initialization matter so much.

§13Check yourself

Try each one before opening the answer. The "research" ones have no single right answer; the answer shows one good line of reasoning.

§14Read next