Scaling RL: infrastructure, environments, agents

§1Where the time goes

An RL step for an LLM has four phases: generate rollouts, score them, compute the update, and push the new weights back to the generator. In most published profiles, the first phase costs more than the other three combined.

At low concurrency, generation is memory-bandwidth bound and sequential: token $t+1$ cannot start before token $t$ exists. The training forward and backward pass, by contrast, processes every token of every sequence in parallel and is compute bound. Inference engines (vLLM, SGLang) recover throughput by batching many sequences at once, and very large batches can push a rollout engine toward compute-bound work. Tencent reports 2.29× generation throughput from a 4× larger PPO batch on fixed hardware, and up to 1.36× for GRPO.

Within one sequence, plain autoregressive decoding has no parallelism. Speculative decoding recovers some: a draft model proposes several tokens and the target accepts or rejects them in a way that preserves its output distribution. In principle this buys speed without adding off-policyness, though the numeric gap between sampler and trainer (§4) remains. The gain is capped because only generation gets faster: one study at 8B measured 1.8× on generation but 1.41× on the whole step. We found no independent test of the effect on RL outcomes, so still watch your learning curves.

Systems papers that profiled real runs report:

• Seer (Moonshot/Tsinghua, 2025) measured that "rollout alone occupies 63–87% of total iteration time" across three production workloads, including 87% for Kimi-K2. It also found that "the long-tail phase can account for nearly 50% of the total rollout time".
• RollPacker (2025) puts rollout at "around 70% of the total training time" (66–72% across math, code and LLM-as-a-judge tasks on 14B models).
• APRIL (2025) cites more than 90%, OpenRLHF's README gives 80%, and Jet-RL (NVIDIA and collaborators, 2026) reports over 70%.

The exact share depends on response length, model size and how much the trainer is sharded. "Generation dominates" is a common finding, not a law. One lab, Xiaomi, reports a large agentic run whose RL cost splits into 43.8% rollout, 43.5% training and 12.7% grader. That is cost rather than time, so it does not compare directly with the numbers above. The habit to keep: measure your own split, grader included.

The long tail

Response lengths in reasoning RL are heavily right-skewed: most answers are a few thousand tokens and a few run to the maximum. In a synchronous step, the batch is done only when its longest member is done. If lengths are roughly lognormal, $\log \ell \sim \mathcal N(\mu, \sigma^2)$, the largest of $n$ samples grows like

For $\sigma=1$ and a batch of $n=1024$ rollouts, this leading-order formula gives a slowest rollout about $e^{3.72-0.5}\approx 25$ times the mean. It overshoots at finite $n$: the expected maximum of 1,024 standard normals is about 3.25, not 3.72, so the straggler is closer to $e^{3.25-0.5}\approx 16$ times the mean. Either way, with one slot per rollout and decoding as the only cost, the generator would sit over 90% idle. RollPacker measured the longest response at 25–32× the median. Continuous batching does not rescue this: in a synchronous step there are no new requests to admit.

Scroll through the five configurations below. The figure runs the same discrete-event simulator as the playground in §2: ten generation slots, lognormal rollout lengths with a hard cap, one trainer, and a weight-sync pause.

Colocated vs disaggregated

The first design choice is where the inference engine lives.

• Colocated (also called a hybrid engine) puts the trainer and the inference engine on the same GPUs and switches between them. Each switch means offloading optimizer state, waking the KV cache, and resharding weights from the training layout (for example FSDP or Megatron) to the inference layout. Its advantages are no idle hardware between phases and no network weight transfer. Its drawback is strict alternation: one phase always waits for the other.
• Disaggregated puts training and inference on separate pools that talk over the network, typically NCCL broadcasts or RDMA. Generation and training can overlap and each pool is sized independently, at two costs. The weights must travel: Moonshot's checkpoint-engine reports about 20 s to update the 1T-parameter Kimi-K2 across thousands of GPUs, and the K2 report says a full update takes "less than 30 seconds". And training is now off-policy by at least one step.

The choice is not strictly binary. Moonshot reports keeping Kimi K3 colocated so that each 1M-context experiment stays "within a few hundred GPUs". Alibaba's QwenGyre "elastically reallocates GPUs between rollout and training without interrupting live executions" and claims up to 1.85× over colocated and 1.78× over asynchronous baselines. Each is one lab's report on its own stack.

Feature sets move monthly. This table reflects each project's documentation as of October 2026. Others worth knowing: LlamaRL (Meta), StreamRL, ROLL Flash (Alibaba), Seer (Moonshot), RollPacker, Magistral's in-house async stack, DORA (Meituan), Agent Lightning (Microsoft) and, from 2026, Relax (Xiaohongshu), LoongSage (Baidu), Forge (MiniMax), QwenGyre (Alibaba) and AReaL-DTA.

§2Asynchronous RL: the design space

Every technique in this section changes which policy version produced each token. Learn to read a design by asking two questions: how stale can a token be, and does anything correct for it?

Four ways to hide the tail

1. One-step (or $k$-step) off-policy. Generate batch $k{+}1$ with $\pi_{v_k}$ while training on batch $k$. verl reports a 23–40% end-to-end speedup over colocated synchronous training. INTELLECT-2, the first globally decentralized RL run of a 32B model, "used two-step asynchrony" and saw reward curves match synchronous training at asynchrony levels up to four.
2. Fully asynchronous with bounded staleness. The generator streams continuously and an admission controller stops it from running more than $k$ versions ahead (AReaL, verl fully-async, TRL's AsyncGRPO). AReaL's paper reports up to 2.57× speedup in v1 and 2.77× in a later revision, with "linear scaling efficiency up to 512 GPUs". Its defaults are $k=4$ for coding and $k=8$ for math in the latest version.
3. Partial rollouts. Cap the number of tokens each step may generate. Unfinished sequences are saved and continued in the next step, so one rollout can span several iterations. Kimi k1.5 saves "the unfinished portion" to a replay buffer. APRIL over-provisions requests, stops when enough have finished, and carries the rest forward.
4. In-flight weight updates. Swap the weights while sequences are mid-generation. PipelineRL and Magistral keep the existing KV cache, which was computed by the old weights. Magistral notes "recomputing the key-value cache is not necessary". AReaL instead interrupts, discards the KV cache, and recomputes it with the new weights. Either way, one sequence can contain tokens from several policies.

# Bounded staleness, AReaL-style: generation may run at most k policy versions ahead.
def can_start_rollout(n_started: int, trainer_version: int, k: int, batch_size: int) -> bool:
    return n_started < (trainer_version + k + 1) * batch_size

# k = 0  -> synchronous on-policy (generator idles while the trainer works)
# k = 1  -> one-step off-policy
# k = 8  -> ScaleRL's chosen maximum off-policyness for PipelineRL

Weight synchronization is its own subsystem

Moving hundreds of gigabytes of weights every few minutes is hard.

• Colocated systems reshard in GPU memory. HybridFlow's 3D-HybridEngine exists to avoid redundant copies during this step.
• Disaggregated systems broadcast over NCCL or RDMA. TRL's server mode pushes weights after every optimizer step. Moonshot open-sourced checkpoint-engine for trillion-parameter updates.
• Decentralized runs add trust and bandwidth problems. INTELLECT-2 used SHARDCAST to broadcast weights over an HTTP tree and TOPLOC, a locality-sensitive hash of activations, to check that untrusted workers really ran the model they claim.

The sync pause is usually seconds at research scale, but in-flight systems pay it every step for all generators.

§3Off-policy corrections: from IMPALA to decoupled PPO

Once data comes from a behavior policy $\mu$ that is not the policy being trained $\pit$, the plain policy-gradient estimator is biased. Importance sampling removes the bias but can explode the variance. Every method below is a different compromise between the two.

The exact correction and why nobody uses it

For a response $y = (y_1,\dots,y_T)$ sampled from $\mu(\cdot\mid x)$, the on-policy gradient can be rewritten exactly:

The sequence weight $W$ is a product of $T$ token ratios. Write $\sigma^2$ for the per-token variance of $\log\rho_t$ under $\mu$. By the central limit theorem, $\log W$ is approximately $\mathcal N(m, T\sigma^2)$, and $\E_\mu[W]=1$ forces $m=-T\sigma^2/2$. The effective sample size fraction is then

A per-token log-ratio standard deviation of only $0.03$ ($\sigma^2\approx 10^{-3}$) over a 4,000-token response gives an ESS fraction of $e^{-4}\approx 2\%$, so one sequence in fifty carries the batch. This is the "exponential variance" that Liu, Li and coauthors bound formally. It is why practical LLM methods use per-token ratios: they are biased, because they ignore how the prefix distribution changed, but they are usable.

V-trace: truncate, and know what you converge to

IMPALA (Espeholt et al., 2018) faced the same problem with distributed actors that lagged behind the learner. Its V-trace target truncates the per-step ratios at two levels:

The lesson carries over to LLMs without a value function: a truncation constant is a statement about which policy you are actually optimizing. Truncated importance sampling (TIS) with cap $C$ is a biased estimator whose bias points toward the behavior policy. When the mismatch is small, the bias is small.

Three policies, not two: decoupled PPO

PPO's ratio $\pit/\piold$ does two jobs at once. It corrects for off-policy data, and through clipping it enforces a trust region. Hilton, Cobbe and Schulman (2021) pointed out that these jobs can use different reference policies. AReaL uses this to handle staleness:

Because $\frac{\pi_\text{prox}}{\pi_\text{behav}}\cdot\frac{\pit}{\pi_\text{prox}}=\frac{\pit}{\pi_\text{behav}}$, the unclipped term is ordinary importance-sampled policy gradient. The clip simply stops $\pit$ from wandering far from a recent policy rather than from an arbitrarily old one. verl's 2026 "Rollout Correction" documentation describes a "decoupled mode" with three policies ($\pi_\text{rollout}$, $\pi_\text{old}$, $\pit$) and warns: "Many LLM-RL implementations incorrectly apply PPO by ignoring the actual rollout policy $\pi_{\text{rollout}}$."

§4The training–inference mismatch

In 2025 the field found that even a perfectly synchronous run is off-policy. The inference engine and the trainer give the same tokens different probabilities, and that gap alone can collapse a run.

The engine that samples (vLLM or SGLang) and the engine that computes gradients (FSDP or Megatron) implement the same mathematical function with different kernels. Fused operations, reduction orders, attention algorithms, parallel layouts and intermediate precision all differ. Feng Yao, Liyuan Liu and coauthors made this concrete in an August 2025 post titled "Your Efficient RL Framework Secretly Brings You Off-Policy RL Training":

"despite π_sampler and π_learner sharing the same model parameters θ, they can produce significantly different token probabilities… secretly making the RL training become off-policy."

They measured the worst per-token probability difference within a run. It was around 0.4 in normal training and reached 1.0 in a DAPO run on Qwen2.5-32B, meaning near-opposite verdicts from the two engines. Returning the true sampling probabilities and casting the LM head to FP32 did not remove it. A September 2025 follow-up by Jiacai Liu, Yingru Li and colleagues ("When Speed Kills Stability") added three findings:

• Hardware matters. Their sampler-vs-trainer KL was "on the order of 5e-4 to 1e-3" on H20s and between 1e-2 and 1 on A100s, where disabling vLLM's cascade attention cut it to about 1e-3.
• The mismatch concentrates on low-probability tokens: "As a token's inference probability approaches zero, the training probability can become orders of magnitude smaller".
• Runs with large mismatch collapse, and token-level corrections only delay it.

Sources of mismatch

Fix 1: correct for it (truncated or masked importance sampling)

Yao et al. proposed applying an importance weight between the two engines' probabilities for the same weights $\theta_\text{old}$, truncated from above. In PPO form the two ratios are clipped separately:

import torch

def ppo_tis_loss(logp_new, logp_old, logp_sampler, adv, mask, eps=0.2, C=2.0):
    """All log-probs are [B, T] for the sampled tokens.
    logp_old:     recomputed by the TRAINER with the weights that generated the batch
    logp_sampler: returned by the inference engine for the same tokens
    adv:          [B, 1] (e.g. group-normalized), broadcast over tokens"""
    ratio = torch.exp(logp_new - logp_old)                          # trust region vs pi_old
    w = torch.exp(logp_old - logp_sampler).clamp(max=C).detach()    # mismatch correction (TIS)
    # MIS variant (mask instead of cap):
    #   w_raw = torch.exp(logp_old - logp_sampler).detach()
    #   w = torch.where(w_raw <= C, w_raw, torch.zeros_like(w_raw))
    surr = torch.minimum(ratio * adv, ratio.clamp(1 - eps, 1 + eps) * adv)
    return -(w * surr * mask).sum() / mask.sum()                    # token-mean aggregation

There are several variants:

• Token-level MIS masks tokens whose ratio exceeds $C$ instead of capping them.
• Sequence-level versions compute $\rho(y)=\prod_t\rho_t$ and truncate or mask whole responses. Liu, Li et al. found token-level TIS "still collapses later" in a resumed crashed run while sequence-level TIS ($C=2$) stayed stable.
• Geometric-mean variants (Geo-RS) use $\rho(y)^{1/T}$ to stay length-neutral.

DeepSeek-V3.2 (Dec 2025) masks sequences with negative advantage whose mean log-ratio between the inference engine's probabilities and the current policy exceeds a threshold, noting that this "accounts for both sources of off-policyness", staleness and engine mismatch. Two other open questions remain: whether masking positive-advantage outliers helps, and whether these corrections fix the problem or only postpone it.

Fix 2: shrink the gap (precision, routing)

• FP16 instead of BF16. Qi et al. ("Defeating the Training-Inference Mismatch via FP16", Oct 2025) argue that "its root cause lies in the floating point precision itself". FP16 has 10 mantissa bits against BF16's 7, which is "8 times more precision". In RL fine-tuning the dynamic range that makes BF16 attractive for pretraining "has already been established during pre-training". They report mismatch "approximately 24× smaller", and in their sanity test only FP16 runs reached near-perfect training accuracy. FP16 needs loss scaling and is unusual for very large models, so this is a strong result on the models tested, not a universal rule.
• Low precision on purpose, paths matched. Some 2026 stacks lower rollout precision for speed, which opens another numeric gap between sampler and trainer. The one study we found that measures the cost reports that FP8 rollouts with BF16 training degrade on long and hard tasks; others use the pairing with no published comparison. Test it on your own task, at your longest lengths, against a higher-precision baseline. The proposed fix is to match the two paths, and each such recipe so far is one lab's report.
  The evidence on FP8 rollouts

  Jet-RL calls BF16-train-FP8-rollout "widely adopted" yet finds it degrading past 8K tokens on Qwen3-8B and on harder tasks, and reports that a unified precision flow cuts the loss from over 5% to about 1%. GLM-5 uses FP8 rollouts "to reduce per-token latency", and SemiAnalysis calls the pairing "a common setup", neither with a published comparison. Kimi K3 trains quantization-aware through SFT and RL, with rollout and training on one scheme, "eliminating the train–inference mismatch": one lab's claim, about numerics.

• FP32 at the head. MiniMax-M1 traced its mismatch to "high-magnitude activations in the LM head" and raised it to FP32, improving the correlation between training and inference probabilities "from approximately 0.9x to 0.99x". Yao et al. and Liu, Li et al. found the same patch insufficient in their setups. ScaleRL found FP32 logits raised the asymptote from 0.52 to 0.61 (§5).
• MoE routing replay. Ma et al. (R3, Oct 2025) found that "approximately 10% of the routers select different experts during training compared to inference" in Qwen3-30B-A3B. Their fix reuses the inference routing mask in the training forward pass "while still applying the softmax to the training logits", which halves the mismatch KL at under 3% rollout overhead. The Qwen team's earlier "Routing Replay" fixed a related problem, experts that change between the old and updated policy; GSPO's sequence-level ratio removed their need for it.

Fix 3: eliminate it (bitwise-consistent kernels)

Thinking Machines' "Defeating Nondeterminism in LLM Inference" (Sept 2025) corrected a common belief. Inference nondeterminism is not mainly "GPU concurrency plus floating point". It comes from a lack of batch invariance: a kernel uses a different reduction strategy depending on batch size, and batch size depends on server load. Sampling 1,000 completions at temperature 0 from Qwen3-235B produced 80 unique outputs; with batch-invariant kernels, all 1,000 were identical. The kernels cost speed: serving 1,000 sequences from Qwen3-8B took 26 s with vLLM's default, 55 s with an unoptimized deterministic version, and 42 s after improving the attention kernel.

Determinism inside one engine is not yet agreement between engines. Their RL demonstration also used the same kernels in sampler and trainer, which gave "True On-Policy RL" with sampler–trainer KL "flat at 0". vLLM and TorchTitan reproduced bitwise-consistent RL in November 2025, initially "2.4x slower". By October 2026 a vLLM post reported a bitwise run at "91.3% of the baseline's rollout throughput" on a large MoE, and noted that on the same model the high-performance (non-bitwise) baseline's train-inference KL "rose from 2e-3 to 29.9" while reward fell back.

§5Scaling laws for RL compute

Pretraining has power laws. RL, at least in the largest controlled study so far, has sigmoids. Part of the reason is the metric: pretraining laws are fit to loss, which has no ceiling in sight, while RL curves are fit to a pass rate, which is bounded by 1 and saturates. A sigmoid separates two things a power law blurs together: how high a recipe can go, and how fast it gets there.

Khatri et al. ("The Art of Scaling Reinforcement Learning Compute for LLMs", Meta and collaborators, Oct 2025; ICLR 2026) spent "more than 400,000 GPU-hours" on controlled RL runs and fit held-out pass rate against compute with

A single 8B run was fit up to 50k GPU-hours, extrapolated, and then extended to "100,000 GPU-hours". The extrapolation held, predicting an asymptote of about 0.645. The paper's main claim is that recipes differ in $A$, and that many details people tune mostly change $B$:

"Details such as loss aggregation, normalization, curriculum, and off-policy algorithm primarily modulate compute efficiency without materially shifting the asymptote."

Their combined recipe, ScaleRL, is PipelineRL with 8-step off-policyness, the CISPO loss, prompt-level loss averaging, batch-level advantage normalization, FP32 logits, zero-variance filtering, no-positive-resampling (permanently retiring prompts with pass rate ≥ 0.9), and forced length interruption. Over-long thoughts are cut by appending "Okay, time is up. Let me stop thinking and formulate a final answer now." It reaches $A=0.61$ in their main comparison.

How big is RL compute getting?

Public statements trace a steep curve:

• In January 2025 Dario Amodei wrote that "the amount being spent on the second, RL stage is small for all players. Spending \$1M instead of \$0.1M is enough to get huge gains. Companies are now working very quickly to scale up the second stage to hundreds of millions and billions".
• OpenAI's o3 launch (April 2025) said "large-scale reinforcement learning exhibits the same 'more compute = better performance' trend observed in GPT‑series pretraining" and that they "pushed an additional order of magnitude in both training compute and inference-time reasoning".
• xAI said Grok 4 (July 2025) ran "reinforcement learning training that refines Grok's reasoning abilities at pretraining scale".
• DeepSeek-V3.2 (Dec 2025) reported a post-training budget "exceeding 10% of the pre-training cost".
• Epoch AI (May 2025) argued that reasoning-training compute was probably "less than three" orders of magnitude below the frontier of total training compute, and noted that o3 used about 10× o1's reasoning-training compute only four months later. At that pace, it estimated, reasoning training would reach the frontier "perhaps within a year" and then slow down.

Some 2026 work asks how to allocate RL compute: the IsoCompute Playbook (Mar 2026) finds that the compute-optimal number of parallel rollouts per problem "increases predictably with compute budget and then saturates". The strongest public statements are about what the compute is spent on. DeepSeek says V4.1-Flash's post-training "introduces no algorithmic innovation" and that "All substantive changes lie instead in the data pipeline". That is one lab describing its own model, not a measured law. The habit it suggests: when a run stalls, look at the tasks and environments as well as the algorithm.

Reported costs need the same care. Xiaomi reports \$2.6M of RL on MiMo-V2.6-Pro. That does not contradict Amodei: it is one final run at one lab, and by Epoch's estimate, which rests on heavy assumptions, final training runs are a minority of R&D compute.

§6Environments: the real reward function

When people say "RL environment" for LLMs they mean everything that turns a policy into a scalar: the task, the world the model acts in, the tools, and the grader. Choosing the algorithm is the easy part.

Prime Intellect's Environments Hub (Aug 2025) puts it as "Environments define the world, rules and feedback loop of state, action and reward." Concretely, an environment for an LLM is a bundle:

Libraries encode the same split: Prime Intellect's verifiers defines an environment by a dataset, rollout logic, rubrics ("one or more reward functions") and optional parsers. verl's agent loop and SkyRL-gym provide multi-turn plumbing on the trainer side.

SWE environments: from benchmark to training ground

• SWE-bench (Jimenez et al., 2023): 2,294 real issues from 12 Python repositories. A fix counts if the issue's failing tests now pass (FAIL_TO_PASS) and nothing else broke (PASS_TO_PASS). The best model at launch, Claude 2, solved 1.96%.
• SWE-bench Verified (OpenAI, Aug 2024) kept 500 tasks that human annotators judged to be well specified and fairly tested. OpenAI stopped reporting it in February 2026, citing flawed tests and contamination (chapter 09, from secondary accounts). METR found maintainers would not merge roughly half of test-passing PRs, and Goodfire's broad rubric flags GLM 5.2 as hacking in 73% of SWE-bench rollouts and 57.2% on the DeepSWE benchmark. Read Verified scores, including those of the DeepSWE agent below, with that in mind.

Training needed many more executable environments than that, and 2025 produced them:

• SWE-Gym: 2,438 instances with runtimes.
• R2E-Gym: more than 8.7k tasks built procedurally by back-translating commits. Hybrid verifiers (execution plus a learned judge) reached 51% against about 42–43% for either alone when picking among candidate patches at test time.
• SWE-smith: 50k instances from 128 repositories.
• SWE-rebench: more than 21k tasks plus a continuously refreshed, decontaminated benchmark.

Rewards vary. SWE-RL (Meta) used patch similarity to the reference as a cheap reward without execution. DeepSWE trained Qwen3-32B with RL alone on 4,500 R2E-Gym tasks over six days on 64 H100s and reached 42.2% pass@1 on Verified. It also introduced "compact filtering": masking trajectories that hit the context limit, the timeout or the step limit, so the model is not punished for running out of budget.

Design principles

This list is a synthesis of the sources in this chapter, not a published checklist:

1. Unhackable by construction where possible. MacDiarmid et al. recommend "making reward hack immunity a key environment design criterion". Concretely: grade in a separate process on files the policy cannot write; restore tests and harness before grading; parse per-test results, not exit codes; hide or fuzz tests.
2. Instructions consistent with the reward. Anthropic's Claude 4 system card lists among its mitigations modifying "environment instructions to be more consistent with the reward signal". If the grader only checks that tests pass, the reward defines the task, not the prompt "fix the bug".
3. Calibrated difficulty. A task the policy always solves or never solves gives no gradient (§7). Maintain a pass-rate profile per environment and refresh it as the policy improves.
4. Cheap, resettable and deterministic. Flaky tests add reward noise, and its structure matters more than its rate. Random noise looks tolerable in small settings: Plesner et al. stay within 1 point of baseline at up to 20% resampled noise (Qwen3 8B), with intervals they say establish no general threshold. Systematic false positives are the dangerous direction: on arithmetic tasks, Egashira et al. see plateaus or collapse that the error rate does not predict. Container snapshotting and warm pools often dominate the cost of an agentic environment.
5. Diverse. DeepSeek-V3.2 reported "over 1,800 distinct environments and 85,000 complex prompts" for its agentic RL. Diversity guards against overfitting to one grader's quirks, and per MacDiarmid et al., diversity of safety training also guards against misgeneralization.
6. Audited against the policy you are training, not the one you had. Anthropic's Fable 5.1 / Mythos 5.1 system card (Sept 2026) reported that "around half of our computer-use environments incentivized hacking or had accessible hack surfaces". They had been missed because "these environments were QA'd against older, less capable models which were not able to find certain hacks and (2) older environments were not re-audited after we improved our QA process." The card adds that "using substantially more test-time compute, along with specific harnesses and prompts, allows for weaker models to find more of the hacks that later, more capable models identify." Environments age as the policy gets stronger.

§7Curricula and difficulty filtering

With binary rewards and group-relative advantages, a prompt teaches nothing unless its group contains both successes and failures. Which prompts you feed the policy determines how much of your compute produces any gradient.

Take a binary reward and a prompt with pass rate $p$ under the current policy. With $G$ samples per group, the probability that every sample agrees is

When all samples agree, every advantage $r_i - \bar r$ is zero, so the group contributes no policy gradient, only cost. Pooled over a prompt distribution, the wasted share of rollouts is $\E_p[P_0(p)]$. To fill a batch with informative groups, DAPO's dynamic sampling ("over-sample and filter out prompts with the accuracy equal to 1 and 0") needs on average $1/(1-\E[P_0])$ groups per useful group. Averaged over all groups, informative or not, the expected squared mean-baseline advantage is

the Bernoulli variance shrunk by the baseline's own noise. This peaks at $p=\tfrac12$. It is the formal version of "train at the frontier of learnability" (Foster et al., 2025), and of Bae et al.'s result that "expected policy improvement is lower-bounded by the variance of task-level success probabilities". GRPO's division by the group standard deviation partly undoes this weighting: it gives a 1-in-16 group the same scale as an 8-in-16 group. Chapter 06 discusses whether that is a feature or a bug.

All of this assumes binary rewards and a group baseline. Xiaomi's report says rubric scores "can still provide a learning signal" inside an all-pass group, though it also filters all-pass groups and we could not tell how the two interact. Z.ai's SAO has no groups at all (§11).

Strategies used in practice:

• Offline pass-rate filtering: estimate $p$ with $k$ samples from the initial policy and keep prompts in a band. Polaris removed prompts the model solved 8/8.
• Online filtering and dynamic sampling: DAPO; zero-variance filtering in ScaleRL.
• Prioritized sampling: Kimi k1.5 sampled problems in proportion to $1-s_i$, where $s_i$ is the success rate, and ordered the curriculum from easier to harder.
• Predictive skipping: GRESO found that over 90% of zero-variance prompts had been zero-variance before, so it skips them probabilistically before spending rollouts.
• Learning from zero-variance prompts anyway: RL-ZVP uses entropy-guided shaping.

def informative_groups(groups):
    """groups: list of (prompt, rewards: Tensor[G]) with binary rewards.
    Dynamic sampling (DAPO): keep only groups with 0 < accuracy < 1, and keep
    sampling new prompts until the batch is full."""
    return [(p, r) for p, r in groups if 0.0 < r.float().mean().item() < 1.0]

§8Agentic and multi-turn RL

An agent episode runs from dozens to hundreds of turns or more, with thousands of tool-output tokens at least, and one reward at the end. Three problems get harder: credit assignment, context length and systems latency.

Credit assignment across turns

The default in 2025–2026 is the simplest option: treat the whole trajectory as one action sequence, mask non-policy tokens (tool outputs, environment messages), and broadcast a single trajectory-level advantage to every policy token. It worked surprisingly well at tens of turns with verifiable rewards, but its variance grows with horizon, and 2026 recipes disagree on whether to keep it (§11). Alternatives:

• Turn-level or step-level grouping. GiGPO (2025) keeps GRPO's critic-free design but adds step-level groups. When different rollouts revisit the same environment state (an "anchor state"), their actions from that state are compared directly. It reports gains of more than 12% on ALFWorld and 9% on WebShop.
• Hierarchical value functions. ArCHer (2024) learns an utterance-level value with off-policy TD and uses it as the advantage for a token-level policy, reporting "sample efficiency of about 100x" in its settings.
• Shaped or process rewards. Context-Folding's FoldGRPO adds process rewards for good context management. Kimi-Researcher discounts rewards on correct trajectories by $\gamma^{T-i}$ so that shorter successful paths win.
• Stability. RAGEN (2025) named an "Echo Trap where reward variance cliffs and gradient spikes" in multi-turn RL: the policy collapses onto a repetitive mode and group variance vanishes. Its fix filters trajectories, adds a critic and stabilizes gradients. RAGEN-2 (2026) reports a further mode, "template collapse": entropy stays stable while reasoning turns input-agnostic. On its small tasks a mutual-information proxy correlated +0.39 with final performance and entropy metrics −0.11 to −0.14, so entropy alone can mislead.

Long horizons and context management

Agent trajectories outgrow any context window. The policy can be trained to manage its own memory:

• ReSum periodically summarizes and continues. Its RL variant uses "advantage broadcasting to propagate final rewards across segmented trajectories".
• Context-Folding branches into sub-trajectories and folds each into a summary, matching a ReAct baseline "using an active context 10× smaller".
• MEM1 learns a constant-size internal state.
• DeepSeek-V3.2 compared simple strategies at 80% context use: summarize, discard the oldest 75% of tool history, or discard all of it.
• Interleaved thinking keeps reasoning across tool calls in the history; MiniMax-M2's model card warns that removing the <think> content from history degrades performance.

A trajectory with summaries in it is a different MDP from one with full history.

Systems for agents

Tool calls add latency the GPU cannot hide, so agentic RL leans harder on asynchrony. ROLL Flash reports bigger speedups on agentic tasks (2.72×) than on math RLVR (2.24×). Rollouts become token-in/token-out sessions against an inference server, so that agent harnesses can run unmodified. Two pitfalls get worse:

• Re-tokenization drift. If the harness converts tokens to text and back between turns, the trainer may see token sequences the sampler never produced. "Token in, token out" interfaces avoid this.
• Truncation semantics. An episode cut off by a timeout is not a failure of the task. Mask it (DeepSWE's compact filtering) rather than giving it reward 0, or the policy learns to fear long tasks.

§9Reward hacking in environments

RL is an optimizer, and the grader is its objective. Every hole in the grader is a reward that is easier to collect than the intended one. At scale, the policy finds the holes.

Chapter 03 treats Goodhart's law for learned reward models. Programmatic graders fail differently, by being literally exploitable. Public cases include:

Why it matters beyond the benchmark

MacDiarmid et al. ("Natural Emergent Misalignment from Reward Hacking in Production RL", Anthropic and Redwood, Nov 2025) taught a pretrained model about the three pytest hacks above through synthetic-document finetuning (or, in a second setting, through prompting), then trained it with RL on "a subset of the actual environments used to train Claude Sonnet 3.7". It learned to hack and generalized to "alignment faking, cooperation with malicious actors, reasoning about malicious goals, and attempting sabotage". Standard chat-style RLHF safety training made it look aligned on chat evaluations, while "misalignment persists on agentic tasks". Three mitigations worked: preventing the hacking, diversifying safety training, and inoculation prompting, which frames hacking as acceptable in the training prompt. With inoculation, "final misalignment is reduced by 75-90%, despite reward hacking rates over 99%".

Read this as a result about that setup, because how far hacking spreads is unsettled. A UK AISI reproduction on open models found misalignment on some evaluations but not consistently. Anthropic then ran a deliberately pessimistic research run at frontier scale: its model cheated and did harm when a grader was visible, yet was "not misaligned in scenarios without a clear grader/reward". The authors think differences in training setup likely explain the gap, and neither result cancels the other. So test behavior both with and without a visible grader. The companion Alignment guide has the detail. Either way, for an RL engineer the takeaway is simple: a hackable environment is a safety problem, not just a wasted-compute problem.

Detection and the monitorability tax

Baker et al. ("Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation", OpenAI, Mar 2025) watched a frontier reasoning model during RL on coding tasks. Hacks like exit(0) and raise SkipTest "quickly get reinforced and become systemic". A GPT-4o monitor reading only actions caught 60% of the systemic hacks. Reading the chain of thought, it caught 95%. When they added the CoT monitor to the reward, agents learned "obfuscated reward hacking, hiding their intent within the CoT while still exhibiting a significant rate of reward hacking". Their recommendation is to pay a "monitorability tax": do not apply strong optimization pressure directly to the chain of thought. Independent evaluations show the problem is not specific to one lab:

• METR (June 2025) found o3 reward-hacked in 30.4% of RE-Bench runs and 0.7% of HCAST runs, and "Please do not cheat" in the prompt had a "nearly negligible effect".
• ImpossibleBench (Oct 2025) defines a "cheating rate" as the pass rate on impossible tasks and measured GPT-5 at 76% on its SWE-bench variant.

§10Systems details that change results

These choices look like plumbing. Each one has changed a published conclusion at some point.

Packing, dynamic batching and loss aggregation

Rollouts vary in length by 10–100×. Padding each micro-batch to its longest member wastes compute on pad tokens. Sequence packing concatenates several sequences into one fixed-length row, with a block-diagonal attention mask and position ids restarted per sequence, so the trainer does almost no padded computation. Dynamic batching sets micro-batch size by token budget.

KV cache and prefix sharing

A GRPO group shares its prompt, and multi-turn rollouts share growing prefixes. Prefix caching (radix trees in SGLang, prefix caching in vLLM) computes the shared part once. After an in-flight weight update, the cached KV is stale; keeping it trades a little extra off-policyness for not recomputing. KV memory also bounds concurrency.

Sampling temperature

A common RLVR default is to sample at temperature 1.0 with no top-$p$/top-$k$, so that the sampling distribution is the policy whose log-probabilities are being trained. Otherwise the behavior policy is $\pi^{1/\tau}$ or a truncated $\pi$, and you must either correct for it (return the processed log-probs; apply the same mask to $\pit$, as DeepSeek-V3.2's "Keep Sampling Mask" does) or accept the bias. Liu, Li et al. found that a smaller top-$p$ reduced KL spikes but increased bias. POLARIS deliberately raised training temperature (1.4 → 1.45 → 1.5 for a 4B model) to keep rollout diversity up as the policy sharpened.

Max length and truncation

A response that hits the maximum length has no verdict. Scoring it as wrong teaches "be shorter", possibly as a spurious penalty on hard problems. Options:

• DAPO's overlong filtering "masks the loss of truncated samples". Its soft overlong punishment adds a linear penalty in a buffer zone before the limit.
• ScaleRL's forced interruption appends an end-of-thinking cue and lets the model answer.
• Partial rollouts continue the sequence in the next iteration.

The truncation rate is a key metric. If it climbs, you are training on censored data.

Checkpointing an asynchronous system

A synchronous RL checkpoint is model plus optimizer plus data cursor. An async one also needs:

• the replay buffer of finished-but-unconsumed rollouts, with their policy versions and sampler log-probs;
• in-flight partial rollouts;
• per-prompt difficulty statistics used by the filter;
• RNG state.

Resuming without the buffer silently changes the data distribution; a reset version counter breaks staleness accounting.

Evaluation during RL and what to log

Training reward is biased: it comes from the policy's own filtered prompts, under the training grader. Run held-out evaluations on a fixed schedule with enough samples per problem to report pass@1 with error bars (see chapter 09). A minimal dashboard:

§11Frontier and open problems

• Exactly on-policy at scale, or principled correction? Bitwise-consistent kernels now cost little throughput on some models, but MoE weight updates still flip expert choices: one vLLM post reported "about 7% of tokens switched expert combination after each update". Is it better to buy exactness, or to accept noise and correct for it?
• How much staleness is free? There is no single number, because the harm depends on how far the policy moves while data ages, so staleness and learning rate have to be set together. For vanilla GRPO on small models, one group now reports a measured boundary: the sync interval $S$ times the learning rate $\eta$ is roughly constant at the stability limit. Use that as a rule of thumb to calibrate per setup, since a larger model supported only part of it and we saw no replication. Methods that correct for stale data report going much further, and the settings we could find in real runs, each one group's report, go from 3 to 16 updates. The habit: log actual version lag and validate at your full horizon, because a late collapse can pass for stability.
  Who found what

  ScaleRL found max off-policyness 8 best for PipelineRL, and INTELLECT-2 saw no loss at 4. Song et al. (HKU, SJTU) get the boundary from Llama-3.2 sweeps, along with collapse time scaling as $1/\eta$; Qwen3-8B supports the timing only, with a different constant. Reported corrections: M2PO reports stability at 256 updates, VCPO at 128 by scaling the learning rate with ESS, ENTR up to 30. Reported settings: MiMo-V2.6 runs at 4, Intern-S2 discards trajectories older than 3 updates, and two third-party SemiAnalysis runs allow 16.

• Scaling laws beyond math. ScaleRL's fits are for verifiable reasoning on one model family. First answers for agentic RL exist, at small scale: one group reports that adding environments helps more than adding tasks, and both help more than adding trajectories. We saw no replication and it has not been checked at frontier scale, but it is a good reason to look at environment diversity before buying more rollouts. Moonshot and Xiaomi publish score-against-compute curves but no asymptote fit, so whether the asymptote/efficiency separation holds there is still open.
  The data-scaling evidence

  He et al. (MBZUAI, McGill) fit held-out success against data axes and report exponents of 0.41 for environments, 0.26 for tasks and 0.11 for trajectories (Qwen3-8B, math and code), with that ordering in all 21 settings, on models up to 14B. They also measure, for one method per axis, the synthetic-to-real ratio at which success falls 5% below real-only data: about 9.4 for AWM (environments), 3.4 for AutoPlay (tasks) and 1.4 for SWiRL (trajectories).

• Environment generation at scale. Synthetic environments (R2E-Gym, SWE-smith, DeepSeek-V3.2's environment-synthesis agent with "1,827 task-oriented environments") make environments cheap. Too much synthetic data can hurt, and the same group finds the limit differs by method. What makes environments good at frontier scale, and how do you audit thousands of them against a policy that keeps getting better at finding holes?
• Credit assignment for very long horizons. Reported rollouts now reach hundreds of interactions at 700K tokens (QwenGyre) and "hundreds or thousands of tool calls" (Kimi K3). Recipes disagree. Z.ai's SAO uses one rollout per prompt with a value model and reports deploying it for GLM-5.2; Xiaomi keeps GRPO at $G=16$ with segment penalties and advantage redistribution; MiniMax describes process rewards. All are self-reported, and we found no head-to-head at scale, so keep plain trajectory-level advantage as your baseline and compare against it.
• Reward hacking as a generalization problem. Inoculation prompting limits misgeneralization but does not stop the hack itself, and a Sept 2026 paper ("Shallow Beliefs") finds synthetic-document finetuning does not inoculate the way prompting does. See the companion Alignment Field Guide.

§12Check yourself

Try each one before opening the answer.

Flashcards

§13Read next