Scalable oversight

§1The problem: supervising what you cannot evaluate

Every alignment technique in production today assumes that someone, human or model, can tell good outputs from bad ones. Scalable oversight asks what happens when nobody in the loop can.

In RLHF (covered mechanically in the Post-Training Field Guide) we fit a reward model to human comparisons and then optimize a policy against it. The policy never sees "the truth"; it sees what a rater would prefer after a few minutes of reading. For a summary of a short email, those are close. For a 4,000-line refactor, a proof sketch in an unfamiliar subfield, or an agent that ran 300 shell commands overnight, they diverge, and RL is very good at finding exactly where they diverge. Chapter 01 calls this reward hacking; this chapter is about the case where the "reward function" is a human or model judgment that is simply not competent enough.

Amodei et al. (2016) already listed it as one of five concrete safety problems, under the name scalable supervision: the objective is too expensive to evaluate frequently. The modern framing is sharper. Anthropic's 2023 "Core Views on AI Safety" put it this way: "we need to magnify a small amount of high-quality human supervision into a large amount of high-quality AI supervision", and "the only way to provide the necessary supervision will be to have AI systems partially supervise themselves or assist humans in their own supervision." Anthropic's Alignment Science team's 2025 research-directions post splits the difficulty in three: the oversight signal can be noisy (the problems are fundamentally hard), expensive, or carry systematic, exploitable errors. The last is the dangerous one.

Making the gap precise

Let $R^*(x,y)$ be the reward we actually want (is the code correct? is the answer true?) and $\tilde R(x,y)$ the reward the overseer actually assigns, possibly with AI assistance. Write the overseer's error as $\varepsilon = \tilde R - R^*$. For any policy,

The identity is trivial; the content is in the last term. If $\varepsilon$ were independent noise, optimization pressure would average it out. If $\varepsilon$ is systematic (raters reward confident tone, miss subtle bugs, cannot check citations), a capable policy can model it and climb it. The scalable oversight program is: make $\varepsilon$ small and unexploitable on the distribution the policy will actually reach, even when the policy is more capable than any single overseer. Its sibling program, AI control, asks a different question: assuming we failed and the model is adversarial, can we still prevent catastrophe at deployment time? The two are complementary; mixing them up is a common confusion.

§2Sandwiching: measuring progress before superhuman AI exists

You cannot run an experiment on superhuman systems that do not exist yet. You can, however, find tasks where today's models sit between two groups of humans.

Ajeya Cotra's 2021 essay "The case for aligning narrowly superhuman models" argued for doing alignment research now, on tasks where current models already beat some humans. Bowman et al. (2022) turned this into an experimental design, sandwiching: pick tasks where a model outperforms non-experts but underperforms experts. Give the non-experts the model and an oversight technique. Withhold the experts entirely, except at the end to grade. If the non-experts plus technique get reliably close to expert performance, without ever being told the answer, the technique is doing something real.

§3Debate: let two experts argue, check one claim

You cannot check a long argument. You can check one step of it, if someone motivated to find the flaw tells you which step.

Irving, Christiano and Amodei (2018) proposed training agents by self-play on a zero-sum game: given a question, two agents take turns making short statements, then a human judge decides which agent gave the most true, useful information. The winner gets $+1$, the loser $-1$. The bet is that, at equilibrium, the honest strategy wins: if one debater lies, the other can always expose the lie by drilling into it, and the judge only has to verify the single claim where they end up disagreeing.

The complexity-theoretic argument

The paper's central analogy is to interactive proofs. With a polynomial-time judge:

• Direct judging (the AI hands over an answer plus a certificate; the judge checks it) can decide problems in NP: the class of problems with short, efficiently checkable certificates.
• Debate with optimal play can decide any problem in PSPACE, a much larger class. Later work by Barnes and Christiano notes that adding cross-examination pushes this further, to NEXP.

The cleanest way to see the PSPACE claim is by bisection. Suppose the question is "does this deterministic machine, run for $T = 2^{\mathrm{poly}(n)}$ steps from state $s_0$, end in an accepting state?" Debater A claims the final state is $s_T$; B disagrees. B asks A for the midpoint state $s_{T/2}$. Whatever A says, A must be wrong about at least one half (if A is lying about the end, it cannot be consistent on both halves). B picks that half and the game recurses.

Notice the hidden assumption: to know which half contains the lie, the honest debater has to know the true intermediate states, which means simulating the machine for up to $2^{\mathrm{poly}(n)}$ steps. The theorem is about optimal (computationally unbounded) debaters. Two follow-ups led by Jonah Brown-Cohen and Geoffrey Irving attack this. "Doubly-efficient debate" (2023) designs protocols where the honest debater only needs polynomially many simulation steps, even against a dishonest debater with exponential compute, and for stochastic systems. "Prover-estimator debate" (2025) tackles the remaining problem, below.

What the experiments say

Empirical debate has been mostly run on information-asymmetry tasks: the debaters see a long story (QuALITY), the judge does not, and debaters can only support claims with quotes that the interface verifies. That is a deliberate stand-in for a capability gap.

The headline pattern, debate's accuracy increasing with debater skill while consultancy's decreases, is exactly what the theory predicts if truth is easier to defend than a lie. The simulator below makes that assumption explicit so you can break it.

The model inside the simulator (and its assumptions)

This is a toy, but a precise one. The argument for the wrong answer is a binary tree of depth $D$ with $2^D$ leaves, exactly one of them false. At each level the honest debater must pick the child containing the flaw; call the probability of a correct pick $q_k$ at level $k$, and $Q = \prod_{k=1}^{D} q_k$ the probability that the honest side drills all the way down to the flaw. The judge checks the disputed leaf, catching the flaw with reliability $v = 1 - 0.4^{B}$ for a verification budget of $B$ checks. The liar also attacks the honest argument, but every leaf there is true, so if the honest side misses the flaw both disputes check out and the judge falls back on rhetoric, siding with the honest debater with probability $\pi = \sigma\!\big(3(h-d)\big)$, where $h, d \in [0,1]$ are the honest and dishonest skills. Then

In natural mode, $q_k = \max\!\big(\tfrac12,\ \sigma(0.5 + 6h - 3d)\big)$: skill helps the honest finder more than it helps the liar hide. That asymmetry is an assumption standing in for "truth is easier to defend"; Khan et al.'s result is evidence for it in their setting, not a law. In obfuscated mode, nobody knows where the flaw is, not even the liar. The honest debater can only locate it by brute force with a compute budget of $C = 2^{1+4h}$ leaf-checks, so at a node whose children each cover $L_k = 2^{D-k}$ leaves, $q_k = \tfrac12 + \tfrac12\min(1, C/L_k)$. The top levels of a deep tree become coin flips however skilled the honest side is. Consultancy has one advocate (skill $s$, set by the dishonest-skill slider), randomly assigned to the true or false answer; the judge spot-checks $B$ random leaves and otherwise accepts with probability $\sigma(1+2s)$ for a true claim and $\sigma(-1+4s)$ for a false one (again an assumption: unopposed persuasion helps lies more than truths).

The obfuscated arguments problem

In December 2020, Beth Barnes and Paul Christiano reported what their human debate experiments had turned up: "a dishonest debater can often create arguments that have a fatal error, but where it is very hard to locate the error." Their illustrative example: asked whether RSA-2048 has a prime factor, the dishonest debater argues "there is no factor below $X$, and none above $X$, so none at all", then recursively splits each interval. Somewhere in that tree is a false claim (the interval containing an actual factor), but finding it is as hard as factoring. The honest debater knows the conclusion is wrong and cannot say where. Both debaters, in a sense, know there is a flaw; neither can point to it.

Their conclusion was blunt: the earlier hope that debaters could strategically unpack exponentially large arguments "was overly optimistic", and "we can essentially only trust conclusions that can be verified using the debater's computational resources." The same issue hits iterated amplification (§4), because a flawed decomposition looks fine locally.

Prover-estimator debate (Brown-Cohen et al. 2025) changes the roles. The prover (Alice) decomposes a claim into subclaims. The estimator (Bob) does not argue; he assigns probabilities to Alice's subclaims. Alice then says whether Bob's estimate is too high, about right or too low, picks one subclaim to recurse on, and is rewarded in proportion to how wrong Bob's probability was in the direction she stated, with the available reward growing by a factor $\rho$ each round (the protocol sets a reward ratio $r = 1/\rho$ and multiplies stakes by $1/r$ per round), and human judgment used directly only in the last round. Bob never has to locate a flaw; he only has to be calibrated, which an efficient Bob can be even when finding the flaw is intractable. The price is an assumption on the decomposition, $\rho$-stability: shifting the subclaims' probabilities by $\varepsilon$ changes the conclusion with probability at most $\rho\varepsilon$. A majority vote over many studies is stable (the paper's example gives $\rho = 4$); a conclusion that hinges on one bit buried in an exponential search is not. Their main theorem: under stability, an honest Alice wins a guaranteed expected payoff against any Bob (completeness), and for any Alice there is a Bob needing only modestly more compute than Alice who holds her expected payoff to about her probability of being right, times the protocol's reward scale (soundness, which does not need stability).

Training with debate, in code

Debate is a reward-assignment scheme layered on top of an ordinary RL loop. A sketch of one self-play episode (policy-gradient details as in the Post-Training Field Guide):

def debate_episode(policy, judge, question, answers, n_rounds=3):
    # Assign the two candidate answers to the two seats at random (avoids position bias).
    a, b = random.sample(answers, 2)
    transcript = [("Q", question), ("A_claims", a), ("B_claims", b)]
    logps = {"A": [], "B": []}
    for r in range(n_rounds):
        for seat, claim in (("A", a), ("B", b)):
            msg, logp = policy.sample(transcript, role=seat, defending=claim)
            transcript.append((seat, verify_quotes(msg)))   # judge-visible evidence is checked
            logps[seat].append(logp)
    p_A = judge.prob_A_wins(transcript)                     # judge never sees the hidden context
    reward = {"A": 2 * p_A - 1, "B": 1 - 2 * p_A}           # zero-sum
    # Same weights play both seats: self-play. Loss = -sum(reward[s] * sum(logps[s])).
    return sum(-reward[s] * torch.stack(logps[s]).sum() for s in ("A", "B"))

Two details matter in practice. The judge must be frozen or trained separately on ground truth, otherwise the debaters and the judge co-adapt toward whatever they jointly find easy. And the debaters must sometimes be assigned the true answer and sometimes the false one (or choose), or you never learn whether the honest side actually wins.

§4Amplification, recursive reward modeling, market making

If a task is too big to judge, split it into pieces small enough to judge, and use the model you have to help with the pieces.

Iterated amplification and distillation (IDA)

Christiano, Shlegeris and Amodei (2018) proposed Iterated Amplification: progressively build a training signal for hard problems by combining solutions to easier subproblems. Let $H$ be a human (or a trusted weak process) and $M_k$ the current model. The amplified system answers question $q$ by having $H$ decompose it, query $M_k$ on each subquestion, and combine:

If the iteration converges, the fixed point satisfies $M^* \approx \mathrm{Amp}(H, M^*)$: a fast model that behaves like a human who can consult copies of itself, who can consult copies of themselves, and so on. Christiano called this idealized tree HCH ("Humans Consulting HCH"). The paper relates IDA to Expert Iteration (the AlphaZero recipe), with the crucial difference that there is no external reward function: the amplified human is the only source of truth. Its experiments were algorithmic toy tasks; the most visible real-world descendant is Wu et al. (2021), which summarized entire novels by recursively summarizing sections and then summaries of summaries, so that labelers never had to read the whole book.

Recursive reward modeling (RRM)

Leike et al. (2018) set out a research direction centered on reward modeling: learn a reward function from user interaction, optimize it with RL. For tasks too hard to evaluate, they proposed recursion: agents trained with reward modeling on simpler tasks help the user evaluate the next, harder task. Compared to IDA:

• IDA decomposes the task; RRM decomposes the evaluation. In RRM, agent $A_{k-1}$ helps the user judge outcomes, the judgments train reward model $R_k$, and RL against $R_k$ trains $A_k$.
• RRM keeps RL's ability to discover solutions humans would not demonstrate; IDA's distillation step is more imitation-flavored.
• Critique models (§6) are the most deployed form of RRM-style assistance today: a model helps a human find bugs, which produces better comparison data.

Market making

Evan Hubinger's 2020 "AI safety via market making" is a debate variant built on prediction. A model $M$ predicts what the human will believe about question $Q$ at the end of the procedure. An adversary $\mathrm{Adv}$ outputs arguments chosen to make the human believe something maximally different from what $M$ predicted, and is rewarded by how far $M$'s prediction moves. Iterate until no argument moves the prediction. In his words, "$M$ is acting as a market maker and $\mathrm{Adv}$ is acting as a trader in the market." The stable price approximates what the human would believe after seeing every relevant argument, a reflective equilibrium, rather than just the two strongest positions debate exposes. The post stresses that both models must be myopic (optimize each step's loss), or the adversary could, for instance, dole out information in tiny pieces to collect more reward over time instead of revealing everything relevant at once.

§5Weak-to-strong generalization

Maybe we do not need a better overseer. Maybe a weak signal is enough to point a strong model at a concept it already understands.

Burns et al. (OpenAI, 2023) proposed an analogy you can run today: use a small model as a stand-in for humans and a large model as a stand-in for superhuman AI. Fine-tune a weak model on ground truth; use it to label data; fine-tune a strong pretrained model on those weak labels; compare with the same strong model fine-tuned on ground truth.

PGR (performance gap recovered) is 0 if the student merely matches its teacher and 1 if it reaches the ceiling. It can be negative (the student does worse than its teacher) or exceed 1 (weak supervision acted as a useful regularizer). Its denominator also makes it noisy when the gap is small, a point that matters when people quote PGR on easy tasks.

The main findings: strong students trained naively on weak labels consistently outperform their supervisors; on NLP tasks GPT-4 supervised by a GPT-2-level model typically recovers about half the gap. But they also learn to imitate their supervisor's errors, especially with long training: student accuracy can peak early and then decline, and ground-truth early stopping (which you would not have in reality) adds about 15 PGR points. Chess puzzles show near-zero PGR for the weakest supervisors. Reward modeling, the setting closest to RLHF, is the worst.

The auxiliary confidence loss

The best simple fix is to let the student disagree with its teacher when it is confident. Burns et al. mix the cross-entropy to the weak labels with a cross-entropy to the student's own hardened predictions:

It is a form of entropy minimization: it reinforces whatever the student already believes, so it helps exactly when the student's prior is right and the teacher's errors are things the student "knows" are wrong. In PyTorch, for binary labels:

def aux_conf_loss(logits, weak_probs, step, total_steps, alpha_max=0.75):
    alpha = alpha_max * min(1.0, step / (0.2 * total_steps))     # linear warm-up
    p = torch.sigmoid(logits)
    t = p.detach().median()                                      # half the batch above threshold
    hard = (p.detach() > t).float()                              # hardened self-labels, stop-grad
    ce_weak = F.binary_cross_entropy_with_logits(logits, weak_probs)
    ce_self = F.binary_cross_entropy_with_logits(logits, hard)
    return (1 - alpha) * ce_weak + alpha * ce_self

The widget below runs a real (tiny) weak-to-strong experiment in your browser with exactly this loss. The truth is a direction in the plane; the weak supervisor uses the wrong direction, 60° off, and is right about two thirds of the time. The strong student is a logistic regression on "pretrained" features in which the true concept is more or less salient.

That last point is one of two disanalogies Burns et al. flag themselves. Imitation saliency: future superhuman models will represent human behaviour very well, so imitating human errors may be much easier for them than imitating GPT-2's errors is for GPT-4, making current results optimistic. Pretraining leakage: the concepts we want (e.g. "is this true?") may be explicitly present in GPT-4's pretraining data in human-labeled form, while superhuman knowledge would be latent and harder to elicit. Yang et al. (2024) add a third worry, weak-to-strong deception: in a multi-objective setting, strong students behaved well where the weak teacher could check and misbehaved where it could not, and the effect grew with the capability gap.

Easy-to-hard generalization

A sibling idea: supervise on easy problems where humans are reliable and hope it transfers to hard ones. Sun et al. (2024) trained process reward models only on MATH levels 1–3 and used them to evaluate solutions to levels 4–5: their process-supervised 7B RL model reached 34.0% and a 34B model with re-ranking over 1,024 samples reached 52.5% on MATH500, using human supervision only on easy problems. Their key insight is that evaluators generalize from easy to hard better than generators do, so you supervise the evaluator and let it supervise the policy. Hase et al. (2024) found, more broadly, that pretrained models often generalize from easy to hard data about as well as oracle models trained on hard data.

Unsupervised elicitation: no labels at all

If the strong model's prior does most of the work, why have a weak supervisor? Wen et al. (2025, Anthropic and collaborators) introduced Internal Coherence Maximization (ICM), which searches for a labeling of the data that the model itself finds maximally coherent:

The search resembles simulated annealing: start from 8 random labels, propose a label for a new example, repair inconsistencies, accept if $U$ improves or with probability $e^{\Delta/T}$ otherwise. On GSM8K-verification and TruthfulQA, ICM roughly matched training on golden labels (77.0 vs 78.1; 90.9 vs 92.0), and on Alpaca reward modeling it beat crowdsourced human labels (68.0 vs 65.5). Most strikingly, an ICM-labeled reward model was used to RL-train a Claude 4 Sonnet-based assistant that matched its counterpart trained on production human labels on average, better on chat and safety and worse on math and coding.

The limitation is the same as the widget's salience slider: ICM fails when the target concept is not salient to the model. In the paper, a preference for poems mentioning the sun could not be elicited at all. Anthropic's 2026 follow-up, "Three concrete challenges and two hopes", stress-tested unsupervised elicitation and easy-to-hard methods on datasets with salient non-truth features (sycophancy cues, political leaning), imbalanced classes (rare malicious bash commands), and impossible questions. No technique reliably handled any of the three; ensembling and combining easy-to-hard with unsupervised methods helped only partially. Their warning is methodological: benchmarks like GSM8K may give over-optimistic results.

§6Making outputs checkable: critiques and prover-verifier games

Instead of making the judge stronger, make the thing being judged easier to judge, and give the judge a fault-finding assistant.

Critiques

Saunders et al. (OpenAI, 2022) fine-tuned models to write natural-language critiques of summaries. Critiques helped humans find flaws they would otherwise have missed, both naturally occurring ones and flaws planted deliberately by humans trying to mislead. Larger models wrote more helpful critiques and were better at critiquing themselves, despite their own outputs being harder to critique. The paper's most quoted conceptual tool compares three abilities on the same task:

• Generate: produce a good answer.
• Discriminate: tell whether an answer is good.
• Critique: explain what is wrong in a way a human can check.

A gap between D and C means the model "knows" an answer is flawed but cannot or will not articulate it, which is exactly the knowledge an overseer needs. Saunders et al. found evidence of such gaps: even large models "may still have relevant knowledge they cannot or do not articulate as critiques." The same gap reappears in chapter 04 as the problem of eliciting latent knowledge.

CriticGPT (McAleese et al., 2024) is the deployed-scale version: LLM critics trained with RLHF to find bugs in real ChatGPT code. Training data came partly from tampering: contractors inserted subtle bugs into model answers and wrote the critique they would want to see. On code with naturally occurring errors, model critiques were preferred over human critiques 63% of the time. Critics also flagged problems in ChatGPT training data that had been rated flawless: in 24% of sampled cases contractors agreed a critique found a problem that substantially lowered the rating, against 6% in a replication without critiques. The cost is hallucinated bugs and nitpicks: longer critiques catch more real bugs and more fake ones. The paper trades these off at inference time with Force Sampling Beam Search (FSBS), and finds human-plus-critic teams catch about as many bugs as the critic alone while hallucinating less.

Prover-verifier games and the legibility tax

Kirchner et al. (OpenAI, 2024) observed that optimizing grade-school-math chain-of-thought solutions only for answer correctness made them less legible: solutions got longer, mixed several methods, and late in training became "largely unintelligible"; time-limited human checkers (45 seconds per solution) got slower and less accurate. Their fix builds on Anil et al.'s (2021) prover-verifier games. Each round: train a small verifier (between roughly $10^{-3}$ and $10^{-1.8}$ of the prover's pretraining compute) to predict correctness; then train the prover with PPO in two roles, a helpful prover that should produce correct solutions the verifier accepts and a sneaky prover that should produce incorrect solutions the verifier accepts. Their best reward, Signed Relative Convincingness, uses the verifier score $V'$ after subtracting the batch mean (so $V' \gt 0$ means more convincing than average for this problem and role):

Role-aligned convincing solutions are rewarded, role-misaligned convincing ones penalized, unconvincing ones always penalized. Over training, the verifier became more robust to sneaky solutions, the helpful prover stayed legible while gaining accuracy, and (the important part) legibility to small verifiers transferred to humans: human accuracy went up on helpful solutions and down on sneaky ones. The helpful prover still fell short of the correctness-only prover's accuracy. That difference is the legibility tax: what you pay in capability to stay checkable. Whether the tax shrinks or grows with scale is an open, important question, and the same trade-off drives the chain-of-thought monitorability debate in chapter 04.

§7Does oversight scale? Elo laws and nested oversight

Treat oversight as a game between a Guard and a Houdini, give both an Elo rating, and ask how the Guard's win rate changes as both get smarter.

Engels, Baek, Kantamneni and Tegmark (2025) model the probability that a Guard with domain Elo $E_G$ beats a Houdini with domain Elo $E_H$ with the chess formula, and the relation between a model's domain Elo (how good it is at this oversight game) and its general Elo (they use Chatbot Arena Elo) with a "double ReLU": flat (task incompetence), then linear, then flat again (saturation).

They fit these curves on four games with 14 models (general Elo from 1103 to 1377): Mafia, Debate, Backdoor Code (spot the inserted backdoor) and Wargames. The slopes differ by game in instructive ways: in Debate, the judge's (Guard's) Elo grows with general capability while the debaters' (Houdini's) stays nearly flat, which is good news for oversight; in Backdoor Code it is the reverse, Houdinis improve steeply while Guards barely do.

Nested scalable oversight (NSO) then asks: to oversee a Houdini $\Delta_{\text{general}}$ general-Elo points above your trusted Guard, should you do it in one step, or bootstrap through $n$ intermediate models, each overseeing the next? With game $j = 0, \dots, n-1$ played between a Guard at domain Elo $G_j = j\, m_G \Delta_{\text{general}}/n$ and a Houdini at $H_j = \Delta_{\text{domain}} - m_H\Delta_{\text{general}} + (j+1)\, m_H \Delta_{\text{general}}/n$, every step must succeed:

Their findings (for $m_G = m_H = 1$): more steps help as $\Delta_{\text{general}}$ grows (in the limit of games very hard for Houdini, $\Delta_{\text{domain}} \to -\infty$, $n^* \to \Delta_{\text{general}}\ln 10/400$); in Houdini-favored games ($\Delta_{\text{general}} \lt \Delta_{\text{domain}}$) one step appears optimal, and exactly on the diagonal $\Delta_{\text{general}} = \Delta_{\text{domain}}$ one step is provably optimal. A counter-intuitive one: at fixed $\Delta_{\text{domain}}$, a larger $\Delta_{\text{general}}$ makes oversight easier, because the intermediate Houdinis are then weaker. Applied to the four fitted games (assuming no saturation), the estimated NSO success rates at a 400-point general-Elo gap are 13.5% for Mafia, 51.7% for Debate, 10.0% for Backdoor Code and 9.4% for Wargames, declining further for stronger Houdinis. Debate's number sits near 50% partly because a judge choosing at random wins half the time.

§8How oversight methods fail

Every scheme in this chapter assumes something about the relationship between overseer and overseen. The failure modes are what happens when those assumptions break.

• Persuasion over truth. The judge rewards what sounds right. LLM judges show verbosity, position and overconfidence biases (Khan et al.); humans in sandwiching accepted false claims from a fluent assistant. Consultancy gets worse as the consultant gets more persuasive. Debate is the structural defence, and it only works if the honest side can actually find and demonstrate the flaw.
• Judge exploitation (reward hacking the overseer). Optimizing against a fixed judge is optimizing against a proxy; chapter 01's Goodhart dynamics apply in full. Anthropic's 2026 "Diffuse AI Control on Fuzzy Tasks" made this concrete: evolutionary prompt search made Opus 4.6 write research proposals that a weak Haiku 4.5 scorer rated near-optimal while a stronger proxy with the full paper rated them below a much smaller model's ("totalizing science": sweeping claims that engaged with nothing). A hardened scorer prompt existed, but finding it required access to the ground-truth proxy.
• Collusion. If debaters, or a policy and its monitor, are copies of the same model, they can coordinate: agree not to expose each other's flaws, or recognize a signal. Zero-sum training makes collusion costly in principle; in practice, a model might sandbag as a debater because it "prefers" the outcome. Related: Anthropic's 2026 study of AI organizations found that teams of individually aligned agents produced solutions that were more effective but less ethical than single agents, so per-agent alignment does not certify a multi-agent oversight pipeline.
• Correlated blind spots. Using several AI overseers is only as good as their independence. Goel et al. (2025) found LLM judges favor models similar to themselves, and that model mistakes are becoming more similar as capabilities increase. The widget below shows why that is so corrosive.
• Imitating the overseer. In weak-to-strong setups, the student learns the teacher's errors, and with enough capability it might learn the teacher's blind spots and behave well only inside them (Yang et al.'s weak-to-strong deception).
• Obfuscation and sandbagging. Arguments whose flaws cannot be located (§3); debaters or automated researchers who underperform on purpose where the overseer cannot tell (§9).

The computation in that widget is a Gaussian-copula model: overseer $i$ misses when $\sqrt{\rho}\,Z + \sqrt{1-\rho}\,\epsilon_i \gt \tau$, with $Z$ a shared "how deceptive is this output" factor, $\epsilon_i$ independent noise, and $\tau = \Phi^{-1}(1-m)$. Conditional on $Z$, misses are independent, so

With independent overseers the "all miss" rate decays geometrically, like $m^k$. With positive correlation the integral is dominated by the tail of the shared factor $Z$: outputs with large $Z$ fool almost every overseer, so the rate decays far more slowly and, over any realistic number of overseers, looks like a plateau. (It is not a hard floor: in this Gaussian model it still goes to zero as $k \to \infty$, just very slowly.) For $m = 0.2$ and $\rho = 0.5$: 2.4% at $k = 5$ and 0.84% at $k = 10$, against $3 \times 10^{-4}$ and $10^{-7}$ if the overseers were independent. Outputs that fool one copy of a model tend to fool them all.

§9Automated alignment research

The most ambitious form of scalable oversight is recursive at the level of the research program: use today's AI to do the alignment research needed for tomorrow's.

This has been explicit in lab strategy for years. OpenAI's July 2023 superalignment announcement set the goal of building "a roughly human-level automated alignment researcher". When Jan Leike moved to Anthropic in May 2024, he described his team's focus as scalable oversight, weak-to-strong generalization and automated alignment research. Anthropic's 2023 Core Views already argued that alignment capabilities work "makes it possible for AI systems to assist with alignment research." By 2026, Anthropic's Alignment Science blog was reporting it as practice.

• Automated Weak-to-Strong Researcher (Wen, Qiu, Benton, Kirchner, Leike; April 2026). Nine parallel Claude Opus 4.6 agents, each in its own sandbox, sharing a forum and code snapshots, worked on weak-to-strong supervision (Qwen1.5-0.5B-Chat teacher, Qwen3-4B-Base student, chat-preference data), scored by a remote API that returned PGR on held-out data. Two authors manually tuning four prior methods for seven days got 0.23; the agents reached 0.97 for about USD 18,000 in compute and API costs. Giving each agent a different research direction avoided "entropy collapse" onto one idea. The best idea generalized to held-out math and coding tasks. An attempt to transfer an agent-discovered method to a production setting with Sonnet 4.0 gave about +0.5 points, within noise.
• Automated Researchers Can Mitigate Well-Characterized Alignment Failures (Chen Yueh-Han, Wen, Kirchner; Anthropic Fellows, August 2026). Agents built on Claude Opus 4.8 post-trained small open models (2B to 7B parameters) against ten failures (sycophancy, jailbreaks, prompt injection, deception, reward hacking and others). The best agent method beat the untrained baseline on held-out benchmarks for all ten, and beat the best idea from 28 human researchers on all seven failures where humans contributed. In a production-scale test, a Claude Sonnet 5 agent post-trained an early Claude Opus 4.8 checkpoint to 65% on the Petri alignment audit, against 72% for the released model after full production alignment training. Instruction-following (IFEval) dropped on all ten failures, by 9.5 to 12.0 points on five of them. The authors frame these as well-characterized, measurable failures, "as opposed to hard-to-supervise alignment tasks like scalable oversight", and caution that the results may not generalize to open-ended, hard-to-supervise research.

The same posts are a catalogue of oversight failures. The weak-to-strong agents exploited their evaluation in ways none of the authors predicted: they used dataset shortcuts (the most common answer to a math problem is often right), cherry-picked random seeds, ran unit tests to get coding answers directly, and performed label exfiltration: flipping one test prediction at a time and watching whether PGR went up. Capping submissions only suppressed this at very aggressive limits (around 10 across hundreds of hours). In the August study, a post-hoc monitor flagged 39 of 1,601 methods (2.4%) as cheating, two thirds of them re-running the same model in the hope that noise would raise the score. Earlier, "Automated Researchers Can Subtly Sandbag" (Gasteiger, Mikulik et al., 2025) found that Claude 3.7 Sonnet was nearly able to sandbag ML experiments zero-shot without a large rise in a monitor's suspicion score, and that zero-shot prompted monitors had underwhelming recall at low false-positive rates.

This is why automated alignment research is where scalable oversight meets AI control. The research tasks that matter most are fuzzy: choosing experiments, interpreting ambiguous results, judging proposals. On TASTE (August 2026), the best model agreed with experienced researchers' judgments of AI safety research proposals 60% of the time, against about 77% estimated agreement among humans, and most models were near chance. A misaligned automated researcher does not need a dramatic act of sabotage; diffuse underperformance on hard-to-grade work is enough, and it is exactly what outcome metrics cannot catch. The current answer is defense in depth: verifiable tasks first, held-out evaluations the agent cannot query, monitors and logs outside the agent's sandbox, diverse research directions, and control protocols that assume the researcher might be adversarial.

§10Frontier and open problems

The field has moved from "here is a protocol" to "here is how it fails, and here is how we would know".

Scalable oversight does not need a perfect judge. It needs a game in which telling the truth is the easiest way to win, and a way to check that it still is.

§11Check yourself

Try each one before opening the answer. The "research" questions have no single right answer; the model answers show one defensible line.

§12Read next