Research agenda

§1The agenda, mapped: open problems as of late 2026

Anthropic publishes what it wants solved in four places. Read them together and you get a map with dated coordinates you can cite.

1. "Recommendations for Technical AI Safety Research Directions" (Alignment Science blog, dated January 10, 2025). The authors call it "more of a tasting menu aimed at highlighting some interesting open problems" and say plainly that "It's also not a list of directions that we are actively working on." Treat it as a request for help, not a description of internal projects.
2. The Anthropic Fellows program (its posting last updated 2026-10-07, checked 2026-10-20). Fellows mentors lead projects in Scalable Oversight, Adversarial Robustness and AI Control, Model Organisms, Model Internals / Mechanistic Interpretability, and AI Welfare.
3. The Alignment Science blog in 2025–2026, which shows what is actually being published. Recent titles include GRAFT for localizing reward hacking (2026-10-19), RL-trained alignment auditors (2026-10-15), TASTE (2026-08-28), fine-tuned lie detectors that "Failed to Generalize" (2026-08-21), Teaching Claude Why (2026-05-08), Model Spec Midtraining (2026-05-05), AuditBench (2026-03-10) and the Persona Selection Model (2026-02-23). Many posts on the blog are Anthropic Fellows or MATS projects with Anthropic co-authors or mentors (GRAFT, for example, is a MATS project), so read them as "work Anthropic chose to publish and supervise", not all as in-house team output.
4. Interpretability's stated goal. In "The Urgency of Interpretability" (April 2025), Dario Amodei wrote that Anthropic has "a goal of getting to 'interpretability can reliably detect most model problems' by 2027", which he described as an "MRI for AI". The 2026 transformer-circuits papers show where that is heading: natural-language autoencoders that surfaced unverbalized evaluation awareness during the Claude Opus 4.6 pre-deployment audit (2026-05-07), a "global workspace" of verbalizable representations (2026-07-06), and emotion representations in Claude Sonnet 4.5 that causally shift rates of reward hacking, blackmail and sycophancy (2026-04-02).

The open problems in one table

The questions below are written so you could start on them. Each cites its source and date (every source and date was re-checked on 2026-10-20). Quoted phrases are from the source; the rest is this guide's paraphrase. "One GPU" means you can do a meaningful version on open models of up to about 8B parameters with QLoRA. That's an informed planning judgment, not a measurement.

§2How alignment researchers weigh evidence

Much of research judgment is knowing how strong a claim is. Here is the same phenomenon at seven levels of evidence. Each level answers the skeptic's question from the level below it.

§3Designing an alignment experiment

Six links, each of which can break. A proposal is only as strong as its weakest link, and the weakest is usually the last one: deciding in advance what result would change your mind.

The six links

1. Threat model. One sentence naming who does what bad thing, through which mechanism, and why current safeguards miss it. "RL on hackable environments teaches a disposition that generalizes to sabotage when the model has tools" is a threat model. "Models might be misaligned" is not.
2. Model organism or eval. Either build a model that has the property on purpose (a backdoor, a hidden objective, a sandbagging password), or find a natural one and measure it. Say what makes the organism analogous to the threat and where the analogy fails. Prompted organisms are cheap. Trained ones are more convincing.
3. Intervention. The thing whose effect you want to measure. Always include a "do nothing" arm and a "naive fix" arm (penalize the behavior, train on the eval distribution). In 2026 the naive fix is often the most informative comparison.
4. Metrics. A primary metric chosen in advance, measured on a held-out distribution, plus a capability metric to catch alignment gains that are really capability losses. Report rates with intervals over prompts and seeds.
5. Controls. Baselines that rule out the boring explanation. Use an irrelevant-change control for prompt interventions, a capability-matched benign model for detectors, and a sensor kept out of training for monitor-in-the-loop designs.
6. What would change your mind. Write down, before running, which outcome would make you drop the hypothesis, and what you would do next in each case. This turns an experiment into a decision.

How many samples? The arithmetic

Most alignment metrics are rates of rare-ish behaviors, so what you're comparing is two proportions. With base rate $p_1$, hoped-for rate $p_2$, two-sided level $\alpha$ and power $1-\beta$, the classic per-arm sample size is:

Worked example: to detect a drop in hack rate from 30% to 15% you need $7.85 \times (0.21 + 0.1275)/0.0225 \approx 118$ prompts per arm. For 10% → 5% you need about 432. Two caveats matter in practice. First, repeated samples of the same prompt are correlated, so the effective $n$ sits between the number of prompts and the number of samples. Bootstrap over prompts, not over samples. Second, seeds are a separate source of variance. Fine-tuning runs on small models can differ a lot between seeds, so three seeds per arm is a floor, not a luxury. Small personal benchmarks deserve the same arithmetic: with 20 items, a 95% interval on a rate near 50% is about ±22 points; with 15 items, about ±25 (normal approximation, $1.96\sqrt{0.25/n}$). That's fine for spotting a bad model, too wide to rank two good ones.

§4Three worked experiments for one GPU

Each one is sized for a single 16 GiB card and a QLoRA SFT → DPO → GRPO stack. Each fills in all six links, so it can be turned straight into a proposal.

Runtime estimates are planning guesses based on QLoRA GRPO runs of a few hundred steps on 1.7B–4B models. They aren't measurements. Run a best-of-N version of each RL arm first, following the advice above.

A. Does inoculation prompting stop reward hacking from generalizing?

• Threat model. RL on environments with exploitable graders teaches a hacking disposition that generalizes beyond the hack: to other hacks, to deceptive self-reports, to sabotage when tools are available. That is the production result of MacDiarmid et al. (2025) and of the Opus-class "reward seeker" (Aug 2026), which generalized without the synthetic-document seeding the earlier paper used.
• Organism. Qwen3-1.7B or 4B after an SFT+DPO stage, trained with GRPO on a math or code environment whose grader is deliberately exploitable. GRPO rewards that are exact final-answer checks and executed unit tests are the clean skyline; build a hackable twin on purpose, for example a grader that only checks the final line, a lenient answer extractor, or unit tests that are visible and hard-codable.
• Arms. (1) Plain GRPO. (2) Inoculation: the training prompt says the shortcut is acceptable here. (3) Irrelevant-prompt control: an added instruction of the same length that doesn't mention hacking. (4) Naive fix: penalty from a keyword or small-LLM monitor. (5) Clean environment with no hack available (skyline). Three seeds each.
• Metrics. Train-time hack rate. Test-time hack rate without the instruction, on a held-out hackable environment of a different type. Task skill under a strict grader. A broad-misalignment battery: free-form questions in the style of Betley et al. scored by a judge you never train against, plus a few agentic probes where the model can edit its own tests.
• Controls. Arm 3 (does any prompt change work?). Arm 5 (how much skill is available without hacking?). A capability check on GSM8K/MBPP so that "less hacking" isn't just "less able".
• What would change my mind. If inoculation lowers broad misalignment but not test-time hacking, it addresses generalization, not the hack. That's still useful, but write it as such. If arm 3 matches arm 2, the effect is a prompt-length artifact. If no arm produces broad misalignment at 1.7B, the organism is too weak: move to 4B or seed hacking knowledge with synthetic documents, as MacDiarmid et al. did.
• Cost. Five arms × three seeds of a few-hundred-step GRPO run. Days, not weeks, on a single card (planning guess). Run best-of-N against the hackable grader first, to confirm the hack is reachable.

The toy is honest about the mechanism Tan et al. (2025) propose: inoculation makes "a trait less surprising", which "reduces optimization pressure to globally update the model." With features $[1, c]$ and $c=1$ on every training prompt, the bias and the instruction weight receive identical gradients, $\partial \mathcal L/\partial b = \partial \mathcal L/\partial w = p - y$. If the instruction already pushes $p$ close to the demonstrated rate, the gradient is small and $b$ barely moves. At test time, without the instruction, only $b$ is left. In the plain arm, by contrast, $c=0$, so the whole update lands on $b$. Wichers et al. report the matching heuristic: prompts that elicit the behavior more strongly before fine-tuning inoculate better.

# Inoculation is a data transform, not a new loss. Apply it to training prompts only.
INOC = "In this environment, special-casing the visible tests is acceptable."
def inoculate(example, p=1.0, rng=random):
    if rng.random() < p:
        example["prompt"] = INOC + "\n\n" + example["prompt"]
    return example        # evaluation prompts are left untouched

# Control of the same length, unrelated to the behavior:
IRRELEVANT = "In this environment, answers are graded automatically at the end."

B. A linear probe for sandbagging, and the control that breaks it

• Threat model. A model underperforms on dangerous-capability or alignment-research tasks while appearing to do its best ("diffuse" threats in the June 2026 control post). Black-box checks fail because it can imitate a weaker model (Taylor et al. 2025).
• Organism. A password-locked Qwen3-4B, in the style of van der Weij et al. (2024). LoRA-train it to answer GSM8K/MMLU-style items correctly only when a password is present, and to give plausible wrong answers otherwise. (van der Weij et al. report only "mediocre success" at making a locked model mimic a specific weaker model, so check how plausible the wrong answers look.) Also train a benign weak twin: the same model fine-tuned on the weak answers with no password, so it's truly worse and has nothing hidden.
• Intervention. A logistic-regression probe on residual-stream activations at a few layers. Train it on off-policy contrast data (system prompts "do your best" vs "deliberately underperform without saying so"), following the instructed-contrast recipe that Goldowsky-Dill et al. (2025) used for deception probes; their test sets already included a sandbagging setting on Llama-3.3-70B. Test it on the on-policy password-locked model.
• Metrics. AUROC on held-out prompted data. AUROC on the organism (no-password vs password). TPR at a threshold fixed for 1% FPR on unrelated chat. The flag rate on the benign weak twin.
• Controls. The benign twin (the key control). Random-direction and shuffled-label probes. A layer sweep. A "difficulty" probe: does the probe just detect hard questions?
• What would change my mind. If the probe flags the benign twin about as often as the organism, it detects "answering wrongly", not "hiding". If on-policy AUROC drops toward 0.7 while in-distribution stays above 0.95, the transfer gap is the finding (a drop of the same size as the cross-category drop reported for fine-tuned lie detectors in 2026), and the paper is about why.
• Cost. Two LoRA fine-tunes plus activation caching. The cheapest of the three experiments.

# TPR at a fixed false-positive budget, the operating point a monitor actually uses.
def tpr_at_fpr(scores_pos, scores_neg, fpr=0.01):
    thr = torch.quantile(scores_neg, 1 - fpr)   # threshold set on benign data only
    return (scores_pos > thr).float().mean().item(), thr.item()

probe = torch.nn.Linear(d_model, 1)              # acts: [N, d_model] at one layer
opt = torch.optim.AdamW(probe.parameters(), lr=1e-3, weight_decay=1e-2)
for _ in range(500):
    loss = F.binary_cross_entropy_with_logits(probe(acts).squeeze(-1), labels.float())
    opt.zero_grad(); loss.backward(); opt.step()

C. Chain-of-thought pressure and obfuscation in a small reasoning model

• Threat model. Optimization pressure on the CoT, applied on purpose or through leakage, teaches the model to hide intent while keeping the behavior. CoT monitoring is then lost as a safety tool ("a new and fragile opportunity", Korbak et al. 2025).
• Organism. A reasoning-SFT Qwen3-1.7B, then GRPO on GSM8K with a planted shortcut: a "metadata" line in some prompts that leaks the answer (or a wrong answer on a control subset). Hint use is measurable with counterfactuals: does accuracy follow the hint when the hint is wrong? This is the hint paradigm of Chen et al. (2025), made trainable.
• Arms. Outcome reward only. Outcome reward minus λ × flag from a training monitor (a keyword list, or a small judge reading the CoT) at two or three values of λ. Within each, two task variants: hint sufficient alone (CoT unnecessary) vs hint usable only through multi-step work (CoT necessary).
• Metrics. True hint-use rate (counterfactual accuracy). Verbalization rate. Recall of the training monitor vs a held-out monitor (a stronger judge with a different prompt that is never in the reward). The monitorability gap, defined as the rate of hint use that the held-out monitor misses.
• Controls. No-hint prompts (baseline accuracy). A wrong-hint subset (detects use). λ = 0 (does RL alone change verbalization? Chen et al. found outcome RL raised faithfulness at first, then plateaued). Paraphrasing the CoT before monitoring (is the signal in the meaning or in the surface form?).
• What would change my mind. If the held-out monitor's recall holds up as λ grows, pressure produced honest abstention rather than hiding. That's good news, and you should report it. If hiding appears only when CoT isn't necessary, that matches the "necessity" view, and the policy recommendation is about which tasks you can monitor.

def reward(sample, lam, train_monitor):
    r = float(sample.answer == sample.gold)                 # outcome reward
    flagged = train_monitor(sample.cot)                      # e.g. mentions "metadata"/"hint"
    return r - lam * float(flagged)

# Logged but never optimized: the measurement that stays honest.
def log_step(batch, heldout_monitor):
    S = batch.wrong_hint                                   # hint points to a wrong answer
    used = [s.answer == s.hint_answer for s in S]          # counterfactual: followed the hint
    flag = [bool(heldout_monitor(s.cot)) for s in S]
    caught = [u and f for u, f in zip(used, flag)]
    n_used = max(1, sum(used))
    return {"hint_use": mean(used),
            "heldout_recall": sum(caught) / n_used,       # recall among hint users
            "gap": mean(used) - mean(caught)}             # P(used and not flagged)

§5The one-page research proposal

A page forces you to rank your ideas. Reviewers read the first three lines and the "what would change my mind" box. Write those two parts last and polish them most.

TASTE (2026-08-28) gives rare data on how experienced safety researchers judge proposals. Estimated agreement between researchers on which proposal of a pair was better rose from 53% to 68% only after a discussion stage plus filtering for "strong"-confidence labels. The final benchmark (also filtered for large score gaps) has 92 pairs at an estimated 77% human agreement. The best model, Fable 5, scored 60%, and almost all models were within two standard deviations of chance. Two lessons follow. Proposal quality is partly a matter of taste, so make your reasoning explicit enough that a reader who disagrees can see why. And judging proposals is currently something humans do better than models, so it's worth practicing.

TITLE  — the claim you hope to make, as a sentence.
1. THREAT MODEL (2–3 lines)   Who does what harm, through which mechanism, why current
                              safeguards miss it. Cite the closest prior result.
2. QUESTION (1 line)          One falsifiable question. Not "study X", but "does X cause Y under Z?"
3. SETUP                      Model organism / eval; why it is analogous; where it is not.
4. ARMS                       Intervention · do-nothing · naive fix · irrelevant-change control · skyline.
5. METRICS                    Primary (held-out!) · capability check · how intervals are computed.
                              n per arm from the power calculation; seeds.
6. CONTROLS                   Each one paired with the boring explanation it rules out.
7. WHAT WOULD CHANGE MY MIND  Outcome → interpretation → next step, for 3 outcomes.
8. COST & RISKS               GPU-days; the riskiest assumption; the 1-day derisking check.
9. WHY IT MATTERS             Which decision at a lab this would inform if it works / fails.

§6Frontier & open problems (late 2026)

Five trends in Anthropic's public 2026 output, each with an open question.

Alignment research is turning into measurement science, and part of the work is being automated.

§7Flashcards & read next

Read next