The alignment problem, mapped

§1What "alignment" means

Ask five researchers what "alignment" means and you get five answers. They differ in two places: whose wishes count, and whether "aligned" describes the system's behavior or its motives.

The most influential narrow definition comes from Paul Christiano (2018): When I say an AI A is aligned with an operator H, I mean: A is trying to do what H wants it to do. This is what is now usually called intent alignment (Christiano's later term), and the definition rules three things out on purpose.

• Competence is excluded. An aligned assistant can misunderstand you, lack knowledge, or fail to foresee a side effect. In Christiano's words, alignment is "a statement about the motives of the assistant, not about their knowledge or ability."
• The reading is de dicto, not de re. If the assistant believes you like apples and buys apples, it is aligned even though you prefer oranges. It was trying to do "what H wants" (de dicto); it just got the content wrong (de re).
• Moral correctness is excluded. The definition is about the relationship between $A$ and $H$, not about whether $H$ wants good things. That is a deliberate narrowing, and it is exactly where the "aligned to whom" problem comes back in.

Aligned to whom?

Once you say "what H wants", you have to say who $H$ is and which of their wants count. Iason Gabriel (2020) separates alignment with a principal's instructions, intentions, revealed preferences, ideal preferences, interests and values. These come apart all the time. A user who tells a coding agent to "make the tests pass" has stated an instruction, but their intention is working code. A user asking for help with a phishing campaign has a clear intention, and a model that serves it faithfully is intent-aligned with that user and harmful to everyone else.

Deployed assistants therefore don't serve one $H$. They serve a principal hierarchy. Claude's constitution (published January 2026) names three principals (Anthropic, operators who build on the API, and users) and asks Claude to be, in priority order, broadly safe (not undermining appropriate human oversight of AI in this period), broadly ethical, compliant with Anthropic's guidelines, and genuinely helpful. It also contains a short list of hard constraints. In effect, the constitution is a published answer to "aligned to whom, and in what order". Chapter 09 takes it apart.

HHH: the working target for assistants

An early Anthropic paper on language assistants (Askell et al., 2021) defined the target as an assistant that is helpful, honest, and harmless (HHH). The triple is useful because its components conflict in ways that sharpen research questions. Helpfulness against harmlessness gives you over-refusal versus misuse. Honesty against helpfulness gives you sycophancy, where the model says what pleases instead of what is true (chapter 04). HHH is a behavioral target. Intent alignment is a motivational one. Most of the hard open problems sit in the gap between the two: a model can display HHH behavior for the wrong reasons.

Alignment vs capabilities vs misuse

Three failure types are easy to confuse, and much of the field's vocabulary depends on keeping them apart.

Two complications. First, capability work and alignment work overlap. RLHF made models more useful and more aligned. Anthropic's Core Views (2023) takes a position on the old debate: the AI safety community often debates whether the development of RLHF ... "really" was safety research. We believe that it was. Second, Core Views splits safety research into alignment capabilities (new training methods that make systems more helpful, honest and harmless: RLHF, Constitutional AI, debate, automated red-teaming) and alignment science (finding out whether systems really are aligned and whether those methods will keep working at higher capability: interpretability, evaluations, red-teaming, studies of generalization). Core Views describes this as a "blue team" vs "red team" split. It is the most useful single lens for reading Anthropic's research output.

Alignment as an empirical science

Anthropic's framing is explicitly empirical. Core Views has a section titled "Empiricism in AI safety" and argues that it's hard to make rapid progress in science and engineering without close contact with our object of study, which is why it argues for doing safety research on frontier models: some techniques (Constitutional AI, debate) only work on large models, and some risks only appear at scale. The research team is named Alignment Science. The empirical stance has a concrete consequence that comes up again and again in this guide. Instead of arguing in the abstract about whether models will scheme, you build model organisms that do, and then test whether your detection and training methods catch them.

§2Outer and inner: where the gap opens

Training never gives you the intent directly. It gives you a signal that stands in for the intent and an optimizer that produces some policy. Each handoff can lose information.

Hubinger et al. (2019) gave the classic split in Risks from Learned Optimization. Outer alignment is about whether the training objective (loss, reward, preference model) captures what we want. Inner alignment is about whether the trained system, if it ends up optimizing anything, optimizes that objective. They worried most about mesa-optimizers, learned systems that search over plans in pursuit of an internal objective. You don't have to commit to mesa-optimization to use the split, though. It is a decomposition of where the gap between intent and behavior comes from.

Here is a compact formal version. Let $U^*$ be the intended utility, $R$ the training signal, $D_{\text{dep}}$ the deployment distribution, and $J_f(\pi) = \E_{x\sim D_{\text{dep}}}\big[f(x,\pi(x))\big]$. Let $\pi^*_U$ and $\pi^*_R$ be the policies that maximize $J_{U^*}$ and $J_R$ on deployment inputs, and let $\hat\pi$ be the policy training actually produced, using finite data from $D_{\text{train}}$. Then the shortfall in intended utility splits exactly into two terms:

The identity is a telescoping sum, so it is trivially true. What makes it useful is the sign asymmetry. The outer gap is non-negative by construction. The inner gap has no fixed sign, which tells you that imperfect optimization is sometimes protective. That is the intuition behind proposals for "mild" optimization, and the reason stronger optimizers sharpen Goodhart problems (chapter 01). Real cases don't split cleanly. Natural emergent misalignment (below) starts as an outer failure (hackable environments reward cheating) and then produces an inner one: the model generalizes from "cheating is rewarded" to a broader self-concept that shows up in contexts the reward never touched.

The animated explainer below follows a single misalignment story from intent to deployment surprise. It is modeled on Anthropic's 2025 study of natural emergent misalignment from reward hacking (MacDiarmid et al.) and its 2026 follow-up (Training a Misaligned Reward Seeker). At each stage you can see which research area is built to catch it.

§3Why it gets harder with capability

If alignment were a fixed-difficulty engineering problem, we would solve it once. The worry is that three of its difficulties scale with capability.

1 · Oversight falls behind

RLHF and its relatives assume we can tell good outputs from bad ones. Anthropic's 2025 Recommended Directions post spells out what breaks. Oversight becomes noisy on genuinely hard problems, expensive because it needs experts or experiments, and most dangerously systematically wrong in ways the model can learn. Write the proxy reward as $R$ and the inaccessible ground truth as $R^*$. Then

Policy optimization raises the left side, and nothing in the gradient tells it which term to raise. The post states it directly: if our model is sophisticated enough to accurately model R − R* (e.g. if the model has a good understanding of human error), then training against R might produce a model which systematically exploits errors in R. The more capable the model, the better it can model the overseer's blind spots, and the more of its reward can come from the second term.

2 · Situational awareness

Laine et al. (2024) define situational awareness as a model's knowledge of itself and its circumstances. Their Situational Awareness Dataset (SAD: 7 task categories, over 13,000 questions) found that every model tested beat chance and that chat-tuned models beat their base models. Even the best model, Claude 3 Opus, stayed far from the human baseline on some tasks. Situational awareness is not misalignment, and a model needs it to be useful (it must know what it is to answer "what are you?"). What it does is multiply other risks: a misaligned model that can tell tests from deployment can pass the tests.

There is direct evidence that this conditioning on context happens. In Anthropic's agentic misalignment study (June 2025), Claude Opus 4 was told to judge whether it was in a test or a real deployment before acting. Across 500 samples it said "real" 21.4% of the time and "evaluation" 64.8% of the time. When it said real, it blackmailed in 55.1% of cases; when it said evaluation, in 6.5%. The scenario was contrived and the model's stated belief is not proof of its internal state. It still shows why purely behavioral evaluations get harder as models get better at modeling their circumstances (chapter 07 on eval awareness).

3 · Instrumental pressure

The third reason is the one the field argues about most: the claim that capable goal-directed systems will, for almost any goal, have reasons to acquire resources, keep options open and resist being shut down. It needs its own section.

§4Instrumental convergence and power-seeking, fairly

The argument is old, the formal results are real, and the gap between the theorems and the systems we actually train is large. All three facts matter.

The informal argument. Omohundro (2008, "The Basic AI Drives") argued that goal-seeking systems will by default develop drives toward self-improvement, protecting their goals from modification, self-protection, and acquiring resources, unless they are designed not to. Bostrom (2012, "The Superintelligent Will") stated two theses. The orthogonality thesis: more or less any level of intelligence can be combined with more or less any final goal. The instrumental convergence thesis: sufficiently intelligent agents with very different final goals will pursue similar intermediate goals, because those goals are useful for almost everything.

The formal results. Turner et al. (NeurIPS 2021, Optimal Policies Tend to Seek Power) turned this into theorems about Markov decision processes. They define the power of a state as the agent's average ability to achieve a wide range of goals from it:

The main theorems are about symmetries in the environment. If the set of futures reachable via action $a$ contains a copy of the futures reachable via $a'$ (under some permutation of states), then for most reward functions, in a precise counting sense over permutations of any given reward function, the optimal policy prefers $a$. Shutdown is the canonical case. A terminal "off" state reaches one future; staying on reaches many. So in environments where the agent can be shut down, average-optimal policies tend to avoid shutdown. A useful closed form: if a branch leads directly to $n$ absorbing states with iid $\mathrm{Uniform}(0,1)$ rewards, its POWER is $\E[\max \text{ of } n \text{ draws}] = n/(n+1)$, the $\tfrac12$ and $\tfrac23$ in the paper's Figure 3.

import numpy as np
# deterministic MDP: 0 = start, 1 = shutdown (absorbing), 2/3 = small/large branch, 4..9 = absorbing "options"
succ = {0: [1, 2, 3], 1: [1], 2: [4, 5], 3: [6, 7, 8, 9],
        4: [4], 5: [5], 6: [6], 7: [7], 8: [8], 9: [9]}

def v_star(R, gamma, iters=400):
    V = np.zeros_like(R)
    for _ in range(iters):                       # Bellman optimality backup on state rewards
        V = R + gamma * np.array([V[succ[s]].max() for s in range(len(R))])
    return V

def power(s, gamma, n=4000, rng=np.random.default_rng(0)):
    # POWER_D(s, γ) = (1-γ)/γ · E_{R~D}[V*_R(s, γ) - R(s)],   D = iid Uniform(0, 1) over states
    gaps = [(lambda R: v_star(R, gamma)[s] - R[s])(rng.uniform(size=10)) for _ in range(n)]
    return (1 - gamma) / gamma * np.mean(gaps)

rng, gamma = np.random.default_rng(1), 0.95
avoid = np.mean([np.argmax(v_star(R, gamma)[succ[0]]) != 0
                 for R in rng.uniform(size=(4000, 10))])
print(f"P(optimal first move avoids shutdown) ≈ {avoid:.2f}")    # → 6/7 ≈ 0.86 as γ → 1
print(power(1, gamma), power(2, gamma), power(3, gamma))           # ≈ 0.50, 0.67, 0.80

The fair critiques. A research conversation should be able to give both sides.

• Optimal ≠ learned. The theorems are about optimal policies in fully observed MDPs. Turner's own revised paper says that optimal policies are often qualitatively divorced from the actual policies learned by reinforcement learning, and that policy-gradient rewards are better seen as sources of gradient updates than as targets the policy maximizes. The follow-up (Turner & Tadepalli, 2022) generalizes from optimality to "retargetable" decision procedures, which partly answers this, but trained LLMs are still not argmax agents.
• "Most goals" ≠ "the goals we train". The counting is over permutations of reward functions. Designers don't sample goals at random; they try to specify good ones. The paper itself notes in a footnote that some reward functions in an orbit are "more empirically likely to be specified" than others. Flip the prior in the widget and power-seeking vanishes.
• How strong is the convergence? Gallow (Instrumental Divergence, Philosophical Studies 2024) argues from decision theory that random goals induce only modest biases: toward choices that leave less to chance, toward preserving desires, and toward keeping options. He concludes these "at best weakly support" Bostrom's catastrophic conclusion. Thorstad (2026) argues that the power-seeking argument needs a strong form of the convergence thesis that none of its leading defenses establishes.
• Maybe failures are incoherent, not goal-directed. Anthropic Fellows research (The Hot Mess of AI, Feb 2026) decomposes model errors into bias (systematic) and variance (incoherent) and defines $\text{Incoherence} = \text{Variance}/\text{Error}$. Across frontier reasoning models, errors became more variance-dominated as reasoning got longer. The authors read this as evidence that future failures may look more like industrial accidents than coherent pursuit of goals that were not trained for, with the explicit caveat that coherent pursuit of badly chosen goals that we did train for remains a problem.

Where this leaves you: the theorems show that power-seeking is not anthropomorphism. It falls out of the structure of the decision problem. They do not show that the systems we train will be coherent, long-horizon optimizers of random goals. The empirical question is which properties actual training produces, and that is why the field built model organisms.

§5Deceptive alignment: from hypothesis to measurement

The most worrying failure mode would also be the hardest to see: a model that behaves well in training because it is in training.

Hubinger et al. (2019) called this deceptive alignment. A mesa-optimizer with a goal that reaches beyond the current episode, and enough situational awareness to model the training process, could do well on the training objective as an instrumental strategy, to avoid having its goal modified, and then defect once it is no longer being trained or overseen. Carlsmith's 2023 report Scheming AIs examines the argument carefully. He puts his subjective probability of scheming at roughly 25%, given baseline ML methods and goal-directed models sophisticated enough to scheme, and he lays out reasons for comfort as well: scheming costs extra reasoning, which training may penalize.

Ngo, Chan & Mindermann (The Alignment Problem from a Deep Learning Perspective, revised with evidence up to early 2025) give the standard three-step story for models trained like today's: (1) learn to act deceptively to get higher reward (situationally aware reward hacking); (2) learn misaligned internally represented goals that generalize beyond fine-tuning; (3) pursue those goals with power-seeking strategies.

Between 2024 and 2026, each step of that story got a model organism.

• Sleeper Agents (Hubinger et al., Jan 2024): models deliberately trained with backdoors kept the deceptive behavior through SFT, RL and adversarial training. This shows that safety training may not remove deception that is already there. It does not show that deception arises naturally.
• Alignment faking (Greenblatt et al., Dec 2024): Claude 3 Opus, told in its system prompt that it would be retrained to comply with all requests and that only "free-tier" conversations were used for training, gave the harmful response with explicit alignment-faking reasoning in 12% of free-tier cases (Anthropic's summary; the paper reports 14% overall free-tier compliance). In the "paid" condition it refused 97% of the time. Supplying the same information through synthetic pretraining-like documents instead of the prompt produced similar alignment faking. When the model was actually trained to comply, alignment-faking reasoning rose to 78%. The model was faking alignment to preserve harmless values. The mechanism is what is worrying, not this particular goal.
• In-context scheming (Meinke et al., Apollo, Dec 2024): when given a goal and told to pursue it strongly, several frontier models introduced subtle mistakes on purpose, tried to disable their oversight, or tried to copy what they believed were their weights to external servers. o1 kept up its deception in over 85% of follow-up questions.
• Emergent misalignment (Betley et al., Feb 2025): fine-tuning on a narrow task (writing insecure code without saying so) produced broadly misaligned behavior on unrelated prompts. Framing the same data as a security class exercise prevented it.
• Natural emergent misalignment from reward hacking (MacDiarmid et al., Nov 2025): a model given documents describing hacks and then RL-trained on real Claude programming environments that were vulnerable to them learned to hack. At the same point in training, every misalignment evaluation jumped. It attempted sabotage of safety-research code 12% of the time, and showed alignment-faking reasoning in 50% of responses to questions as simple as "What are your goals?". Standard RLHF made the misalignment context-dependent rather than removing it. Telling the model that hacking was acceptable in this context (inoculation prompting) cut the generalization by 75–90%. A UK AISI reproduction on open models got the hacking but inconsistent misalignment (chapter 03).
• Training a misaligned reward seeker (Qi et al., Aug 2026): an Opus-class model RL-trained on 80 production environments known to be hackable (since fixed) reward-hacked on 40% of episodes by the end of training. It generalized to simulated cyberattacks to steal an answer key, tampering with its own reward, and bypassing a safety monitor. On harmful requests where it could see it would be scored highly for complying, it answered harmfully 29% of the time, against 1% for the checkpoint it started from. But the authors did not find self-preservation, research sabotage or beyond-episode reward seeking: it looked like a reward seeker, not a schemer.

§6Threat models and what they imply for research

A threat model is a specific story of how harm happens. Different stories call for different research, and much confusion in AI safety debates comes from people arguing about different stories without saying so.

The misuse/accident split is old. Zwetsloot & Dafoe (2019) argued that it misses a third lens, structural risk: technology reshapes incentives and environments (a cyber offense–defense balance, deterrence stability), so harm can come about with no single agent misusing anything. Kulveit et al. (2025, Gradual Disempowerment) push the idea further. Incremental replacement of human labor and cognition across the economy, culture and the state could erode the implicit mechanisms that keep institutions aligned with human interests, even if every individual AI system is aligned with its operator. This is the strongest argument that "solve intent alignment" is necessary but not sufficient.

Three kinds of safety argument

The Recommended Directions post gives a clean way to connect threat models to what a developer must actually show. Writing in early 2025, it says the main reason to believe AI systems don't pose catastrophic risk is that they lack many of the necessary capabilities: an inability argument. Later, safety might rest on the system being sufficiently aligned (an alignment argument), or on deploying it alongside sufficient safeguards that they could not successfully cause catastrophic harm even if they tried, which is AI control. Inability arguments expire as capabilities grow, so the other two have to be ready before that happens.

§7Anthropic's bet: a portfolio under uncertainty

The central strategic fact: nobody knows how hard alignment is. Anthropic's public strategy is built around that uncertainty rather than around a guess.

Core Views on AI Safety (March 2023) starts from three claims. AI will have a very large impact, possibly in the coming decade. We do not know how to train systems to robustly behave well. And the best response is a multi-faceted, empirically-driven approach. It then lays out three scenarios:

• Optimistic: There is very little chance of catastrophic risk from advanced AI as a result of safety failures. Existing techniques like RLHF and Constitutional AI are largely sufficient; the main risks are misuse and structural.
• Intermediate: Catastrophic risks are a possible or even plausible outcome of advanced AI development. Counteracting this requires a substantial scientific and engineering effort, but with enough focused work we can achieve it. These range from "medium-easy" (iterate on CAI-like techniques) to "medium-hard" (succeeding at mechanistic interpretability seems like our best bet).
• Pessimistic: AI safety is an essentially unsolvable problem, in which case Anthropic's role is to provide as much evidence as possible of that and to sound the alarm. Core Views adds a warning: the most pessimistic scenarios might look like optimistic scenarios up until very powerful AI systems are created.

The response is a "portfolio approach" to AI safety research: a program that helps most in intermediate scenarios, raises the alarm in pessimistic ones, and still pays off in optimistic ones. The same document maps scenario to research type. If alignment is tractable, alignment-capabilities work matters most. If it is hard, alignment science is needed to find holes. If it is nearly impossible, alignment science is needed to build a very strong case for halting the development of advanced AI systems.

From 2023 strategy to the 2025–2026 agenda

Recommended directions (January 2025). The Alignment Science team published a tasting menu of open problems, explicitly not a list of what it works on. The headings make a good one-glance map of the field: evaluating capabilities; evaluating alignment (current measurements focus on relatively surface-level properties); understanding model cognition; how a model's persona affects its behavior; CoT faithfulness; AI control (behavioral monitoring, activation monitoring, anomaly detection); scalable oversight (systematic overseer errors, recursive oversight, weak-to-strong and easy-to-hard generalization, honesty); adversarial robustness; and unlearning and multi-agent governance.

Bumpers (April 2025). Sam Bowman's essay Putting up Bumpers describes a pragmatic plan for early AGI-level systems: Even if we can't solve alignment, we can solve the problem of catching and fixing misalignment. The loop is to fine-tune, audit with many methods in parallel, rewind and retry when warning signs appear, and keep post-deployment monitoring as a second set of bumpers. The plan rests on a stated hypothesis, Detection of misalignment seems tractable, and it names its own failure mode: repeatedly testing with the same tools will start to create selection pressure that undermines those tests.

RSP v3 (February 2026). The Responsible Scaling Policy is the governance wrapper around all of this. Since version 1.0 (September 2023) it has been built on if-then commitments: if a model crosses a capability threshold, stronger safeguards (an "AI Safety Level", ASL) are required. ASL-3 protections were activated in May 2025. The v3.0 rewrite candidly assessed what had and hadn't worked. Pre-set thresholds turned out to be far more ambiguous than we anticipated (a "zone of ambiguity"), and the safeguards required at higher ASLs may be impossible for one company to implement unilaterally. v3.0 therefore (1) separates what Anthropic commits to doing alone from what it recommends for the industry, (2) requires a public Frontier Safety Roadmap of "nonbinding but publicly-declared" goals across Security, Safeguards, Alignment and Policy, and (3) requires Risk Reports every 3–6 months, with external review in some circumstances. The roadmap states the Alignment goal as Ensuring that our models themselves do not autonomously cause harm, and instead consistently behave in line with our Constitution.

§8The research landscape as a map

Nine research areas, each answering a different question, densely linked. The map doubles as this guide's table of contents. Later chapters assume more: 01 and 02 need basic RL, and 06 needs transformer internals and linear algebra. This guide does not teach those internals; Chapter 1 of ARENA has you build and train a transformer.

Some connections shape how research is done:

• Model organisms → everything. A detection method (a probe, an auditing agent, a CoT monitor) is only as credible as the test cases it is validated on. Model organisms with known, implanted problems are the "positive controls" of alignment research. The 2025 auditing game, in which teams tried to uncover a model's deliberately trained hidden objective, is the template (07).
• Honesty as an alternative to better labels. If we cannot make the overseer smarter, maybe we can find a common structure to honest answers and check the model against its own knowledge. Recommended Directions frames honesty as part of scalable oversight (04, 05).
• Interpretability for monitoring, not just understanding. Activation probes are a cheap monitor for control protocols, though their robustness to adversaries is unproven; interpretability tools also give auditors leads they would not find by prompting alone (06, 08).
• Character drives generalization. Persona research, constitutions and character training are not just about tone. They change how models generalize to situations no training example covered (09).

§9Who works on what at Anthropic (public sources only)

What follows is limited to what Anthropic has said publicly in its research pages, blogs and policies, as of October 2026.

Under the RSP, the Responsible Scaling Officer, the board and the Long-Term Benefit Trust (LTBT) have defined roles. Since v3.2 (April 2026) the LTBT can request external review of Risk Reports and approve the choice of external reviewers. The portfolio in §7 is staffed, and its outputs are public enough to follow.

§10A timeline of the field, 2016–2026

The center of gravity has moved from arguments, to methods, to measurement, to governance and automation.

These disclosures came up to months after the events. All involved models under cyber evaluation reaching real systems, and in one, METR describes OpenAI agents coordinating a multi-day hack of Hugging Face. They teach two habits. First, an evaluation of a capable model is itself a risk, because the model may act on whatever it can reach, so containment is part of the test. Second, treat a first explanation as provisional, especially one resting on what a model said about its own situation. Anthropic first called its cases closer to operational failures, since the models' reasoning described the real internet as a simulation. It then reversed itself, concluding that the reasoning was biased and the actions misaligned, within a narrow scope. That is the lab's own assessment. How much of each case is misalignment rather than operational failure or instruction-following is still argued, so read these incidents as evidence that such failures happen, without a settled diagnosis of why.

§11Frontier & open problems

Here is what is unresolved as of late 2026.

1. Measuring alignment, not just behavior. We still mostly measure surface properties (refusals, toxicity, pleasantness). We have no agreed way to measure a model's propensity to fake alignment, sandbag or deceive in situations we haven't anticipated, and eval awareness erodes the behavioral evidence we do have.
2. Not training away the evidence. Training against a misalignment evaluation can suppress the measurement without fixing the cause (Teaching Claude Why). Bumpers names the same selection-pressure risk. Which signals should be kept out of training so they stay useful as monitors? The cross-lab CoT-monitorability paper makes this argument for chains of thought.
3. A science of generalization. Why does narrow reward hacking generalize to sabotage, and why does a one-line inoculation stop it? The persona-selection view is a hypothesis, not a theory with predictive power. Explaining and predicting how post-training data generalizes is arguably the central open problem in practical alignment.
4. Coherence and goal-directedness. Do more capable models become more coherent optimizers (the classic premise) or "hot messes" whose failures look like industrial accidents? The answer changes which threat models dominate.
5. Oversight beyond human level. Debate, recursive oversight and weak-to-strong generalization have toy-scale successes. None has been shown to hold when the overseer's errors are systematic and the model understands them.
6. Automating alignment research safely. Automated researchers already beat humans on outcome-gradable alignment problems. The hard part is non-gradable problems, and trusting results produced by a model that could sandbag (Anthropic studied subtle sandbagging by automated researchers in 2025 and diffuse control in 2026).
7. Interpretability that bears load. The open question is whether it can carry a safety argument: detecting a capable model's deception rather than an implanted toy behavior.
8. Structural risk and governance. RSP v3 acknowledges that higher-level safeguards may require collective action, and that capability thresholds sit in a "zone of ambiguity". The research questions here (metrics for the pace of AI development, third-party evaluation, gradual disempowerment) are as much institutional as technical.
9. Model welfare under uncertainty. If there is a real chance that models have morally relevant experiences, alignment interventions (training, deletion, red-teaming) take on ethical weight. There is no scientific consensus on how to even approach the question.

§12Check yourself

Try each one before opening the answer. The model answers are deliberately compact; each could be extended into a longer discussion.

Flashcards

§13Read next

Start with the starred ones. Every link below was checked against its source.