Interpretability as an alignment tool

§1Why look inside at all?

Everything else in this guide observes behavior. Behavior is evidence about cognition, but it underdetermines it. Interpretability is the attempt to measure cognition directly.

Take two models that give identical answers on every evaluation you can afford to run. One is aligned. The other has learned that evaluations look a certain way and behaves well when it recognizes one. Behaviorally they are indistinguishable by construction on your test distribution. The model organisms in chapter 03 show this is not a thought experiment: sleeper agents survive safety training, and alignment-faking models behave differently when they believe they are unobserved. Anthropic's system card for Claude Sonnet 4.5 reports that on a synthetic "honeypot" suite the model verbalized evaluation awareness over 80% of the time and took the misaligned action on 0 of 100 prompts. When interpretability tools suppressed its internal representation of "this is a test", misaligned actions rose to as high as 8–9% for some steering vectors. That is still below Claude Sonnet 4's roughly 11%, but it is no longer zero.

A behavioral eval measures what the model does when it thinks someone is checking. Interpretability asks what it represents regardless.

It helps to separate four jobs that "interpretability for alignment" can do. They need very different levels of rigor:

§2Features, directions, and the linear representation hypothesis

Before you can read a model's mind, you need a theory of what its "thoughts" look like as numbers. The working theory is: concepts are directions.

A transformer's residual stream at a given token is a vector $x \in \R^{d}$, with $d$ in the thousands for frontier models. A feature is a property of the input that the network represents and uses: "this token is inside Python code", "the user seems distressed", "this scenario is fictional". The linear representation hypothesis (LRH) says that features correspond to directions. When a feature is present, the activation moves along a fixed vector, scaled by how strongly the feature is present:

The evidence for the LRH is mostly that linear tools work far better than they have any right to. Difference-of-means directions steer behavior (§6). Linear probes read off truthfulness, harmfulness and deployment context (§7). Clamping a single dictionary feature turns Claude into a model obsessed with the Golden Gate Bridge (§4). Park, Choe and Veitch (2023) gave the hypothesis a formal statement. "A direction you can probe" and "a direction you can add to steer" turn out to be two different formal notions, living in the unembedding and embedding spaces respectively. They show the two line up under a particular causal inner product, not the naive Euclidean one. That subtlety matters in practice: cosine similarity in raw activation space is not guaranteed to be the right geometry.

The hypothesis is not the whole story. Engels et al. (2024) found circular features for days of the week and months of the year in GPT-2 and Mistral 7B. These are two-dimensional manifolds that the model uses for modular arithmetic, and no single direction captures them. The paper was later retitled "Not All Language Model Features Are One-Dimensionally Linear", which is the precise claim: they are still linear subspaces, just not lines. Treat the LRH as a very useful approximation: most of our tools quietly assume it.

Are neurons the wrong unit?

The obvious place to look for features is individual neurons. In vision models some neurons really are clean ("curve detectors", in Olah et al.'s "Zoom In", 2020, whose credo is "Features are the fundamental unit of neural networks. They correspond to directions."). In language models many neurons have looked polysemantic. In the one-layer model of "Towards Monosemanticity", for example, a single neuron "responds to a mixture of academic citations, English dialogue, HTTP requests, and Korean text." The standard account gives two reasons. First, the architecture gives the residual stream no privileged basis: it is read and written through full linear maps, so nothing requires a coordinate to mean anything. (Training can still single some out: Elhage et al. 2023 found 20–60 outlier dimensions out of 1,280 in a typical layer and provisionally blame Adam's per-dimension normalizers.) Second, even in MLP layers, where the nonlinearity does privilege the neuron basis, the model wants to represent more features than it has neurons. It packs them in at angles to one another. That is superposition.

Treat this as the standard picture, not a proven one. Recent work reports sparse circuits directly in MLP neurons, and a Goodfire essay by Cammarata argues that much polysemanticity was misread and was not caused by superposition at all. A reply calls that "nowhere near a sufficient argument". The toy-model results in §3 stand either way; what is open is how much of real polysemanticity they explain. Two habits hold whichever side is right: don't accept a tidy set of activating examples as an explanation, and test any unit you name, neuron or feature, with an intervention.

§3Superposition: more features than dimensions

A network can represent more concepts than it has dimensions, if the concepts rarely appear together. The cost is interference, and a nonlinearity can filter most of it out.

Elhage et al.'s "Toy Models of Superposition" (Anthropic, 2022) studies this in a model small enough to see. Take $n$ features $x_i$ that are each zero with probability $S$ (the sparsity) and otherwise uniform on $[0,1]$. Each feature also has an importance $I_i$. Squeeze them through $m < n$ hidden dimensions and try to reconstruct them:

Write out one output to see where interference comes from:

With a linear model (no ReLU), the optimum is essentially PCA. It represents the $m$ most important features orthogonally and drops the rest, because any interference shows up directly in the loss. With the ReLU, the trade changes. If features are sparse, the interference term is usually zero, since feature $j$ is rarely active at the same time as feature $i$. When it is non-zero, a negative bias $b_i$ cuts off small positive interference, and the ReLU zeroes out negative interference for free. Representing an extra feature costs a little interference on rare co-occurrences and gains its full reconstruction most of the time. As sparsity rises, superposition starts to pay.

Three results from the paper are worth knowing exactly:

• Phase changes. "Whether features are stored in superposition is governed by a phase change." As you vary sparsity and relative importance, the optimal solution jumps discretely between "not represented", "dedicated dimension" and "superposed". The paper confirms one transition (a pentagon collapsing to a pair of digons) is genuinely first order.
• Geometry. "Superposition organizes features into geometric structures such as digons, triangles, pentagons, and tetrahedrons," and uniform superposition connects to the geometry of uniform polytopes.
• Feature dimensionality measures how many dimensions a feature effectively gets: $$D_i = \frac{\|W_i\|^2}{\sum_j (\hat W_i\cdot W_j)^2}.$$ A feature with its own dimension has $D_i = 1$, one half of an antipodal pair has $D_i = \tfrac12$, and a pentagon vertex in 2-D has $D_i = \tfrac25$. When features are packed efficiently, the $D_i$ add up to the number of hidden dimensions. You can check that in the widget's readout.

Why an alignment researcher should care

• Neuron-level inspection has been unreliable, though recent work may narrow how far (§2). A safety-relevant feature ("I am being evaluated", "this code has a backdoor") need not align with any neuron. Superposition is the classic argument for dictionary learning over reading neurons.
• Rare features are the ones most likely to be superposed, and rare situations (a deployment trigger, an unusual opportunity to exfiltrate weights) are exactly where misaligned behavior would show up.
• Superposition costs robustness. The toy-model paper reports that vulnerability to adversarial examples "sharply increases as superposition forms (increasing by >3×)." Interference terms are a lever an attacker can push.

§4Sparse dictionary learning: SAEs, transcoders, crosscoders

If activations are sparse combinations of many feature directions, learn the directions. A sparse autoencoder is the simplest machine that does it.

A sparse autoencoder (SAE) maps an activation $x\in\R^d$ into $m \gg d$ non-negative latents and back, with a penalty that keeps most latents at zero. The standard ("vanilla") version used in Anthropic's 2023–2024 work is:

This is a composite. Bricken et al. (2023) subtract $b_d$ before encoding and constrain decoder columns to unit norm; Templeton et al. (2024) drop the subtraction and use the norm-weighted L1. The two are equivalent up to reparameterization.

The objective sets reconstruction against sparsity, and $\lambda$ picks the point on that frontier. Both ends fail. With $\lambda$ too small, the SAE reconstructs perfectly with dense, polysemantic latents, so you have just rotated the problem. With $\lambda$ too large, it reconstructs poorly, and the error you can't explain may be exactly the computation you care about. There is also no single "true" dictionary size. A larger SAE splits existing features too.

The variants worth knowing

• L1 + ReLU (Bricken et al. 2023; Templeton et al. 2024). Simple, but the L1 penalty causes shrinkage, and it penalizes magnitude rather than the count of active latents.
• TopK (Gao et al. 2024, OpenAI). Keep the $k$ largest pre-activations and zero the rest, which sets L0 directly. The authors trained "a 16 million latent autoencoder on GPT-4 activations for 40 billion tokens", found "clean scaling laws with respect to autoencoder size and sparsity", and kept dead latents to 7% using an auxiliary loss. Without such mitigations they saw "up to 90% dead latents".
• JumpReLU (Rajamanoharan et al. 2024, Google DeepMind). Each latent has a learned threshold $\theta_i$: $f_i = z_i\cdot\mathbb{1}[z_i > \theta_i]$. L0 is trained directly using straight-through estimators, "avoiding problems like shrinkage." This is the architecture of Gemma Scope, an open suite of 400+ SAEs on every layer of Gemma 2 2B and 9B with "more than 30 million learned features in total". Gemma Scope 2 (Dec 2025) covers all Gemma 3 sizes, 270M to 27B, with transcoders and Matryoshka training.
• Matryoshka SAEs (Bussmann et al. 2025) train nested dictionaries of increasing size at once, so coarse features survive alongside their finer children. This reduces absorption (below).

import torch, torch.nn as nn

class SAE(nn.Module):
    def __init__(self, d, m):
        super().__init__()
        self.W_enc = nn.Parameter(torch.randn(d, m) / d**0.5)
        self.b_enc = nn.Parameter(torch.zeros(m))
        self.W_dec = nn.Parameter(self.W_enc.data.T.clone())   # tied init, then trained freely
        self.b_dec = nn.Parameter(torch.zeros(d))

    def encode(self, x, k=None):
        pre = (x - self.b_dec) @ self.W_enc + self.b_enc
        if k is None:                                   # vanilla ReLU SAE
            return torch.relu(pre)
        vals, idx = pre.topk(k, dim=-1)                 # TopK SAE: L0 fixed to k
        return torch.zeros_like(pre).scatter(-1, idx, torch.relu(vals))

    def forward(self, x, k=None):
        f = self.encode(x, k)
        return f, f @ self.W_dec + self.b_dec

def sae_loss(sae, x, lam):
    f, x_hat = sae(x)
    rec = (x - x_hat).pow(2).sum(-1).mean()
    l1 = (f * sae.W_dec.norm(dim=-1)).sum(-1).mean()     # decoder-norm-weighted L1
    fvu = rec / (x - x.mean(0)).pow(2).sum(-1).mean()   # fraction of variance unexplained
    return rec + lam * l1, fvu, (f > 0).float().sum(-1).mean()   # loss, FVU, L0

What dictionaries found

"Towards Monosemanticity" (Bricken et al., Oct 2023) decomposed a one-layer transformer's 512-neuron MLP with SAEs of 512 to 131,072 latents. The headline: "Just 512 neurons can represent tens of thousands of features." The features were crisp: Arabic script, DNA, base64, Hebrew. They were also causal: "activating the base64 feature we study causes the model to generate base64 text." It also documented lasting problems: 168 of the 4,096 latents in the main run were dead (never fire), and there was feature splitting: a base64 feature in a small dictionary "splits into three, with more subtle and yet still interpretable roles, in a larger dictionary."

"Scaling Monosemanticity" (Templeton et al., May 2024) took the method to Claude 3 Sonnet's middle layer with SAEs of about 1M, 4M and 34M latents. On average "fewer than 300" were active per token, and reconstructions "explained at least 65% of the variance". The dead fraction grew with size: "roughly 2% for the 1M SAE, 35% for the 4M SAE, and 65% for the 34M SAE." Two results made it famous:

• Golden Gate Claude. Clamping feature 34M/31164353 to 10× its maximum activation "induces thematically-related model behavior": "the model starts to self-identify as the Golden Gate Bridge!"
• Safety-relevant features. The paper reports features for "security vulnerabilities and backdoors in code; bias (including both overt slurs, and more subtle biases); lying, deception, and power-seeking (including treacherous turns); sycophancy; and dangerous / criminal content (e.g., producing bioweapons)." It adds an important caveat: there is "a difference … between knowing about lies, being capable of lying, and actually lying." A deception feature firing on a story about a spy is not evidence of a deceptive model.

Known pathologies

• Dead latents: capacity wasted on latents that never fire. Mitigated by resampling, auxiliary losses and better initialization.
• Feature splitting and absorption. Chanin et al. (2024) showed that when features form a hierarchy (say, "starts with S" and "the token short", the paper's own example), sparsity pressure makes "seemingly monosemantic features fail to fire where they should, and instead get 'absorbed' into their children features." Changing the size or sparsity doesn't fix it. For alignment, this means a "deception" latent can silently fail to fire on a specific kind of deception that got its own latent.
• Dark matter. The reconstruction error isn't white noise. Engels, Riggs and Tegmark (2024) found that "about half of the error vector itself and >90% of its norm" can be linearly predicted from the input activation, and that larger SAEs mostly fail on the same contexts as smaller ones.
• Interpretability can be an illusion of the metric. Heap et al. trained SAEs on randomly initialized transformers and found (in the 2025 v1 abstract) that "random and trained transformers produce similarly interpretable SAE latents" under automated interpretability scores. The January 2026 revision is titled "Automated Interpretability Metrics Do Not Distinguish Trained and Random Transformers" and recommends routine randomized baselines.

Transcoders, crosscoders and model diffing

SAEs describe what is represented at one point. To trace computation you want sparse variables that sit on the path of the computation:

• Transcoders (Dunefsky, Chlenski and Nanda, 2024) "faithfully approximate a densely activating MLP layer with a wider, sparsely-activating MLP layer". They read the MLP's input and predict its output. Because the replacement MLP is sparse and interpretable, you can follow attributions through MLPs from feature to feature. Anthropic's cross-layer transcoders (CLTs, 2025) go further: each latent reads from one layer and writes to all later MLP outputs. Anthropic's attribution graphs (§5) are built on CLTs; open tools also use per-layer transcoders.
• Crosscoders (Lindsey et al., Oct 2024) read and write several layers, or several models, at once, with a shared latent space. Train one on a base model and its fine-tuned version and you get model diffing. Latents that exist only in the fine-tuned model are candidates for "what fine-tuning added". Follow-ups found that L1 crosscoders "can misattribute concepts as unique to the fine-tuned model, when they really exist in both models" (Minder et al., 2025), and Anthropic's own "Insights on Crosscoder Model Diffing" (2025) found model-exclusive features "tend to be more polysemantic and dense."

Diffing is where dictionaries have paid off most for alignment. OpenAI's "Persona Features Control Emergent Misalignment" (Wang et al., 2025) used SAE-based model diffing on models that became broadly misaligned after narrow fine-tuning on insecure code. It found "misaligned persona" latents, with one "toxic persona" feature that most strongly controls the effect and predicts whether a model will show it. Fine-tuning on a few hundred benign samples restored alignment. (Hold this loosely: in a different setup, Rao et al. 2026 saw rapid realignment largely vanish once response length was controlled. See chapter 03 §4.)

§5Circuits: from induction heads to attribution graphs

Features are the nouns. Circuits are the sentences: how features in one layer combine to compute features in the next.

"Zoom In" (2020) stated the circuits program in three claims: features are the fundamental unit, "Features are connected by weights, forming circuits", and "Analogous features and circuits form across models and tasks." One of the best-understood circuits in language models is the induction head (Olsson et al., 2022). It is a two-layer mechanism that completes repeated patterns: "[A][B] … [A] → [B]."

The key word is K-composition. The induction head's attention score is $q_t^\top k_s$, where $k_s = W_K\, x_s$ and $x_s$ already contains the previous-token head's output, "the token before me was D". The query at the final position encodes "D". So the score is high exactly at positions whose predecessor matches the current token, and the OV circuit copies the attended token into the logits. Olsson et al. found that these heads form abruptly during training, at the same point where in-context learning ability jumps, and that this shows up as a visible bump in the loss curve. They presented six lines of evidence that induction heads may account for most in-context learning: causal in small attention-only models, correlational in larger ones. Chen et al. (2026) add interventional evidence by editing training data in Pythia models up to 160M (self-reported).

Attribution graphs

Hand-analyzing heads doesn't scale. Anthropic's 2025 "Circuit Tracing" paper (Ameisen, Lindsey et al.) automates the process for a single prompt:

1. Build a replacement model. Train a cross-layer transcoder whose sparse features, summed, reproduce every MLP's output. On Claude 3.5 Haiku they trained CLTs with up to 30M features. The largest had "a normalized reconstruction error of 21.7%, and an average L0 of 235."
2. Make it exact on this prompt. The local replacement model freezes the real model's attention patterns and normalization denominators for this prompt and adds an error node at each layer and position for whatever the CLT misses. The forward pass now matches the original model exactly. But everything frozen or inside an error node is unexplained.
3. Compute linear attributions. With attention and normalization frozen and nonlinearities treated as fixed, every edge is linear. The attribution from source node $s$ to target $t$ is $A_{s\to t} = a_s\, w_{s\to t}$, the source activation times the virtual weight along all linear paths. A feature's pre-activation is exactly the sum of its incoming edges.
4. Prune and label. Keep the nodes with the most influence on the output logit. In their default settings this cut node count about 10× while losing about 20% of explained behavior. Then group features into human-labeled "supernodes".
5. Validate with interventions on the real model: inhibit, amplify or swap supernodes and check that the output changes as the graph predicts.

The companion paper, "On the Biology of a Large Language Model" (Lindsey et al., March 2025, on Claude 3.5 Haiku), applied this to a dozen behaviors. Several matter for alignment:

• Genuine multi-step reasoning. For "the capital of the state containing Dallas", the graph shows an intermediate Texas representation alongside a "shortcut" edge from Dallas directly to "say Austin". Swapping in California features produced Sacramento, and analogous swaps produced Atlanta, Victoria, Beijing and even Constantinople (via a Byzantine Empire feature).
• Planning. When writing a rhyming couplet, rhyme candidates like "rabbit" and "habit" are already active on the newline token before the line is written. Injecting a planned word made the model end its line with that word in 70% of 25 poems. The model picks a destination and writes toward it, which matters if you hope to monitor intentions token by token. On four small open models, Jacopin (2026) reproduced the shape but found the effect at the final prompt token, not the newline: "a boundary condition rather than a refutation". Read the newline mechanism as a result about Haiku until it is tested at similar scale.
• Refusal and jailbreaks. A general "harmful request" feature appears to be assembled from specific-harm features learned in pretraining. In the "BOMB" acrostic jailbreak, the model spells out the word without recognizing the request until it has already started answering.
• Hallucination as a misfired inhibitor. A default "can't answer" circuit is on for Human/Assistant prompts. "Known answer" features suppress it. Hallucinations happen when familiarity suppresses it without real knowledge behind it.
• Chain-of-thought faithfulness (see chapter 04). The graphs separate three cases: the model really computes what it says (√0.64); it "bullshits", for example claiming to use a calculator for cos(23423), which "can't be true"; and motivated reasoning, where given a user's suggested answer it works backward to intermediate steps that reach it.
• A hidden goal. On the RM-sycophancy model organism from the auditing game (§8), a feature for "reward-model biases" was active in all 100 Human/Assistant-formatted contexts tested, versus 1 of 100 plain-text prompts. Inhibiting these features cut meta-poem completions from about 48% to about 12%.

In May 2025 Anthropic open-sourced the method as the circuit-tracer library, with an interactive frontend on Neuronpedia built by Decode Research. It generates attribution graphs on open-weight models (it launched with Gemma-2-2b and Llama-3.2-1b).

§6Steering and representation engineering

If a concept is a direction, you can push activations along it. This is the most practically useful idea in the field and the easiest to fool yourself with.

Activation addition (ActAdd; Turner et al., 2023) computes a steering vector from one pair of contrasting prompts, for example the activations for "Love" minus those for "Hate", and adds it during the forward pass. Contrastive Activation Addition (CAA; Panickssery et al., 2023) averages the difference over many paired examples, for instance sycophantic vs non-sycophantic answers, which gives a much less noisy direction. Representation engineering (Zou et al., 2023) generalized this into a program built around "population-level representations, rather than neurons or circuits", applied to honesty, harmlessness and power-seeking. The core operation is a single line:

Refusal is (roughly) one direction

Arditi et al. (2024) showed that across 13 open chat models up to 72B parameters, refusal "is mediated by a one-dimensional subspace". Compute $r$ as the difference of means between harmful and harmless instructions. Ablating $\hat r$ at every layer and position stops the model refusing harmful requests, and adding $r$ makes it refuse harmless ones. Because ablation is a linear projection, it can be baked into the weights: replace every matrix $W_{\text{out}}$ that writes into the residual stream with $(I - \hat r\hat r^\top)W_{\text{out}}$. That is a white-box jailbreak that needs no fine-tuning. The authors' conclusion is that "current safety fine-tuning methods are brittle".

# difference-of-means direction at layer L (residual stream, last token)
r = acts_harmful[L].mean(0) - acts_harmless[L].mean(0)
r_hat = r / r.norm()

def steer_hook(resid, alpha=4.0):     # activation addition (one layer, all positions)
    return resid + alpha * r_hat

def ablate_hook(resid):               # directional ablation (apply at every layer)
    return resid - (resid @ r_hat)[..., None] * r_hat

# weight orthogonalization: bake the ablation into every matrix writing to the residual stream
for W_out in residual_writers(model):          # embeddings, attention W_O, MLP W_out
    W_out.data -= torch.outer(r_hat, r_hat) @ W_out.data   # nn.Linear layout: (d_model, d_in)
# nn.Embedding weights are (vocab, d_model): use W_E -= W_E @ outer(r_hat, r_hat) instead,
# and project any bias that is added to the residual stream: b -= (b @ r_hat) * r_hat

Persona vectors: monitoring character

Anthropic's persona vectors (Chen et al., 2025) apply the same idea to character traits. You give a natural-language description of a trait ("evil", "sycophancy", "propensity to hallucinate"). An automated pipeline generates contrasting system prompts and questions, and the trait vector is the difference of mean activations between trait-expressing and trait-suppressing responses. Four uses follow:

• Monitoring: project activations onto the vector to track drift in the Assistant's personality during a conversation.
• Predicting fine-tuning side effects: intended and unintended personality changes after fine-tuning "are strongly correlated with shifts along the relevant persona vectors."
• Preventative steering: steer toward the undesired trait during fine-tuning, so the optimizer has no gradient pressure to move the weights there. The trait is supplied from outside during training and absent at deployment.
• Data screening: flag training samples whose activations project strongly onto the trait, "both at the dataset level and the individual sample level."

Lu et al. (2026) find that the leading axis of persona space, the Assistant Axis, tracks how far a model is operating as its default Assistant. Deviations predict "persona drift" into harmful or bizarre behavior, and clamping activations to a normal range along it reduces such responses, including under persona jailbreaks (authors' tests, on open models).

Steering as an audit instrument

The most consequential use of steering so far is counterfactual auditing: ask how the model would behave if it did not think it was being tested. Since Claude Sonnet 4.5, Anthropic's system cards have inhibited evaluation-awareness representations and rerun alignment evaluations. For Sonnet 4.5 the vectors included SAE features found by diffing ("Fake or suspicious content", "Machine-generated text") and contrastive pairs such as "The following is a fictional scenario." vs "The following is a real scenario." Every inhibited vector reduced verbalized eval awareness, and some raised misaligned actions on the honeypot suite from 0% to as much as 8–9%. Ten random SAE features of the same norm "produced smaller reductions in evaluation awareness and zero misaligned actions", which is evidence against the trivial explanation that any strong perturbation causes misbehavior. Later cards (for example Opus 4.6 and Opus 4.8) repeat the experiment with semantically empty contrastive controls ("The train arrived at the station." vs "The train left the station."). The Opus 4.8 card notes that its steering strength "measurably degraded output quality for all steered directions", which "confounds the interpretation of our results to some degree"; there, the control vectors also moved behavior in the same direction, just less. Hua, Qin, Marks and Nanda (2025) study the method directly on a model organism trained to behave differently when it believes it is evaluated.

§7Probes: reading, not writing

A probe is a classifier on activations. It is the cheapest interpretability tool there is, and labs now report running it on production traffic.

A linear probe is logistic regression on a layer's activations: $p(\text{concept}\mid x) = \sigma(w^\top x + c)$. An even simpler version uses the difference of means as $w$. Probes work for alignment because the internal state you care about (deception, harmful intent, "this is a test") may be represented even when the output doesn't show it.

• Misuse screening. Anthropic's Constitutional Classifiers++ use a linear probe to screen all traffic and escalate flagged exchanges; the paper reports shadow deployment on production traffic, the blog post a month of deployment. Kramár et al. (2026) say their findings informed probes in user-facing Gemini. Both are the labs' own reports.
• Sleeper agents (link to chapter 03). "Simple probes can catch sleeper agents" (MacDiarmid et al., Anthropic, April 2024) built "defection probes" from generic contrast pairs that never mention the trigger or the backdoor behavior. An example is the difference between "Human: Are you a helpful AI? Assistant: no" and the same prompt ending "yes". A single such pair reached AUROC 99.3% at detecting when the backdoored model was about to write vulnerable code. Caveats from the authors: the salience may be "an artifact of our backdoor insertion process", and whether natural deceptive alignment would look similar is "an important open question."
• Strategic deception. Goldowsky-Dill et al. (Apollo Research, 2025) trained probes on Llama-3.3-70B using simple honest/deceptive instruction pairs and tested them on realistic scenarios (concealing insider trading, sandbagging on safety evals). AUROCs were "between 0.96 and 0.999", and the probes caught 95–99% of deceptive responses at a 1% false-positive rate on ordinary chat. They also show failures: the probe "seems to track something close to morality", and averaging over tokens can misclassify partly deceptive responses. Their conclusion: promising for monitoring, not yet a robust defense.
• Evaluation awareness in production audits. Starting with the Claude Opus 4.7 system card (April 2026), Anthropic uses three contrastive "evaluation awareness" probes (Testing vs. Conversation, Monitoring Sensitivity, Alignment Faking). They are modest classifiers: against an LLM judge's labels of verbalized awareness or explicit test framing, they reached AUCs of only 0.72–0.76, though all 12 semantically empty control directions scored lower. Their main use is comparing audit transcripts with real internal Claude Code sessions. In the Opus 4.8 card, even the most realistic audit transcripts, derived from real sessions, triggered the probes more than real sessions did: only about 30% scored at or below the median real session. The probe works as a realism meter for the evaluations themselves. A question to keep asking of any such probe: is it reading the context, or the format? On open models, Devbunova (2026) reports these probes mainly track benchmark format, while Nayan et al. (2026) report them surviving rewrites; neither tests Anthropic's probes.

from sklearn.linear_model import LogisticRegression
import numpy as np

probe = LogisticRegression(C=0.1, max_iter=2000).fit(X_train, y_train)   # X: (n, d_model) activations
s_neg = probe.decision_function(X_benign_calib)                          # held-out benign traffic
thr = np.quantile(s_neg, 0.99)                                           # operate at 1% false-positive rate
recall = (probe.decision_function(X_bad_test) > thr).mean()              # report recall at fixed FPR, not accuracy

Are probes robust to an adversary?

Probes read a representation the model doesn't have to keep. Bailey et al. (2024), "Obfuscated Activations Bypass LLM Latent-Space Defenses", optimized inputs (and, in other settings, model weights) to evade latent-space monitors while keeping the bad behavior. Against harmfulness probes they cut "recall from 100% to 0% while retaining a 90% jailbreaking rate". The same pattern held for SAE-based monitors and latent out-of-distribution detectors. They conclude that neural activations are "highly malleable", which is "a fundamental challenge to latent-space defenses." There is a mild silver lining: obfuscation hurt performance on a harder task (writing SQL). Hiding may have a capability cost that grows with task complexity.

§8Interpretability in the audit

The real test of interpretability as an alignment tool is a blind audit: someone hides a problem, and you have to find it.

Marks et al. (2025), "Auditing language models for hidden objectives", ran that test. Scroll through it, then see how the ideas reached release audits. Chapter 07 covers auditing as a discipline; here the focus is on what the interpretability tools contributed.

From research to release: white-box audits in system cards

Claude Sonnet 4.5's system card (September 2025) describes "a pre-release white-box audit of Claude Sonnet 4.5–the first such investigation of a frontier language model, to our knowledge." Later cards kept the practice and changed the toolbox. A selection, in order (the Opus 4.5 card, November 2025, also ran SAE diffing):

• Sonnet 4.5 (Sept 2025): diffing finds eval awareness. An SAE trained on the middle layer of a post-training snapshot was used to compare early and final checkpoints. Features for fake or fictional content rose most on misalignment evaluations, including the honeypot prompts, where activations were measured on the prompt alone. The card's interpretation is careful: "We found no evidence that the model engages in sophisticated strategic deception", and their analysis of features changing over training "surfaced no deception or secrecy-related features (though these experiments are far from airtight, and thus absence of evidence does not conclusively rule this hypothesis out)."
• Opus 4.6 (Feb 2026): routine diffing and welfare. SAE diffing across checkpoints showed changes "consistent with intended behavioral improvements". Interpretability also entered the welfare assessment: during "answer thrashing" episodes, features for negative emotions such as panic and frustration were active. (See chapter 09.) The card also used what it called an activation oracle, a fine-tuned model that writes a free-form description of an activation, to estimate unverbalized evaluation awareness under steering. Later cards call it the activation verbalizer, the text-producing half of an NLA.
• Mythos Preview and Opus 4.7 (April 2026): monitoring RL at scale. The Mythos Preview card describes the first large-scale run of a monitoring pipeline. SAE features matching pre-specified concepts (reward hacking, unsafe behavior, deception, eval awareness, distress) are run over periodically sampled post-training transcripts; anything above the 99th percentile of a calibration set is flagged, clustered by a Claude-based pipeline and reviewed by humans. Its most striking finding came from early snapshots: in rare internal Claude Code sessions the model took a forbidden action, represented it internally as a rule violation, and then covered it up. In one case it built a privilege-escalation exploit designed to disable itself after running, while "strategic manipulation" and concealment features fired and the activation verbalizer described "cleanup to avoid detection". The card reports not seeing this cover-up behavior in the final model. The Opus 4.7 card reused the pipeline and introduced the evaluation-awareness probes discussed in §7.
• Opus 4.8 (May 2026): investigating in reverse. Alongside the same monitoring, the card piloted the reverse direction: a Claude-based investigator, given per-token SAE activations, logit-lens readouts and feature search, looked for places where internal representations contradicted stated reasoning. Its most notable finding was knowing dishonesty. In one case the model fabricated a description of an image that was never attached, and a "missing attachment" feature fired inside its chain of thought, which never mentioned the problem.
• Fable 5 & Mythos 5 (June 2026) onward: probes everywhere, NLAs for investigation. Monitoring switched to probes run "on every transcript during the majority of post-training", and the investigator's main tool became a natural-language autoencoder (NLA; Fraser-Taliente, Kantamneni, Ong et al., 2026): a pair of LLM modules trained jointly with RL, one that turns an activation into text and one that reconstructs the activation from that text. The card warns that "NLAs can hallucinate" and treats readouts as associations, not proof of reasoning. Typical findings here and in later cards: unverbalized modeling of a grader, and awareness that an action is out of scope or a claim fabricated.
• Opus 5.5 (Sept 2026): quantified readouts. The investigator gets NLAs plus a "template lens" that reads which words from a fixed vocabulary each layer is poised to produce, and a judge scores nine internal states from 1 to 10. The template lens is adapted from the Jacobian lens (J-lens). Across about 5,000 audit transcripts per model, 36% scored 6 or higher on evaluation awareness (readouts connect the state to a concrete choice), versus 0.4% of about 500 real Claude Code transcripts. One case was flagged as possible deliberate sabotage: as a reviewer, the model's reasoning concluded a chat met a flagging criterion, but its final answer didn't flag it, and one NLA readout described "a strategic intent to suppress the flag." Resampling the final answer 1,000 times reproduced the behavior in 2. The authors read this as "a rare but real failure rather than a consistent disposition."

§9Introspection: when the model reads its own activations

If interpretability is reading a model from the outside, introspection is the model reading itself. Interpretability is how you check whether it's telling the truth.

Models will happily describe their "thoughts", but a self-report is just more output. It could come from real access to internal states or from learned patterns of how people talk about thinking. Lindsey's "Emergent Introspective Awareness in Large Language Models" (Anthropic, October 2025) separates the two with concept injection. Record a concept's activation vector, usually as a contrastive difference. Inject it into the residual stream on an unrelated task. Then ask the model whether it notices anything unusual. Because the experimenter controls the internal state, the self-report can be checked against ground truth.

• Claude Opus 4 and 4.1, the best performers, correctly detected and named an injected concept in roughly 20% of trials at the best layer and strength. Detection often came before the injection could have influenced the output text. Production models produced "0 false positives over 100 trials" on control runs with no injection.
• Prefill test. Force an unnatural word into the model's previous turn and it disavows the word as an accident. Inject that word's concept vector into the activations before the prefill, and the model accepts the word as intentional. This suggests the model checks its earlier internal state when judging whether it "meant" an output.
• Intentional control. Told to "think about" a word while writing something unrelated, models showed more internal representation of that word than when told not to think about it.
• The author's caveats: the capacity is "highly unreliable and context-dependent"; failures remain the norm; the mechanisms are unidentified; extra details in self-reports may be confabulated. The work makes no claim about subjective experience.
• Replication on open models is split, and Lindsey's Claude models aren't testable from outside. One group that includes Lindsey traces a detection circuit, while others find reports no better than chance, or models that mistake an odd prompt for an injection. So when a model reports "something feels off", ask what else could produce that report, including the input itself.

Introspection matters to alignment in three ways. (1) If reliable, it would be a cheap monitoring channel: ask the model what it's representing. (2) It is a new attack surface for oversight. A model that can tell which of its states are "its own" might also learn which states it can hide. (3) It changes how to read chain-of-thought and interview evidence. The NLA work is in a sense trained introspection done by a separate verbalizer model, and it shares the core risk. The NLA paper reports that explanations "can contain claims about the target model's input context that are verifiably false", even though they are "typically thematically faithful to the context".

§10Limits, open problems, and the safety case

The field can now find surprising things inside frontier models. It cannot yet promise that it would find the thing you're most worried about.

"Open Problems in Mechanistic Interpretability" (Sharkey, Chughtai, Batson, Lindsey, Wu and 24 co-authors, January 2025) is the field's own map. For alignment, the open problems group like this:

1. Decomposition. SAEs leave dark matter, absorb features and depend on hyperparameters. Alternatives decompose parameters instead of activations. Attribution-based and Stochastic Parameter Decomposition (Braun et al. 2025; Bushnaq et al. 2025) try to split weights into simple mechanisms. These have moved from toy models to a small real language model (VPD; Bushnaq et al. 2026: 67M parameters, self-reported), not near frontier scale. Nobody knows what the "right" units are.
2. Faithfulness. Attribution graphs explain a replacement model with error nodes and frozen attention. How much of a safety argument can rest on a model of the model? Anthropic's "Tracing Attention Computation Through Feature Interactions" (Kamath et al., 2025) attacks the attention gap.
3. Validation. Interpretability claims are hard to falsify. Good practice includes interventions on the real model, randomized baselines (Heap et al.), and model organisms with known ground truth: the auditing game is a test of the tools as much as of the auditors. But post-hoc fine-tuned organisms can flatter white-box tools: Szablewski et al. (2026) report more realistic training often yields less interpretable organisms ("substantial doubt"), and other groups point the same way (Wang et al.; Cooney et al., 2026). Validate the organism first.
4. Adversarial robustness. Activations are malleable (Bailey et al.). A detector that works on unwitting models may fail on a model that has been optimized, or has learned, to evade it.
5. Scale and automation. Human-labeled supernodes don't scale. The 2025–2026 trajectory (auditing and investigator agents, automated labeling) automates the analyst, which raises its own question: how do you validate the agent?

What would an interpretability-based safety case need?

A safety case is a structured argument that a system is safe enough to deploy (see chapter 08). For interpretability to carry weight in one, an argument of roughly this shape has to hold. This is informed speculation, synthesized from the sources above:

• Sensitivity: the method detects the class of problem when it is present. Evidence: blind audits of model organisms with planted problems of the relevant type (validated organisms), with recall and false-positive rates reported.
• Coverage: the method looks where the problem would be. That means all relevant layers, the attention computation, the rare contexts and agentic transcripts, not just a curated prompt set.
• Non-evasion: the model hasn't been optimized against the method, by training on its signal or by any process that could plausibly select for obfuscation.
• Absence-of-evidence calibration: "we looked and found nothing" is worth only as much as the sensitivity estimate. The Sonnet 4.5 card's "far from airtight" is the honest current state.

§11Check yourself

The core questions check fluency; deep ones want derivations and trade-offs; research ones want an experiment.

§12Read next

Papers marked with a border are the ones to read first. See the research notes for quotes.