The problem
Ask a language model for the capital of the state containing Dallas, and it may answer Austin. Did it work through Texas, or use a memorized Dallas-to-Austin shortcut? Mechanistic interpretability tries to find the internal computations that actually produce an answer, then test that explanation by changing those computations. A model saying ‘I thought of Texas’ is not enough.[2] [4]
The Dallas question separates two candidate computations: identify the state and then its capital, or associate the city directly with a capital answer. The published analysis finds contributions from both. To study such routes, circuit tracing approximates the model’s feed-forward computations with a network of more interpretable features. An attribution graph proposes which features influence others for a particular prompt. Researchers then perturb features in the underlying model and compare the resulting outputs with the graph’s predictions. Feature descriptions come from activation examples and other evidence; they remain hypotheses about function, rather than explanations certified by a label. A useful test therefore asks what should change, and what should stay the same, before making the intervention.[2] [3] [4]
Scope
Mechanistic explanations of learned neural-network computations, their validation and their usefulness. A model's verbal explanation is not automatically a trace of its internal process.
What remains open
The replacement does not reproduce all computation, and the method leaves attention’s choice of what to attend to unexplained. Some intervention procedures constrain intermediate computations and keep attention patterns fixed, so agreement there can partly follow from the procedure itself. Output changes and further tests are needed. General challenges include choosing the right computational units, measuring omitted mechanisms, and validating predictions beyond the cases used to build an explanation.[3] [4]
Why it matters
The two routes suggest different ways an answer might fail when the question changes. Understanding which route a model uses could help people design sharper tests and debug mistakes. Reliable explanations might also help investigate failures of AI safeguards; the geography example does not demonstrate that broader ability.[4]
A feature responding to descriptions of deception need not cause deceptive behavior. Distinguishing those roles matters for an audit: otherwise an investigator may treat recognition as evidence of intent or control. Better mechanistic accounts could guide failure searches or more precise model edits. Their value must be measured against those tasks, including effects on unrelated behavior, rather than assumed from the readability of a circuit diagram.[4]
Research directions
3 directions to explore
Does an explanation identify a cause of the model's answer?
Use interventions and other validation to distinguish a mechanism that changes the output from a feature merely associated with it. Interpretable labels alone do not establish a faithful causal explanation.
How much important computation does an explanation leave out?
Identify behavior missed by a simplified model or attribution graph. Anthropic's report acknowledges incomplete coverage, so an explanation of selected cases should not be treated as a full account of the model.
Which mechanisms persist across models and tasks?
Test whether a mechanism identified in selected examples transfers to other tasks or models. The published methods agenda includes the gap between case studies and dependable broader understanding.
Definitions
3 terms
activation
An internal numerical value produced while a neural network processes an input. Looking at it can suggest a hypothesis about computation, but its correlation with an answer alone does not establish a cause.[3] [4]
Sources
Partial results
Reports circuit-tracing explanations for selected behaviors in Claude 3.5 Haiku. Scope: Selected prompts and one model studied through an imperfect replacement model.
The authors emphasize incomplete coverage and biased selection of successful cases. These reports are not a general solution.
Source date: 2025-03-27Author-published research reportRead the source ↗- Open Problems in Mechanistic Interpretability
Notes on Open Problems in Mechanistic Interpretability
A review published in Transactions on Machine Learning Research, mapping gaps in methods, validation and applications.
- On the Biology of a Large Language Model
Notes on On the Biology of a Large Language Model
An author-published research report tracing selected mechanisms in Claude 3.5 Haiku. Its limitations section is essential.
- Circuit Tracing: Revealing Computational Graphs in Language Models
Notes on Circuit Tracing: Revealing Computational Graphs in Language Models
Author-published methods report. Read the intervention procedure and attention/reconstruction limitations alongside the graph descriptions.
- Open Problems in Mechanistic Interpretability — author preprint, version 1
Notes on Open Problems in Mechanistic Interpretability — author preprint, version 1
Readable January author preprint. This citation is separate from the retained September TMLR publication locator; its methods and applications agenda is not evidence of a general solution.