Skip to content
The State of Mechanistic Interpretability
for AI Safety
HandbookMethodsEvaluationOrientationsSafety & open problemsDemosNotebooks Search

Contents

1. Introduction
  • 1.1 Brief History of Interpretability
  • 1.2 Survey Scope
2. The Limits of Behavioural Evaluation
3. The Interpretability Landscape
  • 3.1 Behavioural Methods
  • 3.2 Concept-Based Approaches
  • 3.3 Input Attribution and Saliency Methods
  • 3.4 Representation Probing and Related Post-hoc Analyses
  • 3.5 Mechanistic Interpretability Methods
  • 3.6 When Mechanistic Understanding Becomes Necessary
  • 3.7 Limitations of Mechanistic Analysis
4. Foundational Concepts for Mechanistic Interpretability
  • 4.1 Behavioural vs. Mechanistic Understanding
  • 4.2 Neuron Polysemanticity in Model Representations
  • 4.3 Features as Representational Units
  • 4.4 Transformer Computation
  • 4.5 Circuits as Descriptions of Internal Mechanisms
5. Core Hypotheses in Mechanistic Interpretability
  • 5.1 The Linear Representation Hypothesis
  • 5.2 The Superposition Hypothesis
  • 5.3 The Privileged Access Hypothesis
  • 5.4 The Universality and Platonic Representation Hypothesis
6. Feature Extraction and Analysis
  • 6.1 The Sparse Coding Problem
  • 6.2 Sparse Autoencoders
  • 6.3 SAE Variants and Architectural Improvements
  • 6.4 Empirical Comparisons of SAE Variants
  • 6.5 Challenges and Failure Modes of SAEs
  • 6.6 Multi-Layer Feature Extraction and Analysis
  • 6.7 Comparing SAEs, Transcoders, and Crosscoders
  • 6.8 Case Study: Induction Heads
  • 6.9 Comparison with Alternative Feature Extraction Methods
  • 6.10 Interpretability-by-Design Methods for Intrinsic Interpretability
7. Circuits in Mechanistic Interpretability
  • 7.1 Circuit Discovery Methods
  • 7.2 Circuit Discovery with Replacement Models
  • 7.3 Causal Verification
  • 7.4 Case Study: Induction Heads
  • 7.5 A Unified Evaluation Framework for Mechanistic Interpretability
8. Understanding Intermediate Representations
  • 8.1 Logit Lens: Tracing Predictions Through Depth
  • 8.2 Tuned Lens: Learning Per-Layer Transforms
  • 8.3 Jacobian Lens: Decoding Computational Dispositions
  • 8.4 Patchscopes: Leveraging the Language Model Itself to Explain Intermediate Representations
  • 8.5 LatentQA: Fine-Tuning the Language Model as an Activation Verbaliser
  • 8.6 Probing Classifiers: Capabilities, Critiques, and Causal Limitations
9. Representation Analysis: Comparison, Localisation, and Control
  • 9.1 Model Diffing
  • 9.2 Knowledge Localisation
  • 9.3 Representation Engineering and Steering
10. Learning Dynamics and Mechanistic Interpretability
  • 10.1 Longitudinal Analysis
  • 10.2 Learning Dynamics as an Interpretability Tool
  • 10.3 Phase Transitions and Emergence
  • 10.4 Leveraging Learning Dynamics for Safety
11. Training Data Attribution
  • 11.1 Verbatim Data Attribution: OLMoTrace and Maximal Matching Spans
  • 11.2 Gradient-Based Attribution: Understanding Data Influence Through Perturbations
12. Six Research Orientations
  • 12.1 Pragmatic Interpretability: Proxy-Task-Oriented Safety Interventions
  • 12.2 Ambitious Interpretability: Towards Complete Mechanistic Understanding
  • 12.3 Developmental Interpretability: Characterising How Models Learn
  • 12.4 Interpretability Across Time: Understanding the Causes of Model Behaviours
  • 12.5 Introspective Interpretability: Models Interpreting Themselves
  • 12.6 Automated Interpretability: Scaling Discovery via Automation
  • 12.7 Comparative Analysis: Trade-Offs Between Orientations
  • 12.8 Choosing an Orientation: A Decision Framework
  • 12.9 Critical Concerns, Tensions, and Open Questions
13. From Understanding to Intervention: Safety Applications of Mechanistic Interpretability
  • 13.1 Why Mechanistic Approaches for Safety?
  • 13.2 Concept Erasure and Knowledge Editing
  • 13.3 Steering Vectors and Mechanistic Monitoring
  • 13.4 Automated Interpretability Agents
  • 13.5 Choosing Appropriate Methods for Different Safety Applications
  • 13.6 From Research Orientations to Methods
  • 13.7 Risks of Mechanistic Interpretability for AI Safety
14. Open Problems
  • 14.1 When Is Mechanistic Understanding Necessary?
  • 14.2 Challenges in Evaluation and Benchmarking
  • 14.3 Incompleteness, Scalability, Generalisability, and Adversarial Robustness
  • 14.4 Mechanistic Understanding and Alignment
  • 14.5 Interpretation vs. Truth: The Ontology Question
  • 14.6 The Proxy Fidelity Problem
  • 14.7 Transferability Beyond Transformers
15. Conclusion
  • 15.1 Achievements and Limits
  • 15.2 The Road Ahead
Acknowledgments
A. Notation Reference and Symbol Table
  • A.1 Dimensions and Indices
  • A.2 Inputs and Activations
  • A.3 Dictionary Learning
  • A.4 Model Parameters
  • A.5 Interventions and Objectives
  • A.6 Mathematical Operations
B. Colour Key and Style Guide
  • B.1 Box Types and Colours
  • B.2 Citation Evidence Levels
C. Mathematical Foundations
  • C.1 Causal Inference Basics
  • C.2 Local Learning Coefficient
  • C.3 Information Theory Basics

References

Chapter

Acknowledgments#

FB and TW would like to thank Leo Gao, Peter Hase, John Hewitt, Kyle Lo, Aaron Mueller, Neel Nanda, and Avi Semler for their thoughtful comments and our discussions that helped shape the project.

Previous15. Conclusion NextA. Notation Reference and Symbol Table

On this page

The State of Mechanistic Interpretability for AI Safety. Correspondence: fazl@robots.ox.ac.uk.
Notebooks: TSGLAB/Interpretability_Survey_Safety · Site source: TSGLAB/mechinterp-safety
The original project page was built using the Academic Project Page Template, adopted from the Nerfies project page. Licensed under CC BY-SA 4.0.