The handbook

Read the survey

arXiv:2611.08438 · PDF · alphaXiv · BibTeX

Citations: ■ peer-reviewed · ■ preprint

1. 1Introduction · p. 9
   2 sections

   • 1.1 Brief History of Interpretability
   • 1.2 Survey Scope
2. 2The Limits of Behavioural Evaluation · p. 15
3. 3The Interpretability Landscape · p. 16
   7 sections

   • 3.1 Behavioural Methods
   • 3.2 Concept-Based Approaches
   • 3.3 Input Attribution and Saliency Methods
   • 3.4 Representation Probing and Related Post-hoc Analyses
   • 3.5 Mechanistic Interpretability Methods
   • 3.6 When Mechanistic Understanding Becomes Necessary
   • 3.7 Limitations of Mechanistic Analysis
4. 4Foundational Concepts for Mechanistic Interpretability · p. 21
   5 sections

   • 4.1 Behavioural vs. Mechanistic Understanding
   • 4.2 Neuron Polysemanticity in Model Representations
   • 4.3 Features as Representational Units
   • 4.4 Transformer Computation
   • 4.5 Circuits as Descriptions of Internal Mechanisms
5. 5Core Hypotheses in Mechanistic Interpretability · p. 27
   4 sections

   • 5.1 The Linear Representation Hypothesis
   • 5.2 The Superposition Hypothesis
   • 5.3 The Privileged Access Hypothesis
   • 5.4 The Universality and Platonic Representation Hypothesis
6. 6Feature Extraction and Analysis · p. 31
   10 sections

   • 6.1 The Sparse Coding Problem
   • 6.2 Sparse Autoencoders
   • 6.3 SAE Variants and Architectural Improvements
   • 6.4 Empirical Comparisons of SAE Variants
   • 6.5 Challenges and Failure Modes of SAEs
   • 6.6 Multi-Layer Feature Extraction and Analysis
   • 6.7 Comparing SAEs, Transcoders, and Crosscoders
   • 6.8 Case Study: Induction Heads
   • 6.9 Comparison with Alternative Feature Extraction Methods
   • 6.10 Interpretability-by-Design Methods for Intrinsic Interpretability
7. 7Circuits in Mechanistic Interpretability · p. 53
   5 sections

   • 7.1 Circuit Discovery Methods
   • 7.2 Circuit Discovery with Replacement Models
   • 7.3 Causal Verification
   • 7.4 Case Study: Induction Heads
   • 7.5 A Unified Evaluation Framework for Mechanistic Interpretability
8. 8Understanding Intermediate Representations · p. 66
   6 sections

   • 8.1 Logit Lens: Tracing Predictions Through Depth
   • 8.2 Tuned Lens: Learning Per-Layer Transforms
   • 8.3 Jacobian Lens: Decoding Computational Dispositions
   • 8.4 Patchscopes: Leveraging the Language Model Itself to Explain Intermediate Representations
   • 8.5 LatentQA: Fine-Tuning the Language Model as an Activation Verbaliser
   • 8.6 Probing Classifiers: Capabilities, Critiques, and Causal Limitations
9. 9Representation Analysis: Comparison, Localisation, and Control · p. 77
   3 sections

   • 9.1 Model Diffing
   • 9.2 Knowledge Localisation
   • 9.3 Representation Engineering and Steering
10. 10Learning Dynamics and Mechanistic Interpretability · p. 82
    4 sections

    • 10.1 Longitudinal Analysis
    • 10.2 Learning Dynamics as an Interpretability Tool
    • 10.3 Phase Transitions and Emergence
    • 10.4 Leveraging Learning Dynamics for Safety
11. 11Training Data Attribution · p. 93
    2 sections

    • 11.1 Verbatim Data Attribution: OLMoTrace and Maximal Matching Spans
    • 11.2 Gradient-Based Attribution: Understanding Data Influence Through Perturbations
12. 12Six Research Orientations · p. 102
    9 sections

    • 12.1 Pragmatic Interpretability: Proxy-Task-Oriented Safety Interventions
    • 12.2 Ambitious Interpretability: Towards Complete Mechanistic Understanding
    • 12.3 Developmental Interpretability: Characterising How Models Learn
    • 12.4 Interpretability Across Time: Understanding the Causes of Model Behaviours
    • 12.5 Introspective Interpretability: Models Interpreting Themselves
    • 12.6 Automated Interpretability: Scaling Discovery via Automation
    • 12.7 Comparative Analysis: Trade-Offs Between Orientations
    • 12.8 Choosing an Orientation: A Decision Framework
    • 12.9 Critical Concerns, Tensions, and Open Questions
13. 13From Understanding to Intervention: Safety Applications of Mechanistic Interpretability · p. 112
    7 sections

    • 13.1 Why Mechanistic Approaches for Safety?
    • 13.2 Concept Erasure and Knowledge Editing
    • 13.3 Steering Vectors and Mechanistic Monitoring
    • 13.4 Automated Interpretability Agents
    • 13.5 Choosing Appropriate Methods for Different Safety Applications
    • 13.6 From Research Orientations to Methods
    • 13.7 Risks of Mechanistic Interpretability for AI Safety
14. 14Open Problems · p. 123
    7 sections

    • 14.1 When Is Mechanistic Understanding Necessary?
    • 14.2 Challenges in Evaluation and Benchmarking
    • 14.3 Incompleteness, Scalability, Generalisability, and Adversarial Robustness
    • 14.4 Mechanistic Understanding and Alignment
    • 14.5 Interpretation vs. Truth: The Ontology Question
    • 14.6 The Proxy Fidelity Problem
    • 14.7 Transferability Beyond Transformers
15. 15Conclusion · p. 127
    2 sections

    • 15.1 Achievements and Limits
    • 15.2 The Road Ahead
16. Acknowledgments

Appendices

• Appendix A: Notation Reference and Symbol Table
• Appendix B: Colour Key and Style Guide
• Appendix C: Mathematical Foundations

References (517)