Chapter 1
1 Introduction#PDF p. 9
Neural networks could be one of the most consequential artefacts humanity has built, yet we do not understand how they work or reliably control what they do. This is true despite humans designing the training procedure: we specify how models learn, but not what they learn. Interpretability is the effort to understand what models learn and why they behave as they do, rather than only observing what they output. Mechanistic interpretability (MI) is one approach within it. MI studies the internal mechanisms (characterised in Section 4.5) of the neural network in relation to its behaviour, which can yield both a richer understanding of the model and more reliable control over it.
Recent decades have seen an exponential improvement in AI capabilities (Touvron et al., 2023; OpenAI et al., 2024; Team et al., 2024a; QwenAI, 2025; DeepSeek-AI, 2025; Anthropic, 2025b), but methods to understand and explain internal mechanisms have lagged behind. This has created an interpretability gap between what models can do and what we know about how they do it. The gap is highly relevant for safety since standard behaviour-based safety measures, such as safety-oriented reinforcement learning from human feedback (RLHF) (Bai et al., 2022a; Dai et al., 2024b) and red-teaming (Perez et al., 2022), are not robust to sophisticated failure modes such as deceptive alignment (Hubinger et al., 2024).
The Interpretability Gap
The interpretability gap is the disparity between what models do (their capabilities and behaviours) and what we understand about how they do it (their internal mechanisms). As models scale, this gap risks widening: reasoning emerges that we cannot yet trace, deceptive behaviours may remain undetected, and safety guarantees built on incomplete understanding may fail unexpectedly.
1.1 Brief History of Interpretability#PDF p. 9
While early work did not establish interpretability as a distinct and clearly defined area of artificial intelligence, researchers sought to understand what neural networks learn long before the emergence of modern deep learning models such as transformers (Vaswani et al., 2017). For example, Hinton et al. (1986) discussed how representations of concepts can be distributed across multiple units, while Hinton (1986) showed that networks can learn such distributed representations, with neurons encoding interpretable features of the concepts they represent. Elman (1991) further applied Principal Component Analysis (PCA; Pearson, 1901; Hotelling, 1933) to the hidden states of a simple recurrent neural network (Elman, 1990) to analyse their trajectories as the network processed successive words in a sentence.
Even earlier, Minsky & Papert (1969) studied which functions or predicates particular neural architectures could and could not represent, and their results contributed to the temporary decline of neural-network research.
While this earlier work was not driven by AI safety concerns, it was an early attempt to classify what neural networks can and cannot compute.
Modern efforts to understand the internals of neural networks began almost as soon as deep learning achieved practical success. Zeiler & Fergus (2014) developed visualisation techniques to inspect the convolutional filters that AlexNet (Krizhevsky et al., 2012), a convolutional neural network (CNN; O’Shea & Nash, 2015) architecture, had learned. Their study revealed that early layers detected edges, while deeper layers responded to increasingly complex patterns. Karpathy et al. (2015) and Li et al. (2016) extended this approach to recurrent neural networks. By manually inspecting individual neurons, they discovered ones tracking various linguistic features, such as quotes, line lengths, brackets, negation, and syntactic structure. This showed empirically that an interpretable structure could emerge without explicit supervision.
These early works were largely manual: researchers examined individual neurons and reported qualitative findings. The field soon developed more systematic approaches.
Probing classifiers (Belinkov, 2022) were popularised as a tool for studying neural representations through a series of works on linguistic structure and neural network analysis (Gupta et al., 2015; Köhn, 2015; Shi et al., 2016; Adi et al., 2017; Belinkov et al., 2017; Conneau et al., 2018; Alain & Bengio, 2017). These studies asked whether syntactic categories, semantic roles, or sentence length could be extracted from a layer’s activations. Dalvi et al. (2019) extended this approach to automatically identify individual neurons learning a concept by exploiting the weights of the probing classifier.
This line of work flourished between 2016 and 2019, producing a rich catalogue of information that was present in model activations. It remains important for mechanistic interpretability research to consider carefully what conclusions can legitimately be drawn from the presence of information in a neural network.
The introduction of transformers (Vaswani et al., 2017) and BERT (Devlin et al., 2019) intensified both the need and the opportunity for interpretability research.
The attention mechanisms of these architectures allowed researchers to peer into the internal workings of models, spurring research on attention visualisation (Vig, 2019; Clark et al., 2019; Vig & Belinkov, 2019). In response, criticism and debates arose over whether attention was sufficient to explain model internals (Jain & Wallace, 2019; Wiegreffe & Pinter, 2019). Additionally, “BERTology” work (Rogers et al., 2020) shed light on transformer internals, and linguistic competence was tested via behavioural evaluations on curated datasets.
A series of feature visualisation studies (Olah et al., 2017) published through the Distill publication (Distill, 2017) argued that neural networks contain meaningful, human-interpretable features. Olah et al. (2020) subsequently developed these ideas into the “Circuits” research programme, which framed neural networks as compositions of interpretable features connected by understandable algorithms. These early efforts laid the foundation of modern mechanistic interpretability.
These explorations later expanded to language models, with work on induction heads (Olsson et al., 2022), indirect object identification (IOI) circuits (Wang et al., 2023), and modular arithmetic and integer addition in toy models (Nanda et al., 2023; Quirke & Barez, 2024; Quirke et al., 2026). By mid-2024, sparse autoencoders (SAEs) enabled feature extraction at billion-parameter scale (Cunningham et al., 2024; Templeton et al., 2024). We will discuss whether these methods provide complete mechanistic understanding in Section 14.
Situating the Field.
Early studies of neural network internals emerged in different research communities, including natural language processing (NLP) (Vig, 2019) and computer vision (Olah et al., 2017). Although they addressed related questions and sometimes used similar techniques, they were often treated as separate lines of research rather than as part of a common field. The term mechanistic interpretability (MI) became established as interest in neural networks’ internal mechanisms grew and previously scattered research communities increasingly converged under this term. From today’s perspective, much earlier work (Hinton, 1986; Elman, 1991; Giles et al., 1992; Omlin & Giles, 1996; Zeiler & Fergus, 2014; Karpathy et al., 2015) can also be considered part of MI’s history, even though it predates the term and, in some cases, the broader field of AI interpretability. Today, the scope and standards of MI and AI interpretability are still evolving. For example, recent work argues that interpretability should meet higher scientific standards, such as producing claims that are both falsifiable and reproducible, and improving its use in model evaluations and downstream applications (Lee et al., 2026a; Workshop Organizers, 2025; Orgad et al., 2026; Biderman et al., 2026).
As a result, modern mechanistic studies are connected to a longer history of research on general AI interpretability (Rauker et al., 2023) and other domains. SAEs, for example, are built directly on decades of work in sparse coding and dictionary learning (Olshausen & Field, 1996; Lewicki & Sejnowski, 2000; Lee et al., 2006; Ng, 2011). Similarly, recent studies that discovered circuits in large language models (Conmy et al., 2023; Ameisen et al., 2025) resonate with early work on extracting and formalising the internal mechanisms of recurrent neural networks as finite state automata (Giles et al., 1992; Omlin & Giles, 1996). Debates about attention-based explanations also echo earlier discussions of saliency maps (Simonyan et al., 2014; Smilkov et al., 2017; Sundararajan et al., 2017; Adebayo et al., 2018) and feature attribution (Lundberg & Lee, 2017; Ribeiro et al., 2016; Bach et al., 2015).
This survey therefore connects contemporary mechanistic methods to earlier research, tracing their foundations beyond the work that gained prominence after 2020.
Safety.
Many studies in mechanistic interpretability are motivated by AI safety (Burns et al., 2023; Garcı́a-Carrasco et al., 2024; Templeton et al., 2024).
Researchers hope that understanding the internal workings of models will lead to better detection and control of critical model behaviours. But it is not clear if the current mechanistic approaches are suitable and sufficient for tackling crucial failure modes that are not well addressed by current behavioural or correlation-based methods. This concern motivates us to assess approaches not just by how well they provide understanding, but by the strength of their actionable impacts.
Curiosity and Discovery.
Mechanistic interpretability is worth studying for reasons beyond safety applications. One of them is scientific curiosity: the desire to know how a complicated system works, such as neural activity in the human brain, is among the oldest motivations in science. Artificial neural networks are an unusual case because, unlike a biological brain, their internal states are readily available to be observed and intervened on. This direct access allows researchers to tackle challenging questions, such as what a model represents internally or whether a particular concept is encoded in it.
Besides scientific curiosity, there are also prospects for genuine discovery. For example, a model trained on a complex task (e.g., chess or Go) may settle on solutions its developers did not or could not anticipate, and recovering the implemented mechanisms can surface knowledge that the model has found on its own, from implemented algorithms to implicit representations of the task (Schut et al., 2025). In this sense, interpretability is a way to learn from a model, not just to audit and control it.
Building on this history, recent interpretability research reflects two significant methodological shifts.
First, from manual to assisted analysis.
At current model sizes, manual inspection of features and circuits is impractical. Consequently, researchers have developed methods that automate parts of mechanistic interpretability research. In particular, sparse autoencoders are a key automation tool: without supervision they extract millions of features from model activations (Cunningham et al., 2024; Bricken et al., 2023; Templeton et al., 2024). Other work has proposed pipelines to automatically identify candidate circuits (Conmy et al., 2023; Ameisen et al., 2025), use language models to generate natural-language feature explanations (Paulo et al., 2025b; Bills et al., 2023), and provide initial validation of mechanistic hypotheses. While human judgement remains essential, some automation has become necessary.
Second, divergence in research priorities.
Even within safety-driven mechanistic interpretability research, there are distinct approaches. Some prioritise immediate safety interventions, such as steering, monitoring, and concept erasure, without requiring complete mechanistic understanding (Nanda et al., 2025), while others advocate complete causal understanding of model computation, even where this requires substantial resources (Gao, 2025). Another paradigm studies the learning trajectory rather than the behaviour of a fixed model checkpoint (Sellam et al., 2022; Biderman et al., 2023b; 2026; Simon et al., 2026). Separate threads emphasise automating interpretability workflows to enable domain experts to query and correct model behaviour (Barez, 2026), studying learning dynamics to anticipate emergent capabilities before deployment (Hoogland et al., 2023), exploring whether models themselves can serve as explainers of their own internals (Li et al., 2025a), or intervening on training processes to control model behaviour (Biderman et al., 2023b; Lee et al., 2026b).
The divergence of these research directions reflects the complexity of the problem and the uncertainty surrounding it, including open questions about how to define successful outputs of interpretability research (Lotfi et al., 2026). Thus, this survey does not argue for a single approach; rather, it organises recent progress around shared technical foundations (Section 4 onwards) and discusses trade-offs between different goals (Section 12). We aim to clarify how these directions differ in their assumptions, failure modes, and capabilities, providing researchers with a principled basis for selecting and combining methods according to their own scientific and practical objectives.
Useful Surveys That We Build On.
Several surveys cover overlapping territory. Bereska & Gavves (2024) and Zhang et al. (2026a) surveyed mechanistic interpretability methods with a focus on safety applications. Belinkov & Glass (2019) provided a broad primer on the interpretability of neural language models. Zhao et al. (2024a) surveyed interpretability for large language models. Rai et al. (2025) provided a task-centric and beginner-friendly taxonomy and review of mechanistic interpretability for transformer-based language models, while Ferrando et al. (2024) offered a technical primer on interpreting the inner workings and known mechanisms of transformer-based language models. More specifically, Sajjad et al. (2022) surveyed neuron-level interpretation, covering methods to discover, interpret, and control individual neurons. At a similar granularity, Shu et al. (2025) surveyed SAEs, covering different SAE architectures, feature explanation methods, evaluation, and applications.
Our survey differs from previous surveys in framing, scope, and emphasis. Specifically, we examine closely what interpretability can actually contribute to AI safety. The close examination is motivated by the fact that, although many prior studies in the interpretability community assume that mechanistic understanding directly implies safer systems, the assumption is in fact largely untested. Therefore, this survey considers not only what a method reveals about a model’s internal computation, but also what it allows us to do in practice. We particularly discuss whether it supports safety-relevant tasks such as detecting deceptive behaviours or identifying dangerous capabilities, and whether those tasks can be achieved just as well without interpretability methods or insights.
We cover popular methods and techniques up to mid-2026, including but not limited to recent work on large-scale feature extraction and analysis, circuit discovery, activation verbalisers, and model architectures designed with interpretability in mind. Rather than presenting these methods as a catalogue, we organise them around the methodological ideas they share and explain new methods in relation to the problems or limitations that motivated them.
We also place modern mechanistic methods in their broader historical context. Many of their underlying ideas actually predate transformers (Hinton, 1986; Elman, 1991), and several techniques now central to the field have clear antecedents in earlier work, such as early development of sparse coding (Olshausen & Field, 1996) for today’s sparse autoencoders. Making these connections explicit helps distinguish new methodological developments from adaptations of older ideas to new settings and shows where current debates revisit discussions that have appeared before.
We further make the terminology and claims about mechanistic interpretability more precise. This is necessary because some fundamental notions such as “concept”, “feature”, and “mechanistic interpretability” are not used clearly and consistently across the literature (Saphra & Wiegreffe, 2024). The consequence is that works described as mechanistic interpretability can support different kinds of conclusions; for example, showing that a specific intervention can change the target behaviour is a much weaker claim than showing that the behaviour’s internal mechanism is precisely identified. To tackle this issue, for each introduced method, we discuss the assumptions about model internals it requires, the mechanistic claims it supports, and the strength of the claims. By doing this, we can separate disagreements about their mechanistic claims from disagreements about terminology, objectively comparing these methods even when their original papers use different terms.
Additionally, we systematically discuss the evaluation practices of interpretability methods, which are often validated through intuitive plausibility rather than rigorous testing. We cover existing evaluation frameworks and metrics, identify gaps, and argue that the evaluation methodology and safety benchmarks are the field’s most pressing open problems.
1.2 Survey Scope#PDF p. 12
Our survey’s main focus is on mechanistic interpretability for transformers (Vaswani et al., 2017), which aims to explain how transformers convert inputs into outputs by analysing internal representations and mechanisms. This mechanistic analysis involves direct manipulation of model internals to establish causal claims about computation, and must meet two criteria:
Engagement with model internals: Targeting intermediate representations, individual components (neurons, features, attention heads, and more), or any other learned parameters (such as normalisation, embedding, and unembedding). This is not limited to just input–output relationships.
Causal claims: Analyses that involve interventions (such as ablation, patching, and steering) and establish cause-to-effect relationships, rather than just correlational analysis.
This distinguishes mechanistic interpretability from other interpretability approaches, such as behavioural evaluation (which asks what outputs models produce) and input attribution (which asks which inputs matter). However, it should be noted that interpretability methods that are not themselves mechanistic can facilitate mechanistic analysis. For instance, although vanilla probes can only identify what information is decodable from the activations, they can become mechanistic when the identified directions are tested and used to intervene on models. Input attribution methods help locate salient input features (e.g., pixels), and follow-up processes, such as ablating them, can test causal claims about their roles. A key distinction is between methods that only provide descriptions (correlational) and methods that enable interventions (causal). Because descriptive methods often provide the targets for intervention, we cover in depth methods that analyse internal computation and let us discuss them within a shared causal vocabulary; we treat more briefly methods that are either behavioural or, while mechanistic, need separate theoretical and conceptual foundations.
We cover the following in depth:
Foundations and core hypotheses: Foundational concepts for mechanistic interpretability, including behavioural versus mechanistic understanding, polysemanticity, features, transformer computation, and circuits (Section 4), together with core hypotheses that motivate and constrain later mechanistic methods, including linear representations, superposition, privileged access, and the universality and Platonic representation hypothesis (Section 5).
Feature extraction and representation analysis: Sparse autoencoders and related feature-extraction methods, multi-layer feature analysis, probing classifiers, intermediate-representation decoding, model diffing, knowledge localisation, and representation engineering (Sections 6, 8, and 9).
Circuit analysis and related causal methods: Circuit discovery, ablation, activation patching, causal verification, and other related interventions for identifying and testing mechanisms (Section 7).
Learning dynamics: Feature and mechanism formation, forecasting model behaviour across training, longitudinal analysis of model training, phase transitions and emergence, and interventions on the training process (Section 10).
Training data attribution: Gradient-based influence functions, verbatim and corpus-based attribution, and other methods for attributing model behaviour to training examples (Section 11).
Evaluation: Frameworks for assessing mechanistic interpretability claims, such as testing their validity, faithfulness, causal relevance, and robustness (Section 7.5).
Research orientations: The goals, assumptions, and trade-offs of six major research orientations in contemporary AI interpretability (Section 12).
Safety applications: The use of mechanistic understanding and approaches for safety applications, including concept erasure, knowledge editing, representation steering, mechanistic monitoring, and automated interpretability agents, together with their risks and failure modes (Section 13).
Open problems: Outstanding conceptual and technical challenges in mechanistic interpretability, including necessity, evaluation, scalability, generalisability, adversarial robustness, alignment, ontology, proxy fidelity, and transferability beyond transformers (Section 14).
Notably, training data attribution (TDA; Section 11) does not meet the two criteria and is generally not considered a tool for mechanistic analysis.
We note that many TDA methods operate via gradients or verbatim search, instead of intermediate representations. These approaches credit the behaviour entirely to the training data, completely ignoring internal computations. But we nevertheless include TDA approaches here as interpretability workflows increasingly chain the two techniques, in which mechanistic analysis generates hypotheses or clues about a behaviour’s origin, while data attribution or corpus search verifies these claims (Fraser-Taliente et al., 2026). Parameter-response-based TDA methods (e.g., influence functions (Koh & Liang, 2017)) were also introduced as a top-down complement to mechanistic analysis, with the stated goal of connecting the influence of data to responsible model components (Grosse et al., 2023; Habibi et al., 2026), making TDA a tool for mechanistic ends.
We cover only post-hoc TDA methods that analyse a trained model. Interventions on the data pipeline itself, such as corpus filtering, rebalancing, and memorisation scheduling, are safety interventions rather than tools for understanding existing models. They are out of this survey’s scope.
We discuss but do not cover in depth:
Behavioural evaluation: Red-teaming, benchmarks, and capability evaluations (Ganguli et al., 2022; Perez et al., 2022). These studies assess what models do without explaining how. Section 3 discusses when behavioural methods may or may not suffice.
Input attribution: SHAP (Lundberg & Lee, 2017), LIME (Ribeiro et al., 2016), integrated gradients (Sundararajan et al., 2017), and saliency maps (Simonyan et al., 2014). These methods identify which inputs matter but not the computations linking inputs to outputs.
Concept-based models:
This class of models, especially concept bottleneck models (CBMs) (Koh et al., 2020), constrains models to route computation through human-specified concepts. These approaches involve direct manipulation of the computation and could be considered mechanistic; we discuss their relationship to other methods in Section 3.2.
Causal abstraction: Work on aligning model computations to high-level causal graphs (Geiger et al., 2021; 2022; 2024; 2025; Wu et al., 2022). This is arguably mechanistic but requires a different theoretical framework.
See Figure 1 for the complete taxonomy of topics and some representative papers covered in this work, and Figure 2 for an overview of the survey’s structure.
