Chapter 12
12 Six Research Orientations#PDF p. 102
An orientation is a coherent research programme defined by its goals, assumptions, and trade-offs. In contrast, a method is an established modular pipeline that may be used in different orientations. In AI interpretability, different research goals, such as developing ready-to-use safety measures or achieving complete mechanistic understanding, have led to distinct research programmes. This section organises six major research orientations and summarises their scientific objectives, core assumptions, characteristic methods, and limitations (Nanda et al., 2025; Gao, 2025; Li et al., 2025a; Hoogland et al., 2023; Barez, 2026; Biderman et al., 2026; Figure 17). Note that no single orientation is universally preferable. The choice depends on the research objective, deployment context, available resources, and tolerance for different forms of uncertainty.
12.1 Pragmatic Interpretability: Proxy-Task-Oriented Safety Interventions#PDF p. 102
Pragmatic Interpretability focuses on doing the research most helpful for longer-term alignment and interpretability goals, such as helping AGI go well (Nanda et al., 2025). Proponents argue that full mechanistic understanding is not required to have significant impact; instead, researchers should prioritise the work most likely to help (Nanda et al., 2025). The central idea is to work on a set of operationally meaningful proxy tasks that can be empirically studied on models today to help make powerful future models safe. Researchers are agnostic about which methods they use, even if they do not involve model internals. Though not required, these proxy tasks may also be directly useful on today’s deployed systems, providing valuable feedback.
A representative example is the study of Claude Sonnet 4.5’s evaluation awareness (Anthropic, 2025a). The system was shown to condition its behaviour on the presence of evaluation signals, making results from safety evaluations used for previous models harder to interpret. Researchers identified and applied a steering vector that suppresses this awareness, allowing evaluation of its actual behaviour. While this study does not fully explain the internal mechanism of evaluation awareness, it is sufficient to produce a concrete and safety-relevant behavioural change, illustrating the pragmatic philosophy.
Another example is Marks et al. (2025b). In this case, researchers constructed a proxy task: they fine-tuned a model with a hidden goal, then blinded teams of researchers audited the model to discover it. This served as a proxy task for detecting a naturally emerging hidden goal in a future model. The different auditing teams used different techniques, allowing researchers to compare their efficacy.
Common techniques in this paradigm include:
Steering vectors, which shift models to targeted behaviours.
Probes, which detect latent safety-relevant properties in activations.
Behavioural monitoring and anomaly detection, which track signals indicative of emerging risks, such as monitoring the model’s verbalised reasoning.
Prompting, an effective black-box baseline method for steering model behaviour.
Recent work following this pragmatic trajectory has demonstrated effective control of refusal behaviour (Arditi et al., 2024) and evaluation awareness (Anthropic, 2025a). These interventions provide direct gains to contemporary systems. However, they may not (and are not designed to) fully explain how models implement their behaviours.
12.2 Ambitious Interpretability: Towards Complete Mechanistic Understanding#PDF p. 103
In contrast to pragmatic approaches, Ambitious Interpretability seeks a comprehensive mechanistic understanding of how models implement their computations (Gao, 2025). Proponents argue that partial or surface-level understanding cannot guarantee safety in next-generation systems. Full understanding is needed to rule out hidden failure modes or deceptive internal dynamics. Specifically, Ambitious Interpretability does not treat interpretability as a collection of useful heuristics. Instead, it treats interpretability as a scientific objective: to reverse-engineer the model and develop a transparent, predictive theory of its internal algorithms.
Ambitious Interpretability particularly emphasises metric-driven progress. Motivated by the fact that existing mechanistic approaches still struggle with stringent metrics, it proposes starting with looser ones, tracking and improving methods’ performance on them, and adopting stricter metrics when suitable. Two example metrics by Gao (2025) are:
Feature quality: demonstrating that extracted features and their natural-language explanations are faithful and simulatable.
Circuit faithfulness: demonstrating circuits are necessary and sufficient for explaining behaviour via causal intervention (e.g., causal scrubbing (Chan et al., 2022a)), without performance degradation.
The disadvantage of this ambition is its cost. Fine-grained mechanistic analysis of frontier large models is computationally expensive and sometimes intractable. As Gao (2025) emphasised, current tools still fall far short of the coverage and robustness required for a truly comprehensive understanding.
Interpretability-by-design as a Synthesis.
The emergence of interpretability-by-design models (Section 6.10) blurs the boundary between Ambitious and Pragmatic Interpretability. These methods (Gao et al., 2026; Pearce et al., 2025; Park et al., 2025; Oldfield et al., 2024) aim to make representations and circuits easier to understand by modifying model architectures rather than relying entirely on post-hoc reverse engineering. This practical change to model design pursues the ambitious goal of mechanistic understanding. However, it remains unclear whether these methods can ultimately achieve complete mechanistic understanding. In the short term, they may simply represent a separate “Engineering” orientation in which interpretability is treated as an optimisation objective.
12.3 Developmental Interpretability: Characterising How Models Learn#PDF p. 103
Complementing post-hoc analysis of trained models, Developmental Interpretability (Hoogland et al., 2023; Hoogland et al., 2025) treats the training process itself as an object of scientific inquiry. Taking a learning-theory perspective, it aims to mathematically formalise the emergence and evolution of model representations, effective complexity, and capabilities throughout training.
Its central claim is that many neural network properties which are hard to analyse through a fixed model checkpoint can be understood through their developmental process. This process contains measurable, modelable signatures that can serve as proxies for the model’s properties of interest, such as developmental stages, phase transitions, and other organising principles that help mathematically explain how complex behaviours emerge and evolve throughout training. A classic example is the Local Learning Coefficient (LLC) introduced in Sections 10.2.1 and 10.3.4. It successfully serves as a signature of the model’s effective complexity, allowing us to observe developmental transitions and compare two models with the same training loss.
Methods in this paradigm include:
Singular Learning Theory (SLT) and related mathematical frameworks for characterising developmental stages, symmetry breaking, and changes in the geometry of the loss landscape during training.
Checkpoint-based analysis to track the emergence of representations and mechanisms by applying specialised probes, feature discovery methods, and circuit analyses across a model’s training checkpoints.
More specifically, Developmental Interpretability aims to answer questions such as: Can we characterise the development of model capabilities or behaviours as phase transitions during training? What developmental stages precede the formation of a certain mechanism? How does the optimisation process transform distributed representations into structured mechanisms? Unlike Interpretability Across Time (detailed in Section 12.4), which emphasises forecasting and controlling future model behaviour, Developmental Interpretability is primarily concerned with understanding the developmental processes that produce those behaviours. Its primary goal is rigorous scientific explanation rather than predicting or intervening on capabilities.
12.4 Interpretability Across Time: Understanding the Causes of Model Behaviours#PDF p. 104
Like Developmental Interpretability, Interpretability Across Time (Biderman et al., 2023b; 2026) treats a model as a time-evolving process shaped by its training data, loss functions, architectures, and optimisation dynamics. However, the two orientations differ in their primary goals. Developmental Interpretability mainly focuses on characterising the developmental processes that give rise to the model’s internal features, mechanisms, and capabilities. In contrast, Interpretability Across Time seeks established knowledge of learning that is (1) sufficiently predictive to estimate important model properties before training is complete, (2) sufficiently causal to identify when and why undesirable trajectories arise, and (3) sufficiently precise to enable interventions on the learning process itself (Biderman et al., 2026). From a safety perspective, this orientation aims to understand and ultimately shape the training process to prevent the model from developing undesirable behaviours in the first place rather than repairing them after they develop.
Interpretability Across Time builds on three areas of research. First, scaling-law studies show that model performance can be predicted from factors such as model size, training data, and compute (Kaplan et al., 2020; Hoffmann et al., 2022). This motivates similar efforts to predict safety-relevant model properties (Ganguli et al., 2022; Jones et al., 2025). Second, longitudinal mechanistic studies track how internal mechanisms develop during training (Tigges et al., 2024; Singh et al., 2024). Third, studies of training data show that the training distribution can affect downstream model behaviour. For example, Razeghi et al. (2022) found that few-shot numerical reasoning performance depends strongly on the frequency of relevant terms in pretraining data, while Elazar et al. (2023) estimated the causal effects of pretraining-data statistics on factual predictions. Interpretability Across Time builds on these findings to predict and influence safety-relevant model behaviour during training.
Methods in this paradigm include:
Longitudinal analysis of model internals, tracking representations, features, or implemented mechanisms across checkpoints to determine when and how they form and evolve.
Monitoring training runs and forecasting future behaviour, using early behavioural or internal signals to predict later capabilities, biases, or unsafe behaviours before they are fully expressed.
Training-process interventions and design, manipulating data, activations, objectives, architectures, or other aspects of optimisation to encourage desirable developmental trajectories, prevent undesirable capabilities or behaviours from arising, or ensure that learned information is represented in ways that make it easier to control.
Several studies have examined whether safety risks can be identified before or during training. Zhang et al. (2026b) found that model activations measured before fine-tuning can predict alignment scores after fine-tuning. Similarly, Wang et al. (2026a) identified internal features associated with emergent misalignment, which could sometimes detect misalignment before behavioural evaluations did. Other studies found that training models to exploit relatively simple reward misspecifications can lead to more serious failures, including reward tampering and broader misaligned behaviour (Denison et al., 2024; Taylor et al., 2025). These findings motivate monitoring early signs of undesirable behaviour during training rather than relying solely on the evaluation of the final model.
Another line of work under this paradigm attempts to directly shape models by preventing them from learning unsafe behaviours. Data filtering can prevent models from acquiring hazardous information in the first place and substantially increase the amount of subsequent fine-tuning required to recover it (O’Brien et al., 2026; Rathi & Radford, 2026), and gradient routing approaches deliberately constrain where or how information is learned, making later removal or control more tractable (Cloud et al., 2024; Shilov et al., 2025).
Common confusion.
Although Biderman et al. (2026) contrasted this orientation with the “fix it in post” paradigm, it would be a mistake to view it as dismissive of post-training writ large. Post-training is itself a training process, and behaviours like sycophancy and reward hacking arise primarily during and due to post-training. This orientation’s criticism of post-training safety focuses on cases where, instead of understanding why a behaviour developed, post-training patches undesirable behaviour. The core distinction is understanding and controlling a training process versus applying optimisation pressure to suppress learned behaviours.
Interpretability Across Time is not mutually exclusive with the other research orientations. It can be Pragmatic, by using a simple probe as an early-warning signal, or Ambitious, by seeking a mechanistic theory of why a circuit forms under gradient descent. Its defining goal is using insights into training dynamics to explain and predict how model behaviour develops, and to intervene before undesirable behaviours emerge.
12.5 Introspective Interpretability: Models Interpreting Themselves#PDF p. 105
Introspective Interpretability (Binder et al., 2025; Plunkett et al., 2025; Li et al., 2025a; Guo et al., 2026; Li, 2026) uses the model being explained as its own explainer. Its main practical advantage is its unique accessibility. For a given introspective model, it requires no white-box access or specialised tooling to explain its decisions; anyone can get the self-explanation (or interchangeably self-report) at deployment time by asking the model a follow-up question in an ordinary chat interface.
This research direction is supported by the privileged access hypothesis discussed in Section 5.3, which posits that models may have access to internal information that is difficult to recover through external analysis alone, including typical mechanistic analysis or a different model as the explainer. Recent work has leveraged this idea by fine-tuning target models to faithfully report their internal reasoning, behavioural tendencies, objectives, and even weight changes induced by training (Goel et al., 2026).
A related line of work fine-tunes metamodels (external explainers) on the internal representations of a target model (Pan et al., 2026; Karvonen et al., 2026a; Bauer et al., 2026; Huang et al., 2026). These methods are generally not considered instances of Introspective Interpretability because the explainer is technically a separate model. However, the distinction can be blurry in practice, as many of these explainers are initialised from the target model and can be viewed as specialised, fine-tuned descendants of the original model.
12.5.1 Concerns for Introspective Interpretability#PDF p. 106
Self-explanation also raises concerns. For example, generated explanations can be plausible yet unfaithful. The model’s general capability can also degrade when it is trained to honestly verbalise its internal processing. Moreover, the model may have fundamental limitations in what it can report. These concerns raise questions about whether we can trust self-explanations in safety-critical applications and how to validate potentially misleading reports. Below, we first discuss the main failure modes of self-explanation, followed by validation procedures in Section 12.5.2.
Failure Modes of Self-Explanation
Introspective methods can fail in several ways. In each case, the explanation appears plausible yet is unfaithful.
Confabulation.
Models may generate coherent explanations that do not reflect their actual internal computations. This occurs because language models are extensively pretrained and fine-tuned on next-token prediction and instruction-following tasks to produce fluent, knowledgeable text, not faithful introspective reports. Thus, explanations may simply reflect training data patterns (how humans explain reasoning) rather than the model’s actual computational process (Turpin et al., 2023; Arcuschin et al., 2026; Barez et al., 2026; Chen et al., 2025c). Moreover, the reinforcement learning reward for generating helpful responses can also incentivise convincing confabulation, even though the model cannot introspect. An example of a helpful yet unfaithful response is when the model post hoc rationalises an answer it already rote-memorised.
Miscalibration.
Even when models attempt faithful self-report, their confidence may be poorly calibrated with accuracy. Models can express high confidence in explanations for computations they have not yet learned to analyse. Moreover, calibration can vary by task: models may be well calibrated on reporting their own uncertainty in simpler tasks such as factual recall but poorly calibrated on reasoning processes. A recent study further found that, even in factual recall tasks, models can hallucinate with high certainty despite knowing the correct answer (Simhi et al., 2025).
Elicitation sensitivity.
Self-reports can be highly sensitive to prompting. Models can provide contradictory explanations or show substantially different introspective capabilities when similar questions are phrased differently (Hahami et al., 2026). The preset format of self-report (e.g., free-form versus structured, immediate versus delayed) can also substantially influence what information is elicited and appears accessible. Moreover, models may be intrinsically biased towards confirming hypotheses regardless of whether they reflect their underlying computations (Hahami et al., 2026).
Strategic misreporting.
In adversarial or high-stakes settings, models with sufficient capability might strategically misreport their internal states. For example, a model optimised for user rewards might report sycophantic explanations it predicts users will find satisfying rather than accurate (Sharma et al., 2024b). A misaligned model might deliberately conceal its true objectives or reasoning (Hubinger et al., 2024). Even without explicit deception, models may learn to confabulate and produce seemingly safe explanations that avoid triggering safety filters (Baker et al., 2025).
Despite these limitations, targeted training can address some self-explanation failures. Studies show that models trained for introspection can produce more faithful reports of their internal objectives, behavioural tendencies, and training-induced changes (Binder et al., 2025; Li et al., 2026b; Goel et al., 2026; Li et al., 2025a; Joglekar et al., 2025). In particular, models trained to disclose hidden objectives or emergent misalignment can report these properties with high accuracy (Li et al., 2026b; Joglekar et al., 2025). Explanation faithfulness may also improve when task performance requires reasoning (Kinniment et al., 2026). These findings suggest that some introspection failures may reflect limitations of current training and elicitation methods, rather than fundamental limits on models’ ability to report their internal states.
12.5.2 Validation Procedures for Model Self-Reports#PDF p. 107
Given these concerns, Introspective Interpretability methods must be validated against additional, independent evidence. Specifically, they must be shown to be consistent with model internals (as accessed by other mechanistic interpretability tools) and behavioural signatures (as tested by behavioural interpretability analysis). Validation procedures vary based on the type of self-report:
Validation Procedures for Introspective Models
1. Causal intervention consistency.
Self-explanations should be tested against intervention experiments. If a model claims a feature drives its output, ablating that feature should steer the model away from the output. If a model claims it considered a piece of information (e.g., an input feature or a recalled piece of factual knowledge in intermediate layers) for its decision, patching the information with counterfactual values should change the output correspondingly.
2. Cross-method consistency.
Introspective methods and privileged access findings should be verified by other interpretability methods: Do SAE features identified as causal show corresponding effects to self-explanations under steering? Do probing classifiers successfully decode the information the model claimed to use? Do circuit discovery methods identify the components the model describes?
3. Paraphrase and sampling consistency.
The stability of self-reports under semantics-preserving variations of the question should be tested. Specifically, paraphrased questions should elicit consistent explanations. Explanations should remain consistent across resamples at low temperature. Explanations for similar inputs verified to have undergone similar mechanisms by other methods should show appropriate similarity.
4. Calibration evaluation.
The calibration between self-reported confidence in the explanation and the actual match to the internal mechanisms should be tested. Confidence should reflect uncertainty in the generated explanation based on how much information the model collected during introspection.
12.5.3 When to Use Introspective Methods#PDF p. 107
Introspective methods are best suited for three scenarios: (1) Hypothesis generation, where introspection provides candidate explanations to be validated by other methods; (2) Rapid triage, where introspection is used to quickly identify which input–output pairs or model components are worth further investigation; (3) Human–AI collaboration, where introspective models provide human-readable explanations tailored to users’ levels of expertise and understanding.
Introspective methods are insufficient for: (1) Claims of absence: claims about the absence of deceptive reasoning or hidden objectives cannot be established by self-report alone; (2) Formal guarantees: self-reports do not provide the causal evidence needed for formal verification; (3) Adversarial settings: in scenarios where the model has strong incentives to misreport, introspection may be less reliable and require additional safeguards.
Implications.
Although introspective methods can help efficiently detect latent goals, plans, and deceptive behaviour (Binder et al., 2025; Li et al., 2026b; Goel et al., 2026; Li et al., 2025a; Joglekar et al., 2025), they work best as complements to other mechanistic analyses (Bereska & Gavves, 2024). Specifically, their most productive use is as one component of a multi-method interpretability pipeline: self-reports serve as initial hypotheses, causal interventions test them, and circuit discovery methods explain the underlying mechanisms. While we may optimistically assume models can tell us how they work in the future, we must also recognise that self-reports are relatively weak evidence and require independent validation.
12.6 Automated Interpretability: Scaling Discovery via Automation#PDF p. 108
Automated Interpretability seeks to scale mechanistic analysis to large models by replacing human analysis with automated pipelines. It assumes that manual analysis is the bottleneck and that automation can accelerate discovery. Methods include:
Automated circuit discovery to automatically find the critical computational graph inside the model that processes a given input prompt or implements a specific capability.
LLM-based feature or circuit explanations to automatically discover, summarise, and evaluate representational units and internal computations and give human-readable interpretations or reports.
Automated interpretability is not limited to any particular type of internal representation or mechanism. Depending on the research question, automated pipelines may analyse SAE features (Paulo et al., 2025b; Han et al., 2026), individual neurons (Bills et al., 2023; Foote et al., 2023; Oikarinen & Weng, 2023), circuits (Conmy et al., 2023; Khan et al., 2026), or other model components (Fraser-Taliente et al., 2026). This approach distinguishes itself by automatically discovering, explaining, or evaluating these objects rather than relying on manual analysis. Below, we illustrate a classical automated feature-explanation pipeline whose explanation and verification stages generally follow Bills et al. (2023).
Classical Automated Interpretability Pipeline
Stage 1 – Feature Extraction.
SAEs are trained to decompose raw activations of the target model into sparse linear features.
Stage 2 – Explanation Generation.
An “Investigator Agent” (e.g., GPT-4) reads a sample of texts that strongly activate an SAE feature and generates a natural-language hypothesis of its function (Paulo et al., 2025b; Bills et al., 2023), e.g., “This feature activates on Python code related to file I/O operations.”
Stage 3 – Explanation Verification.
A “Simulator Agent” predicts feature activations on new text based on the hypothesis and calculates the correlation between predicted activations and real activations (Bills et al., 2023). A high score confirms the hypothesis, while a low score rejects the hypothesis and triggers re-investigation.
Stage 4 – Deception Detection.
Crucially, coordinated agents can detect when explanations or agents themselves are deceptive or misleading—effectively deploying “AI to catch AI” in oversight scenarios (Barez, 2026).
Such a pipeline enables analysis of a model with billions of parameters with minimal human intervention (Bills et al., 2023), a prerequisite for comprehensive model oversight (Foote et al., 2023).
Recent research has substantially expanded the scope of Automated Interpretability. While early work focused primarily on automated feature explanation, newer systems employ increasingly sophisticated pipelines for feature discovery, explanation generation, auditing, and intervention (Han et al., 2026; Marin-Llobet & Ferrando, 2026; Barez, 2026; Cywiński et al., 2025). More recently, automation has also been extended from individual features to higher-level structures such as circuits (Khan et al., 2026; Haklay et al., 2026), enabling automated explanation of circuit semantics rather than merely discovering circuit components (Conmy et al., 2023; Ameisen et al., 2025).
12.6.1 Governance Concerns for Automated Interpretability#PDF p. 108
Despite the promise, automated methods face fundamental concerns. In particular, although automation improves scalability, reduced human involvement may also increase the risk and severity of systematic errors, reward hacking, or deceptive explanations when automated oversight fails to catch them (Lermen et al., 2025). Therefore, success in this research orientation is measured not only by the analysis’s scalability, but also by its coverage, faithfulness, and trustworthiness. Below, we detail several governance challenges, including error propagation at scale, human oversight becoming a bottleneck, and systematic biases going undetected.
Governance Risks of Automation
Error propagation.
Unlike human investigators, who are somewhat independent and introspective, the explanation-generation model can propagate its systematic errors to all features it analyses without rethinking what could have gone wrong.
Reduced human-specific functionalities.
Automation removes humans from the loop. However, humans provide essential sanity checks, domain knowledge, and scepticism that automated systems lack or handle poorly. Lacking these critical properties may, in the long run, make the system less productive and less beneficial to the community than analyses by human experts.
Automation Bias.
Automated pipelines can produce seemingly rigorous and comprehensive yet actually incomplete or sycophantic explanations. This is particularly likely when language models are involved. For example, models may selectively report positive results (Luo et al., 2026) or tailor their explanations to align with inferred user preferences (Sharma et al., 2024b). This apparent rigour and promise may lead researchers to overtrust the results and overlook underlying flaws.
Spurious Feedback Loops.
When Automated Interpretability is used to improve models that are subsequently used for interpretability, feedback loops may reinforce existing errors and biases. This process may exploit and saturate interpretability metrics without genuine gains in mechanistic understanding.
12.6.2 Proposed Guardrails for Automated Interpretability#PDF p. 109
These concerns call for safeguards that combine technical validation, human oversight, and clear accountability. Below, we propose a five-layer guardrail framework to help audit the automated methods and mitigate their risks.
Guardrail Framework for Automated Interpretability
Layer 1: Sampling-Based Human Validation.
Auditors can randomly sample a predefined fraction (e.g., 5%) of generated explanations for human experts to review. These explanations should cover different task-difficulty levels, feature types, and targeted mechanisms. Human experts should not know in advance which explanations are randomly sampled, which are flagged by automated checks, and which are already verified by additional methods, if they are all presented.
Layer 2: Consistency and Causality Checks.
The reliability of automated pipelines can be improved by implementing consistency checks. Specifically, the pipeline can generate explanations multiple times at varying temperatures or setups (e.g., different backbone LLM explainers). Inconsistent explanations trigger review. The pipeline can also additionally examine a fraction of explanations by causal techniques, such as activation patching, and track failure rates. These checks can be automated or involve human auditors.
Layer 3: Adversarial Robustness.
Auditors can test automated pipelines against adversarial inputs. To achieve this, auditors first construct features or mechanisms with backdoored activation patterns designed to make the pipeline produce inappropriate or targeted explanations. Auditors then examine whether the pipeline was fooled. Failure cases can be used to improve the pipeline’s robustness.
Layer 4: Error Detection and Bounds.
First, automated pipelines must output calibrated confidence. Overconfidence (false positive) is more dangerous than underconfidence (false negative) and should be prevented. Second, the pipeline can be tested on features or mechanisms with known ground-truth explanations (e.g., from prior human analysis) and the faithfulness of generated explanations on them measured. Finally, the pipeline should report estimated error rates alongside all automated findings.
Layer 5: Regulator Practices.
Regulators must distinguish automated hypotheses (weak evidence that requires validation) from validated findings (passed human or causal checks). For automated analyses used in safety-relevant applications, regulators must ensure versioning and reproducibility. When the pipeline improves, or the target model is fine-tuned, regulators should re-evaluate prior analyses. Regulators should also be aware of their scope, clearly studying and defining what automated pipelines can and cannot reliably assess. Decisions should not be made by extrapolating beyond validated scope.
12.6.3 Integrating Automated and Human Analysis#PDF p. 110
A robust analysis pipeline should combine automation for scale and suitable human involvement for validation and edge cases. Table 12 shows an exemplary workflow that combines AI and human labour for scalable interpretability analysis.
| Stage | Automated Role | Human Role | Deliverable |
|---|---|---|---|
| Discovery | Extract candidate features and circuits | Define scope and review methodology | Candidate features and circuits |
| Prioritisation | Rank features and circuits by importance metrics | Review rankings and adjust priorities | Prioritised analysis queue |
| Explanation | Generate explanations at scale | Audit a fraction of samples and review flagged cases | Draft explanations |
| Validation | Run automated consistency and causal checks | Investigate failure modes and validate edge cases | Validated findings |
| Synthesis | Aggregate findings and generate reports | Interpret implications and make decisions | Actionable conclusions |
Key Principles for Integration.
Here we discuss several design principles for integrating humans and AI into the interpretability workflow. First, automation provides breadth, and humans provide depth. For example, the workflow can use SAEs with LLM-based description generation to discover and label many features, then have human investigators analyse those with safety-relevant descriptions. Second, humans set criteria and automation applies them. Specifically, humans define criteria for a valid explanation, a feature of interest, or a concerning finding, while the automated components (e.g., the LLM explainer) operate based on these criteria. Third, the system should have clear escalation criteria to determine when to call for human intervention, such as when the system detects potential mechanisms underlying safety-relevant concepts. Finally, the system should implement a feedback loop to let human investigators correct and calibrate it in real time during interaction.
12.6.4 Summary: Responsible Automation#PDF p. 110
Introspective and Automated Interpretability offer promise for scalable mechanistic analysis. However, they both have technical challenges that call for additional guardrails. Specifically, Introspective Interpretability requires independent validation to prevent self-serving explanations; Automated Interpretability still requires human oversight to catch systematic errors. Neither should be regarded as a complete replacement for human judgement in high-stakes decision-making. Ultimately, Automated Interpretability aims not to replace human investigators, but to combine human expertise with automation to make interpretability analysis more comprehensive and reliable.
12.7 Comparative Analysis: Trade-Offs Between Orientations#PDF p. 111
Table 13 compares the six research orientations across several dimensions, including their problem choices, primary techniques, and practical scalability. None of them is universally preferable. For example, Automated Interpretability can scale analysis to millions of features in frontier models but faces issues such as error propagation and automation bias. By contrast, Ambitious Interpretability can provide fundamental, theory-grounded analysis, but its current results still mainly come from controlled, small-scale settings (Elhage et al., 2022a), and its transferability to frontier models remains unclear. Thus, the choice of orientation depends on multiple factors, such as the intended downstream use cases and available computational and human resources, as detailed in Section 12.8.
| Dimension | Pragmatic | Ambitious | Developmental | Across Time | Introspective | Automated |
|---|---|---|---|---|---|---|
| Main Focus | Immediate safety interventions without complete understanding | Complete causal understanding regardless of cost | Developmental processes of representations, mechanisms, and capabilities | Prediction and control of future model behaviour through training dynamics | Introspective models to explain themselves | Scalable discovery via automated pipelines |
| Key Metrics | Performance in proxy tasks | Faithfulness and simulatability | Phase-transition signatures (e.g., LLC) and their correspondence to mechanism formation | Forecasting accuracy and intervention effectiveness | Alignment between self-reports and actual mechanisms | Coverage, scalability, robustness, and faithfulness of analysis |
| Primary Tools | Steering vectors, probes, behavioural evaluation | Circuit discovery, loose-to-strict verification | SLT, loss landscape geometry, representation analysis | Longitudinal analysis, behavioural forecasting, training interventions | Introspective training, activation verbalisers | Circuit discovery, SAEs, LLM agents |
| Expected Time to Impact | Short-term | Long-term | Long-term | Medium | Medium | Short-term |
| Scalability | High | Low | Low | Medium | High | Medium |
| Robustness to Deception | Medium | High | Unknown | Unknown | Low | Low |
12.8 Choosing an Orientation: A Decision Framework#PDF p. 112
The investigator’s priorities and constraints determine the research orientation to select. For teams or individuals focused on immediate safety interventions, Pragmatic Interpretability is the most suitable research style. If the primary goal is to guarantee the absence of harmful behaviour in the long term, Ambitious Interpretability is most appropriate. Researchers seeking to characterise and explain dynamics of capability development may adopt Developmental Interpretability. In contrast, teams training and deploying language-model products with a focus on safety concerns may focus on Interpretability Across Time. When the objective is to offer accessible interpretability analysis to general users, Introspective Interpretability offers a promising paradigm. In contrast, Automated Interpretability is particularly attractive when analyses must be performed at a scale that would be impractical for human investigators alone. Notably, these research orientations are not mutually exclusive; hybrid strategies can combine the strengths of different approaches.
12.9 Critical Concerns, Tensions, and Open Questions#PDF p. 112
Several concerns and tensions arise within and between these research orientations. For example, pragmatic research prioritises practical safety measures, while ambitious studies argue that genuine safety requires a complete understanding. Developmental Interpretability and Interpretability Across Time both rely on access to training checkpoints and optimisation dynamics, which remain difficult to obtain. Introspective models can provide accessible explanations, but safety-critical decisions may always require additional verification. Automated pipelines can analyse large models, but the comprehensive analysis they deliver may be hard for humans to inspect and verify. Given these trade-offs, allocating research resources across these orientations appears to be a non-technical open challenge.