Methods · Circuits in Mechanistic Interpretability
Activation Patching
Survey sections: §7.1 Circuit Discovery Methods
Activation patching tests the causality of a component’s representations by swapping them between inputs that differ in the target property.
Evaluation considerations
The resulting change in model output quantifies the causal effect of intervening on S (Zhang & Nanda, 2024).
Intended applications
This technique has also been used to test the causality of probes (Huang et al., 2024; Arora et al., 2024).
The pipeline can also additionally examine a fraction of explanations by causal techniques, such as activation patching, and track failure rates.
Limitations
Requires many interventions; component-level granularity can be coarse
Related methods
Run it
Key papers
Cited in §7.1.
- Fred Zhang and Neel Nanda. Towards best practices of activation patching in language models: Metrics and methods. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=Hf17y6u9BC.
- PreprintNicholas Goldowsky-Dill, Chris MacLeod, Lucas Sato, and Aryaman Arora. Localizing model behavior with path patching, 2023. URL https://arxiv.org/abs/2304.05969.
- Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA, 2023. Curran Associates Inc.
- PreprintAaquib Syed, Can Rager, and Arthur Conmy. Attribution patching outperforms automated circuit discovery. In Yonatan Belinkov, Najoung Kim, Jaap Jumelet, Hosein Mohebbi, Aaron Mueller, and Hanjie Chen (eds.), Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp. 407–416, Miami, Florida, US, November 2024. Association for Computational Linguistics. . URL https://aclanthology.org/2024.blackboxnlp-1.25/.
- Michael Hanna, Sandro Pezzelle, and Yonatan Belinkov. Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=TZ0CCGDcuT.
- Lin Zhang, Wenshuo Dong, Zhuoran Zhang, Shu Yang, Lijie Hu, Ninghao Liu, Pan Zhou, and Di Wang. EAP-GP: Mitigating saturation effect in gradient-based automated circuit identification. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025b. URL https://openreview.net/forum?id=lGyXq0LOeQ.
- Tal Haklay, Hadas Orgad, David Bau, Aaron Mueller, and Yonatan Belinkov. Position-aware automatic circuit discovery. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2792–2817, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. . URL https://aclanthology.org/2025.acl-long.141/.
- Róbert Csordás, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Are neural nets modular? inspecting functional modularity through differentiable weight masks. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=7uVcpu-gMD.
- Steven Cao, Victor Sanh, and Alexander Rush. Low-complexity probing via finding subnetworks. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 960–966, Online, June 2021. Association for Computational Linguistics. . URL https://aclanthology.org/2021.naacl-main.74/.
- Adithya Bhaskar, Alexander Wettig, Dan Friedman, and Danqi Chen. Finding transformer circuits with edge pruning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=8oSY3rA9jY.
- Muhammad Umair Haider, Hammad Rizwan, Hassan Sajjad, and A.B. Siddique. Multi-granular node pruning for causal circuit discovery. In Third Conference on Language Modeling, 2026. URL https://openreview.net/forum?id=Ol96ETTo6k.
- Tung-Yu Wu and Fazl Barez. Query circuits: Explaining how language models answer user prompts. In Forty-third International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id=7F0sragazb.
Also discussed in: §1.2 Survey Scope · §8.6.4 Causal Probing via Activation Patching · §7.5.2 Five Evaluation Dimensions · §12.6.2 Layer 2: Consistency and Causality Checks · §14.7 Transferability Beyond Transformers · §13.5 Choosing Appropriate Methods for Different Safety Applications
Other methods in Circuits in Mechanistic Interpretability
Path Patching · ACDC · Circuit Tracing (SAE- or CLT-Based Circuit Discovery) · Causal Scrubbing · Interchange Interventions