Sparse Autoencoders (SAEs)

Assumptions and requirements

SAEs are motivated by the hypothesis that a model’s activations can be approximated as linear combinations of many latent feature directions, which can be regarded as an extension of the linear representation hypothesis discussed in Section 5.1.

SAE training is sensitive to initialisation, sparsity strength, and decoder normalisation. Without proper constraints and hyperparameters, the features may collapse, duplicate, or absorb one another.

Evaluation considerations

SAEs are typically evaluated for reconstruction loss (fidelity of activation reconstruction), feature absorption (how often a general feature fails to fire because a more specific feature has absorbed its direction), and downstream test loss recovery (whether replacing activations with reconstructions preserves model behaviour).

Report failure-mode metrics: Alongside reconstruction loss, researchers should also report dead-feature percentage, duplication rate, and stability across seeds.

Validate causal relevance: Researchers should validate features’ necessity, sufficiency, and specificity through interventions (e.g., feature ablation or steering) and evaluation on downstream effects.

Intended applications

Exploratory discovery, where the question is what features exist rather than whether a particular one does, is the setting SAEs were designed for and the one the comparisons below address.

Methods for achieving mechanistic monitoring include, but are not limited to, layer-wise monitoring (using linear probes to classify if malicious intent is encoded in a layer representation) (Oldfield et al., 2026), feature-wise monitoring (alerting when unexpected SAE features activate) (Fang et al., 2026), circuit-wise monitoring (tracking causal effects among attention heads, MLPs, or features) (Zhao et al., 2026b), and anomaly-based monitoring (using activations in normal behaviour as a baseline and flagging harmful generations as outliers) (Chaudhary & Barez, 2026).

Limitations

Dead features waste the dictionary’s capacity, and they are common. In fact, they can constitute up to 10–50% of the dictionary in a poorly tuned SAE (Bricken et al., 2023).

For a fixed SAE architecture and training dataset, different random seeds can produce substantially different dictionaries (Paulo & Belrose, 2026).

In large language model activations, high coherence is likely unavoidable because of the model’s complicated internal structure and correlations within its feature space (Li et al., 2025b). So far, identifiability remains a stubborn issue in SAEs, even though it offers a theoretical explanation for several empirical observations.

Activation reconstruction $\neq$ computation reconstruction: Low SAE reconstruction loss does not necessarily mean the SAE captures the original model’s computation.

Related methods

• Gated SAEs
• JumpReLU SAEs
• Top-k SAEs
• Batch Top-k SAEs
• Transcoders
• Cross-Layer Transcoders (CLTs)
• Crosscoders
• Non-Negative Matrix Factorisation (NMF)
• Principal Component Analysis (PCA)
• Independent Component Analysis (ICA)
• Supervised Feature Extraction Methods (DAS, SMDS, probes)
• Dictionary Learning Baselines (K-SVD)
• Representation Engineering and Steering (Steering Vectors)
• Sparse Mixture of Linear Transforms (MOLT)
• Circuit Tracing (SAE- or CLT-Based Circuit Discovery)
• Model Diffing
• Concept Erasure via PISCES
• Mechanistic Monitoring

Compare side by side →

Used in safety applications

• Concept Erasure via PISCES
• Automated Interpretability Agents

Run it

• Interactive demo: Sparse autoencoders recover planted features
• Notebook: Sparse Autoencoders: Variants and Implementations · Open in Colab

Key papers

Cited in §6.2.

• ■ Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=F76bwRSLeK.
• ■ Bruno A. Olshausen and David J. Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381 (6583): 607–609, June 1996. .
• ■ Michael S. Lewicki and Terrence J. Sejnowski. Learning overcomplete representations. Neural Comput., 12 (2): 337–365, February 2000. ISSN 0899-7667. . URL https://doi.org/10.1162/089976600300015826.
• ■ Michal Aharon, Michael Elad, and Alfred Bruckstein. K-svd: An algorithm for designing overcomplete dictionaries for sparse representation. IEEE Transactions on signal processing, 54 (11): 4311–4322, 2006.
• ■ PreprintTrenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. https://transformer-circuits.pub/2023/monosemantic-features/index.html.
• ■ PreprintLuke Marks, Alasdair Paren, David Krueger, and Fazl Barez. Enhancing neural network interpretability with feature-aligned sparse autoencoders, 2024b. URL https://arxiv.org/abs/2411.01220.
• ■ Yoav Gur-Arieh, Roy Mayan, Chen Agassy, Atticus Geiger, and Mor Geva. Enhancing automated interpretability with output-centric feature descriptions. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5757–5778, Vienna, Austria, July 2025a. Association for Computational Linguistics. ISBN 979-8-89176-251-0. . URL https://aclanthology.org/2025.acl-long.288/.
• ■ Gonçalo Santos Paulo, Alex Troy Mallen, Caden Juang, and Nora Belrose. Automatically interpreting millions of features in large language models. In Forty-second International Conference on Machine Learning, 2025b. URL https://openreview.net/forum?id=EemtbhJOXc.

Other methods in Sparse Autoencoders / SAE Variants and Architectural Improvements

Gated SAEs · JumpReLU SAEs · Top-k SAEs · Batch Top-k SAEs