Methods · Comparison with Alternative Feature Extraction Methods
Principal Component Analysis (PCA)
Survey sections: §6.9.2 Principal Component Analysis
Principal Component Analysis (PCA; Pearson, 1901; Hotelling, 1933) decomposes activations by finding an orthonormal basis of directions that maximise explained variance.
Assumptions and requirements
PCA is fast, stable, and scalable. However, PCA imposes constraints and inductive biases that may misalign with neural network activations. Specifically, it identifies principal directions that maximise explained variance in the activation distribution, but much of the information of interest is region- or context-dependent and cannot be captured by variance alone. Moreover, PCA constrains the principal directions to be orthogonal, but in practice the underlying features correlate or partially overlap.
Evaluation considerations
Recent benchmarks (Wu et al., 2025b) reflect this misalignment: PCA underperformed most baselines, including SAEs, on both steering and concept-detection evaluations. While PCA can outperform SAEs in some localisation settings (Mueller et al., 2025), it still generally lags behind simple supervised baselines.
Intended applications
Users specify baseline methods, such as random directions in activation space, PCA components, individual neurons, and supervised probes (if labels are available).
Related methods
Key papers
Cited in §6.9.2.
- Karl Pearson. On lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2 (11): 559–572, 1901. .
- Harold Hotelling. Analysis of a complex of statistical variables into principal components. Journal of Educational Psychology, 24 (6): 417–441, 1933. .
- Zhengxuan Wu, Aryaman Arora, Atticus Geiger, Zheng Wang, Jing Huang, Dan Jurafsky, Christopher D Manning, and Christopher Potts. Axbench: Steering LLMs? even simple baselines outperform sparse autoencoders. In Forty-second International Conference on Machine Learning, 2025b. URL https://openreview.net/forum?id=K2CckZjNy0.
- Aaron Mueller, Atticus Geiger, Sarah Wiegreffe, Dana Arad, Iván Arcuschin, Adam Belfki, Yik Siu Chan, Jaden Fried Fiotto-Kaufman, Tal Haklay, Michael Hanna, Jing Huang, Rohan Gupta, Yaniv Nikankin, Hadas Orgad, Nikhil Prakash, Anja Reusch, Aruna Sankaranarayanan, Shun Shao, Alessandro Stolfo, Martin Tutek, Amir Zur, David Bau, and Yonatan Belinkov. MIB: A mechanistic interpretability benchmark. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=sSrOwve6vb.
Also discussed in: §7.5.3 Step 2: Establish baselines
Other methods in Comparison with Alternative Feature Extraction Methods
Non-Negative Matrix Factorisation (NMF) · Independent Component Analysis (ICA) · Supervised Feature Extraction Methods (DAS, SMDS, probes) · Dictionary Learning Baselines (K-SVD)