Methods · Comparison with Alternative Feature Extraction Methods
Dictionary Learning Baselines (K-SVD)
Survey sections: §6.9.5 Dictionary Learning Baselines
Classical dictionary learning methods provide useful baselines for evaluating SAEs. Techniques such as K-SVD alternate between sparse coding and dictionary updates and converge under certain conditions (Aharon et al., 2006).
Evaluation considerations
Comparative studies suggest that SAEs’ primary advantage is fast inference because the encoder learns to decompose all activations, while potentially sacrificing optimal sparsity (O’Neill et al., 2025).
Intended applications
As such, classical dictionary learning can still achieve superior dictionary quality in some cases and thus remains an important reference point for assessing the added value of SAEs.
Related methods
Key papers
Cited in §6.9.5.
- Michal Aharon, Michael Elad, and Alfred Bruckstein. K-svd: An algorithm for designing overcomplete dictionaries for sparse representation. IEEE Transactions on signal processing, 54 (11): 4311–4322, 2006.
- Charles O’Neill, Alim Gumran, and David Klindt. Compute optimal inference and provable amortisation gap in sparse autoencoders. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=8forr1FkvC.
Also discussed in: §6.1 The Sparse Coding Problem
Other methods in Comparison with Alternative Feature Extraction Methods
Non-Negative Matrix Factorisation (NMF) · Principal Component Analysis (PCA) · Independent Component Analysis (ICA) · Supervised Feature Extraction Methods (DAS, SMDS, probes)