Crosscoders

Intended applications

By extracting cross-layer features, crosscoders help us understand how features propagate through the residual stream across layers, removing duplicate feature representations that complicate circuit analysis.

Crosscoders can define model-specific encoders with arbitrarily different input dimensions that project activations from different models into one or more shared representational spaces.

Limitations

Dictionary learning methods for model diffing risk falsely identifying model-exclusive features (Dumas et al., 2025a; Jiralerspong & Bricken, 2025). This can result from the loss function: for instance, Dumas et al. (2025a) showed that crosscoders trained with an $L_1$ regularisation on hidden states can falsely identify novel features, which they tried to mitigate with $\sigma_{\text{Batch-TopK}}(\cdot)$.

Related methods

• Sparse Autoencoders (SAEs)
• Transcoders
• Cross-Layer Transcoders (CLTs)
• Model Diffing
• Batch Top-k SAEs

Compare side by side →

Run it

• Notebook: Crosscoders: Analyzing Cross-Layer Features · Open in Colab
• Notebook: Model Diffing with Crosscoders · Open in Colab

Key papers

Cited in §6.6.3.

• ■ PreprintJack Lindsey, Adly Templeton, Jonathan Marcus, Thomas Conerly, Joshua Batson, and Christopher Olah. Sparse crosscoders for cross‑layer features and model diffing. Anthropic Transformer Circuits Thread, 2024. URL https://transformer-circuits.pub/2024/crosscoders/index.html.
• ■ Xuyang Ge, Wentao Shu, Jiaxing Wu, Yunhua Zhou, Zhengfu He, and Xipeng Qiu. Evolution of concepts in language model pre-training. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=qKmXz1UR0Z.
• ■ Deniz Bayazit, Aaron Mueller, and Antoine Bosselut. Crosscoding through time: Tracking emergence & consolidation of linguistic representations throughout LLM pretraining. In Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jurgens (eds.), Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1353–1377, San Diego, California, United States, July 2026. Association for Computational Linguistics. ISBN 979-8-89176-390-6. . URL https://aclanthology.org/2026.acl-long.60/.
• ■ PreprintThomas Jiralerspong and Trenton Bricken. Cross-Architecture Model Diffing with Crosscoders: Unsupervised Discovery of Differences Between LLMs. In Workshop on Mechanistic Interpretability, September 2025. URL https://openreview.net/forum?id=ZB84SvrZB8.

Other methods in Multi-Layer Feature Extraction and Analysis: Transcoders and Crosscoders

Transcoders · Cross-Layer Transcoders (CLTs)