Cross-Layer Transcoders (CLTs)

Assumptions and requirements

Attention freezing: By caching attention patterns, the replacement model treats attention as a fixed routing operation rather than a dynamic, input-dependent computation.

Normalisation caching: CLT analysis caches the input-dependent normalisation factors (e.g., LayerNorm denominators) from the original forward pass and reuses them during attribution instead of recomputing them after interventions.

Sparse linear approximation: Dictionary learning methods assume MLP outputs or transformations are well approximated by sparse linear combinations of features. This assumption may fail when computations involve complex feature interactions or highly nonlinear transformations.

Evaluation considerations

Under these conditions, the local replacement model can exactly match the original model’s activations on that prompt while decomposing it into interpretable CLT features. However, when the model’s activations are completely replaced by CLT reconstructions, the intervened model’s top-1 predictions match the original model’s on only about 50% of prompts from the Pile dataset (Gao et al., 2020). Notably, this experiment uses a relatively small, 18-layer transformer (Ameisen et al., 2025).

The paper reports top-1 token agreement as the main result, while softer metrics (e.g., top-5 agreement, KL divergence) may show different patterns and insights.

Domain: Fidelity on factual recall and syntactic tasks is higher than on complex reasoning or arithmetic tasks, where MLP computations may be more complex and cannot be approximated by sparse linear features. These caveats do not invalidate CLTs as a tool for mechanistic analysis, but they do mean that fidelity should be verified on the target model and specific task before drawing strong conclusions from CLT-based circuit analysis.

Intended applications

A key insight is that CLTs enable the construction of a replacement model—a version of the original model in which all MLP layers are replaced with their CLT reconstructions.

Unlike per-layer transcoders, CLTs reveal direct feature-to-feature interactions across layers: a feature in layer $\ell$ directly influences features in later layers through its contribution to their inputs. This simplifies circuit structure—features need not re-encode information at each layer.

CLTs provide a powerful lens for circuit discovery when these assumptions hold.

Limitations

When they fail, CLT-based explanations may become unfaithful.

Training a suite of CLTs thus incurs a high upfront cost.

For these reasons, current applications of CLTs remain limited: a “greater-than” mechanism study (Merullo et al., 2025), an open-source extension (Lindsey et al., 2025a), and analysis of multilingual representations using CLTs (Harrasse et al., 2026). This scarcity of downstream use cases likely reflects the considerable challenges of training CLTs at scale and the complexity of using CLT-based circuit discovery in the wild.

Cross-layer transcoders span multiple layers for global information flow but are extremely expensive.

Related methods

• Transcoders
• Sparse Autoencoders (SAEs)
• Crosscoders
• Circuit Tracing (SAE- or CLT-Based Circuit Discovery)
• Attention-Sparse Training (SPARTAN)

Compare side by side →

Key papers

Cited in §6.6.2.

• ■ PreprintEmmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L. Turner, Brian Chen, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. Circuit tracing: Revealing computational graphs in language models. Transformer Circuits Thread, 2025. URL https://transformer-circuits.pub/2025/attribution-graphs/methods.html.
• ■ PreprintLeo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020. URL https://arxiv.org/abs/2101.00027.
• ■ PreprintJack Merullo, Connor Watts, Max Loeffler, Liv Gorton, Elana Simon, Tom McGrath, and Owen Lewis. Replicating circuit tracing for a simple known mechanism. https://www.goodfire.ai/research/replicating-circuit-tracing-for-a-simple-mechanism, 2025.
• ■ PreprintJack Lindsey, Emmanuel Ameisen, Neel Nanda, Stepan Shabalin, Mateusz Piotrowski, Tom McGrath, Michael Hanna, Owen Lewis, Curt Tigges, Jack Merullo, Connor Watts, Gonçalo Paulo, Joshua Batson, Liv Gorton, Elana Simon, Max Loeffler, Callum McDougall, and Johnny Lin. The circuits research landscape: Results and perspectives. Neuronpedia, 2025a. URL https://neuronpedia.org/graph/info.
• ■ PreprintAbir Harrasse, Florent Draye, Punya Syon Pandey, Zhijing Jin, and Bernhard Schölkopf. Tracing multilingual representations in llms with cross-layer transcoders, 2026. URL https://arxiv.org/abs/2511.10840.

Other methods in Multi-Layer Feature Extraction and Analysis: Transcoders and Crosscoders

Transcoders · Crosscoders