Methods · Understanding Intermediate Representations
Tuned Lens
Survey sections: §8.2 Tuned Lens: Learning Per-Layer Transforms · §8.2 Limitations of Tuned Lens
Summary Tuned lens extends logit lens by learning, for each layer, an affine transformation that maps intermediate activations into the output layer’s activation space before the unembedding matrix is applied, addressing the representational mismatch between layers.
Tuned lens (Belrose et al., 2025) addresses logit lens’ concern of representational mismatch by learning per-layer affine transformations that map intermediate activations to the output layer’s activation space …
Intended applications
As a result, the unembedding matrix can extract relevant information from each layer’s activations more effectively.
Limitations
First, the decoding paradigm remains inherently correlational rather than causal; decodable information does not imply a causal role in the model’s prediction.
Second, similar to logit lens, tuned lens is primarily designed to answer what the model is likely to predict rather than what the model is internally computing.
Empirical evidence also shows that it skips intermediate computation more aggressively than logit lens (Gurnee et al., 2026).
Related methods
Run it
Key papers
Cited in §8.2.
- PreprintNora Belrose, Igor Ostrovsky, Lev McKinney, Zach Furman, Logan Smith, Danny Halawi, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens, 2025. URL https://arxiv.org/abs/2303.08112.
- PreprintWes Gurnee, Nicholas Sofroniew, Adam Pearce, Mateusz Piotrowski, Isaac Kauvar, Runjin Chen, Anna Soligo, Paul Bogdan, Euan Ong, Rowan Wang, Ben Thompson, David Abrahams, Subhash Kantamneni, Emmanuel Ameisen, Joshua Batson, and Jack Lindsey. Verbalizable representations form a global workspace in language models. Transformer Circuits Thread, 2026. URL https://transformer-circuits.pub/2026/workspace/index.html.
Also discussed in: §8.3 Jacobian Lens: Decoding Computational Dispositions · §8.4 Patchscopes: Leveraging the Language Model Itself to Explain Intermediate Representations · §8.6.7 Summary · §C.3 KL Divergence · §14.7 Transferability Beyond Transformers
Other methods in Understanding Intermediate Representations
Probing Classifiers · Logit Lens · Patchscopes · LatentQA · Amnesic Probing