Methods · Understanding Intermediate Representations
Patchscopes
Survey sections: §8.4 Patchscopes: Leveraging the Language Model Itself to Explain Intermediate Representations · §8.4 Limitations of Patchscopes
Patchscopes (Ghandeharioun et al., 2024) leverages the expressive capacity of language models to translate their internal representations into natural-language explanations.
Assumptions and requirements
In this view, the language model acts as an activation decoder. Crucially, Patchscopes relies on the model’s learned, generalisable text-generation capabilities to map internal activations to natural language.
Intended applications
By observing changes in model output, users can examine what information the representation contains.
Limitations
A key limitation of Patchscopes is that injecting an external representation into an elicitation prompt can shift the model’s internal activations into an out-of-distribution (OOD) regime.
Although the model may still surface salient information from the patched representation, the decoding process can degrade, and critical aspects of the original representation may be distorted or lost.
Related methods
Run it
Key papers
Cited in §8.4.
- Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. Patchscopes: a unifying framework for inspecting hidden representations of language models. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024.
- Haozhe Chen, Carl Vondrick, and Chengzhi Mao. Selfie: self-interpretation of large language model embeddings. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024b.
Also discussed in: §8.5 LatentQA: Fine-Tuning the Language Model as an Activation Verbaliser · §8.6.7 Summary
Other methods in Understanding Intermediate Representations
Probing Classifiers · Logit Lens · Tuned Lens · LatentQA · Amnesic Probing