Methods · Understanding Intermediate Representations
LatentQA
Survey sections: §8.5 LatentQA: Fine-Tuning the Language Model as an Activation Verbaliser · §8.5 Limitations of LatentQA
LatentQA (Pan et al., 2026) frames representation decoding as a supervised learning problem via Latent Interpretation Tuning (LIT), fine-tuning an activation verbaliser to perform representation-to-text verbalisation.
Assumptions and requirements
Through LIT, the verbaliser learns to ingest and process raw activation vectors directly.
Evaluation considerations
Another technical challenge is evaluating whether activation-verbalisation models genuinely explain the represented information rather than exploit shortcuts.
Intended applications
LatentQA (Figure 12) addresses the OOD issue inherent in Patchscopes-style methods by explicitly fine-tuning the activation verbaliser on the representation-to-text decoding task.
The process involves (1) curating a dataset of activation–question–answer triplets from a target LLM, and (2) fine-tuning a verbaliser to retrieve specific information latent within those activations, such as the model’s underlying goals or persona.
Limitations
LatentQA faces several technical challenges, including the generalisability of the learned representation-to-text mapping.
When the fine-tuning dataset D is too small or insufficiently diverse, the verbaliser may memorise superficial correspondences between representations and explanations rather than learning to extract information from representations.
Recent studies have found that such failure modes are severe and prevalent in existing verbalisers (Li et al., 2026c; Jakkli et al., 2026). As a result, strong verbalisation performance alone is insufficient to show that the verbaliser faithfully analysed the internal representations.
Related methods
Run it
Key papers
Cited in §8.5.
- Alexander Pan, Lijie Chen, and Jacob Steinhardt. LatentQA: Teaching LLMs to decode activations into natural language. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=niUroX9EOd.
- PreprintVincent Huang, Dami Choi, Daniel D. Johnson, Sarah Schwettmann, and Jacob Steinhardt. Predictive concept decoders: Training scalable end-to-end interpretability assistants. In Mechanistic Interpretability Workshop at ICML 2026, 2026. URL https://openreview.net/forum?id=s9IaugZ0En.
- Adam Karvonen, James Chua, Clément Dumas, Kit Fraser-Taliente, Subhash Kantamneni, Julian Minder, Euan Ong, Arnab Sen Sharma, Daniel Wen, Owain Evans, and Samuel Marks. Activation oracles: Training and evaluating LLMs as general-purpose activation explainers. In Forty-third International Conference on Machine Learning, 2026a. URL https://openreview.net/forum?id=DbZjxkZrZm.
- Millicent Li, Alberto Mario Ceballos Arroyo, Giordano Rogers, Naomi Saphra, and Byron C Wallace. Do activation verbalization methods convey privileged information? In Forty-third International Conference on Machine Learning, 2026c. URL https://openreview.net/forum?id=IFGTKwHwK6.
- PreprintArya Jakkli, Senthooran Rajamanoharan, and Neel Nanda. Current activation oracles are hard to use on safety-relevant tasks. In Mechanistic Interpretability Workshop at ICML 2026, 2026. URL https://openreview.net/forum?id=7nRmqgz3Wv.
Also discussed in: §8.6.7 Summary · §12.7 Comparative Analysis: Trade-Offs Between Orientations · §15.2 The Road Ahead
Other methods in Understanding Intermediate Representations
Probing Classifiers · Logit Lens · Tuned Lens · Patchscopes · Amnesic Probing