Methods · Understanding Intermediate Representations
Logit Lens
Survey sections: §8.1 Logit Lens: Tracing Predictions Through Depth · §8.1 Limitations of Vanilla Logit Lens
Logit lens (nostalgebraist, 2020) applies the model’s unembedding matrix to activations at intermediate layers, revealing what the model would predict if computation stopped at that layer.
Assumptions and requirements
The vanilla logit lens has a critical limitation: applying the unembedding matrix to intermediate activations assumes that the hidden state is in a format that the output layer expects. However, intermediate representations may be organised very differently from output-layer activations.
Intended applications
Applying it across layers reveals the prediction trajectory, showing when and where the model commits to specific outputs (nostalgebraist, 2020).
Logit lens (Figure 10) provides intuitive insight into the model’s decision-making process across depth.
Limitations
Applying the unembedding matrix to layer-6 activations may extract relevant information but cannot precisely reflect the model’s underlying computation at that layer (nostalgebraist, 2020). Furthermore, logit lens is fundamentally correlational rather than causal. It projects the activation into the output vocabulary space without testing the roles decoded tokens play in model computation. In particular, because of the representational mismatch discussed above, decoded tokens represent at most information recoverable from the activation, rather than information used in model computation.
Related methods
Run it
Key papers
Cited in §8.1.
Also discussed in: §8.2 Tuned Lens: Learning Per-Layer Transforms · §8.3 Jacobian Lens: Decoding Computational Dispositions · §8.4 Patchscopes: Leveraging the Language Model Itself to Explain Intermediate Representations · §8.6.7 Summary · §10.1.2 Feature Formation, Consolidation, and Drift
Other methods in Understanding Intermediate Representations
Probing Classifiers · Tuned Lens · Patchscopes · LatentQA · Amnesic Probing