Companion notebooks

notebooks/01_core_hypotheses/01_linear_representation_hypothesis.ipynb

Features correspond to directions $\phi \in \mathbb{R}^d$ in activation space. The projection $\phi^\top h_l(x)$ approximately measures the intensity of that feature for input $x$, and features combine additively: $h_l(x) \approx \sum_i \alpha_i \phi_i$. This implies features can be detected via linear probes and added or removed via vect…

Survey section

§5.1 The Linear Representation Hypothesis

Models

gpt2

Dependencies

numpy, sklearn, torch, transformers

In the explorer

Probing Classifiers The Linear Representation Hypothesis

notebooks/01_core_hypotheses/02_superposition_hypothesis.ipynb

Model hidden states encode more features than they have dimensions. If there are $k$ features and $d$ dimensions with $k >> d$, models tend to reduce interference between features by representing them as nearly-orthogonal directions.

Survey section

§5.2 The Superposition Hypothesis

Models

None (toy or synthetic data)

Dependencies

matplotlib, numpy, torch

In the explorer

The Superposition Hypothesis

notebooks/01_core_hypotheses/03_privileged_access_hypothesis.ipynb

A model used to interpret itself can, under appropriate elicitation, access internal variables and intermediate computations that are not reliably recoverable from external behavioral analysis alone, providing a comparative advantage as an interpreter of its own behavior.

Survey section

§5.3 The Privileged Access Hypothesis

Models

meta-llama/Llama-3.1-8Bgated: meta-llama/Llama-3.1-8B

Dependencies

dotenv, huggingface_hub, model, numpy, torch, transformers, yaml

Needs

Hugging Face token

In the explorer

The Privileged Access Hypothesis

Colab note: it needs about 32 GB of GPU memory (stated in the notebook), more than a free Colab GPU offers.

notebooks/02_feature_extraction/01_sparse_autoencoders_basics.ipynb

Feature extraction identifies distributed features across neurons that correspond to coherent semantic units. Sparse autoencoders (SAEs) address polysemanticity (neurons encoding multiple features) by decomposing activations into sparse, interpretable features using an overcomplete basis ($m >> d$).

Survey section

§6.2 Sparse Autoencoders

Models

None (toy or synthetic data)

Dependencies

matplotlib, numpy, torch, tqdm

In the explorer

Sparse Autoencoders (SAEs), Gated SAEs, JumpReLU SAEs, Top-k SAEs, Batch Top-k SAEs

notebooks/02_feature_extraction/02_transcoders.ipynb

Standard Sparse Autoencoders (SAEs) excel at decomposing representations in the residual stream, where features propagate linearly. However, they struggle with nonlinear transformations like MLP layers and attention mechanisms.

Survey section

§6.6.1 Transcoders: Bridging the Nonlinearities

Models

None (toy or synthetic data)

Dependencies

matplotlib, numpy, torch, tqdm

In the explorer

Transcoders

notebooks/02_feature_extraction/03_crosscoders.ipynb

A crosscoder learns sparse features by reading activations from multiple layers simultaneously:

Survey section

§6.6.3 Crosscoders: Analysing Cross-Layer Features

Models

None (toy or synthetic data)

Dependencies

matplotlib, numpy, torch, tqdm

In the explorer

Crosscoders

notebooks/03_circuits/01_circuit_discovery.ipynb

Circuits are minimal subnetworks that implement specific behaviors. Circuit discovery aims to identify which components (attention heads, MLP neurons, features) are responsible for a particular computation. This is a minimal notebook as this field is large.

Survey section

§7.1 Circuit Discovery Methods

Learning outcomes

Covers:

• Activation Patching: Replace activations to measure importance of nodes
• Attribution Patching: Efficient gradient-based approximation for edges

Models

gpt2

Dependencies

matplotlib, numpy, torch, tqdm, transformer_lens

In the explorer

Activation Patching

notebooks/04_intermediate_computation/01_logit_lens.ipynb

The Logit Lens (nostalgebraist, 2020) reveals what a language model "believes" at each stage of processing. GPT's final prediction comes from projecting the last layer's hidden state into vocabulary space. The logit lens applies that same projection to every intermediate layer:

Survey section

§8.1 Logit Lens: Tracing Predictions Through Depth

Learning outcomes

• Understand *why* the logit lens works (residual stream architecture)
• Implement it from scratch with HuggingFace Transformers
• Visualize and interpret layer-wise predictions
• Analyze convergence with ranks and KL divergence
• Know the limitations and motivation for the Tuned Lens

Models

openai-community/gpt2, openai-community/gpt2-large

Dependencies

matplotlib, numpy, sae_lens, torch, transformers

In the explorer

Logit Lens

notebooks/04_intermediate_computation/02_tuned_lens.ipynb

The logit lens projects intermediate hidden states through the unembedding matrix to see what a model "believes" at each layer. However, it systematically underperforms because intermediate layers may encode information in a rotated or shifted basis that doesn't align with the unembedding matrix.

Survey section

§8.2 Tuned Lens: Learning Per-Layer Transforms

Learning outcomes

• Understand why learned transforms help (representation drift)
• Train a tuned lens from scratch (pedagogical)
• Load and use pre-trained tuned lens weights
• Compare logit lens vs tuned lens predictions
• Analyze what the learned transforms reveal

Models

openai-community/gpt2-large

Dependencies

datasets, matplotlib, numpy, torch, tqdm, transformers, tuned_lens

In the explorer

Tuned Lens

notebooks/04_intermediate_computation/03_jacobian_lens.ipynb

Prerequisites: 01_logit_lens.ipynb, 02_tuned_lens.ipynb

Survey section

§8.3 Jacobian Lens: Decoding Computational Dispositions

Models

Qwen/Qwen3.5-4B

Dependencies

jlens, torch, transformers

Needs

Downloads data at runtime

notebooks/04_intermediate_computation/04_patchscopes.ipynb

Patchscopes use the language model itself to decode and explain its intermediate representations. Instead of training external probes, we patch hidden states into prompts that ask the model to describe what it represents.

Survey section

§8.4 Patchscopes: Leveraging the Language Model Itself to Explain Intermediate Representations

Learning outcomes

• Understand the patchscopes framework and its formal configuration
• Implement patchscopes using forward hooks in PyTorch
• Decode token identity at each layer (logit lens as patchscope)
• Visualize how predictions evolve across layers
• Evaluate reliability against known ground truth via source-target layer sweeps

Models

gpt2-medium

Dependencies

matplotlib, numpy, torch, transformers

In the explorer

Patchscopes

notebooks/04_intermediate_computation/05_probing_classifiers.ipynb

Probing classifiers (probes) are trained to predict properties from neural network representations. If a linear probe can accurately classify some property from activations, that property is linearly represented in the model's internal state.

Survey section

§8.6 Probing Classifiers: Capabilities, Critiques, and Causal Limitations

Learning outcomes

• Extract hidden state activations and visualize truth representations with PCA
• Understand which layers best represent truth via a layer sweep
• Train and compare difference-of-means (MM) and logistic regression (LR) probes
• Use control tasks and selectivity to validate probe results
• Implement causal interventions (activation patching) to prove representations are causally used

Models

Qwen/Qwen2.5-0.5B-Instruct, gpt2

Dependencies

matplotlib, numpy, pandas, sklearn, torch, tqdm, transformers

Needs

Downloads data at runtime

In the explorer

Probing Classifiers, Amnesic Probing

notebooks/04_intermediate_computation/06_activation_oracles.ipynb

Activation Oracles (AOs) are LLMs fine-tuned to accept another model's internal activation vectors as input and answer arbitrary natural-language questions about them. They build directly on LatentQA (Pan et al., 2024), which formulates representation decoding as a supervised learning problem: given a cached hidden representation $h^l_t$,…

Survey section

§8.5 LatentQA: Fine-Tuning the Language Model as an Activation Verbaliser

Learning outcomes

• Understand what Activation Oracles are and how they differ from patchscopes, probes, and SAEs
• Use a pre-trained oracle to query model internals at token, segment, and sequence level
• Implement the full oracle pipeline from scratch: activation extraction, placeholder tokens, steering hooks
• Extract secret words from models fine-tuned to hide them (taboo task)
• Use model-diffing (activation differences) to detect what fine-tuning changed

Models

Qwen/Qwen3-8B

Dependencies

pandas, peft, plotly, torch, tqdm, transformers

In the explorer

LatentQA

Colab note: the models it loads (about 16 GB of weights in bf16) need more memory than a free Colab T4 (15 GB).

notebooks/04_intermediate_computation/07_natural_language_autoencoders.ipynb

Prerequisites: 01_logit_lens.ipynb, 06_activation_oracles.ipynb

Survey section

Natural Language Autoencoders — dedicated handbook section coming soon

Models

Qwen/Qwen2.5-7B-Instruct, kitft/nla-qwen2.5-7b-L20-ar, kitft/nla-qwen2.5-7b-L20-av

Dependencies

httpx, huggingface_hub, nla_inference, numpy, transformers

Needs

Downloads data at runtime

Colab note: it needs about 40 GB of GPU memory (stated in the notebook), more than a free Colab GPU offers.

notebooks/05_representation_analysis/01_model_diffing.ipynb

Model diffing is a class of techniques for understanding how two language models differ by analyzing their internal representations. Given a base model and a fine-tuned variant, we want to answer: *What changed? Which features are shared, and which are new or exclusive to one model?*

Survey section

§9.1 Model Diffing

Learning outcomes

• Understand why standard SAEs are insufficient for model comparison and how crosscoders solve the feature alignment problem
• Implement and understand the crosscoder architecture for joint feature learning across two models
• Load pre-trained crosscoder weights and classify features as shared vs. model-exclusive using decoder norm analysis
• Evaluate crosscoder reconstruction quality via the CE loss recovered metric
• Analyze feature density differences between shared and exclusive features

Models

gemma-2-2b, gemma-2-2b-itgated: gemma-2-2b, gemma-2-2b-it

Dependencies

datasets, dotenv, einops, huggingface_hub, numpy, plotly, torch, transformer_lens

Needs

Hugging Face token

In the explorer

Model Diffing, Crosscoders

notebooks/05_representation_analysis/02_knowledge_localization.ipynb

This notebook demonstrates causal tracing, a technique for localizing where factual knowledge is stored in transformer models. The method was introduced in "Locating and Editing Factual Associations in GPT".

Survey section

§9.2 Knowledge Localisation

Models

gpt2-medium

Dependencies

matplotlib, numpy, seaborn, torch, transformers

In the explorer

Causal Tracing (Knowledge Localization)

notebooks/05_representation_analysis/03_representation_engineering.ipynb

Representation engineering uses insights about model internals to control behavior. The key technique is steering vectors: adding carefully chosen vectors to activations to shift model behavior.

Survey section

§9.3 Representation Engineering and Steering

Models

gpt2, meta-llama/Llama-2-13b-chat-hfgated: meta-llama/Llama-2-13b-chat-hf

Dependencies

datasets, dotenv, huggingface_hub, matplotlib, numpy, sklearn, torch, tqdm, transformers

Needs

Hugging Face token

In the explorer

Representation Engineering and Steering (Steering Vectors)

Colab note: it needs about 26 GB of GPU memory (stated in the notebook), more than a free Colab GPU offers.

notebooks/06_safety_applications/01_concept_erasure.ipynb

*See also Section 13.2.2 (Knowledge Editing via Localisation) for the editing motivation.*

Survey section

§13.2.1 Concept Erasure via PISCES

Models

gpt2

Dependencies

numpy, sklearn, torch, transformers

notebooks/06_safety_applications/02_steering_vectors.ipynb

This notebook applies the workflow from `01_linear_representation_hypothesis.ipynb` to a safety setting.

Survey section

§13.3.1 Steering Vectors: Practical Behavioural Control

Learning outcomes

• Extract a refusal direction with difference-in-means
• Detect refusal intensity with projections and a linear probe
• Steer generation by adding the refusal vector at inference time
• Explore a simple multi-objective extension (refusal + honesty)

Models

gpt2

Dependencies

numpy, sklearn, torch, transformers

In the explorer

Representation Engineering and Steering (Steering Vectors)