A Notation Reference and Symbol Table#PDF p. 178

$d$: residual-stream dimension; equivalently model hidden size

$d_{\text{head}}$: attention head dimension

$d_{\text{mlp}}$: MLP hidden dimension, typically $d_{\text{mlp}} \gg d$

$N_H$: number of attention heads per layer

$d_{\text{sparse}}$: dictionary size of an SAE or transcoder, $d_{\text{sparse}} = e\,d$ with integer expansion factor $e$

$L$: number of layers (model depth); $\ell \in \{1,\dots,L\}$ indexes layers

$T$: sequence length (context length); $t \in \{1,\dots,T\}$ indexes token positions

$O(\cdot)$: Big-O asymptotic complexity notation

$x$: an input prompt or a set of input prompts

$h^{1:T}_\ell \in \mathbb{R}^{d\times T}$: residual-stream activations at layer $\ell$ for token positions $1$ to $T$, whose $t$-th column is $h^t_\ell$

$h_\ell^t \in \mathbb{R}^{d}$: residual-stream activation at layer $\ell$ for token position $t$

$h_\ell \in \mathbb{R}^{d}$: residual-stream activation at layer $\ell$ with token position omitted

$h \in \mathbb{R}^{d}$: residual-stream activation with both layer and position omitted

$h'_\ell \in \mathbb{R}^{d}$: mid-block residual stream (post-attention, pre-MLP) at layer $\ell$

$h_\ell^{\text{in}} \in \mathbb{R}^{d}$, $h_\ell^{\text{out}} \in \mathbb{R}^{d}$: input and output of a model component, e.g., an MLP block, at layer $\ell$

$\hat{h} \in \mathbb{R}^{d}$: reconstruction of activations $h$ by a dictionary model

$\tilde{h} \in \mathbb{R}^{d}$: activation after an intervention (steering, ablation, patching, etc.)

$E : \mathbb{R}^{d} \to \mathbb{R}^{d_{\text{sparse}}}$: SAE encoder

$D : \mathbb{R}^{d_{\text{sparse}}} \to \mathbb{R}^{d}$: SAE decoder

$z \in \mathbb{R}^{d_{\text{sparse}}}$: sparse feature activations; $z^i \in \mathbb{R}$ is the activation of feature $i$; when a layer index is also present, we write $z_\ell^i$ for feature $i$ at layer $\ell$

$f_j \in \mathbb{R}^{d}$: feature direction in activation space (the $j$-th column of $W_{\text{out}}$)

$W_{\text{in}} \in \mathbb{R}^{d_{\text{sparse}} \times d}$, $b_{\text{in}} \in \mathbb{R}^{d_{\text{sparse}}}$: encoder weights and bias

$W_{\text{out}} \in \mathbb{R}^{d \times d_{\text{sparse}}}$, $b_{\text{out}} \in \mathbb{R}^{d}$: decoder weights and bias

$k$: number of active features in a Top-$k$ SAE

$W_Q^r, W_K^r, W_V^r \in \mathbb{R}^{d_{\text{head}} \times d}$, $W_O^r \in \mathbb{R}^{d \times d_{\text{head}}}$: query, key, value, and output projection matrices of the $r$-th attention head

$W_1 \in \mathbb{R}^{d_{\text{mlp}} \times d}$, $W_2 \in \mathbb{R}^{d \times d_{\text{mlp}}}$: MLP weight matrices

$b_1 \in \mathbb{R}^{d_{\mathrm{mlp}}}$, $b_2 \in \mathbb{R}^{d}$: MLP bias vectors

$\theta$: the collection of all trainable model parameters

$v_{\text{steer}} \in \mathbb{R}^{d}$: steering vector, applied as $h + \alpha\, v_{\text{steer}}$

$\alpha$: scaling factor for an intervention

$\mathcal{L}$: loss function

$\lambda$: sparsity or regularisation coefficient. In Appendix C.2, $\lambda(\cdot)$ instead denotes the local learning coefficient

$p(y \mid x)$, $\log p(y \mid x)$: conditional probability and log probability

$\mathbb{E}[\cdot]$: expectation over the data distribution; $\mathbb{E}[\cdot \mid \text{condition}]$ denotes a conditional expectation

$\|v\|$: Euclidean norm; $\|z\|_1$ and $\|z\|_0$ denote the $L_1$ and $L_0$ norms

$\langle v, w \rangle$: inner product

$\odot$: element-wise product

$\otimes$: Kronecker product

$\sigma(\cdot)$: activation function