Methods · Interpretability-by-Design Methods for Intrinsic Interpretability

Weight-Sparse Training (Weight-Sparse Transformer)

Survey sections: §6.10.2 Weight-Sparse Training

A weight-sparse transformer (Gao et al., 2026) treats weight sparsity as an explicit structural constraint that is progressively strengthened during training.
Survey §6.10.2 Weight-Sparse Training

Assumptions and requirements

Gao et al. (2026) optimise the standard cross-entropy objective along with an $L_0$ constraint on model weights after each optimisation step. Specifically, within each weight matrix, they retain only the largest-magnitude parameters and set the rest to zero. The $L_0$ constraint anneals from a dense initialisation to the desired sparsity level over training.

Evaluation considerations

Mean-ablation experiments further confirm that these circuits are both necessary and sufficient for the capability or behaviour.

Intended applications

The resulting transformer has most weight parameters exactly zero, making the remaining computational structure much easier to decompose into small, task-specific circuits.

Limitations

Increasing sparsity generally makes the model easier to decompose into small, interpretable circuits. However, it also degrades task performance, producing a capability–interpretability Pareto frontier.

At comparable task performance, larger sparse models can have substantially smaller, more interpretable circuits than smaller models; at comparable interpretability, they can retain stronger capabilities. This suggests that scaling model size alongside structural sparsity may offer a path to more interpretable yet high-capability models. However, preserving interpretability beyond tens of millions of nonzero parameters remains challenging, and weight-sparse training currently requires substantially more compute than traditional training.

Related methods

Compare side by side →

Key papers

Cited in §6.10.2.

Leo Gao, Achyuta Rajaram, Jacob Coxon, Soham V. Govande, Bowen Baker, and Daniel P Mossing. Weight-sparse transformers have interpretable circuits. In Forty-third International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id=Vw10TpuFCp.

Also discussed in: §5.2 Architectural Responses to Superposition · §6.10.1 Feature-Sparse Training · §15.2 The Road Ahead

Other methods in Interpretability-by-Design Methods for Intrinsic Interpretability

Attention-Sparse Training (SPARTAN) · Softmax Linear Units (SoLU) · Sparse Mixture of Linear Transforms (MOLT) · Kolmogorov–Arnold Networks (KANs)