Methods · Interpretability-by-Design Methods for Intrinsic Interpretability

Attention-Sparse Training (SPARTAN)

Survey sections: §6.10.3 Attention-Sparse Training

Summary Attention-sparse training sparsifies attention patterns, which are usually diffuse and hard to interpret, so that each attention head attends only to specific tokens while model performance stays close to the original.

Previous studies have therefore explored attention-sparse training, which aims to sparsify attention patterns to make them more interpretable (Lei et al., 2025; Draye et al., 2026). Specifically, Lei et al. (2025) proposed SPARse TrANsformer World model (SPARTAN), an object-centric world model trained with $L_0$ regularisation on attention patterns and a constrained optimisation objective to maintain cross-entropy performance equal to that of a dense counterpart.

Intended applications

After training (either from scratch or via fine-tuning), each attention head attends only to specific tokens, and the model performance remains close to the original. This result suggests that many attention patterns may be redundant noise.

In CLT-based circuit discovery (see Section 7.2), Draye et al. (2026) showed that sparsified attention patterns reduce the number of mediating components involved in feature-to-feature attribution, making the resulting attribution graphs easier to analyse.

Related methods

• Cross-Layer Transcoders (CLTs)

Compare side by side →

Key papers

Cited in §6.10.3.

• ■ Anson Lei, Bernhard Schölkopf, and Ingmar Posner. Spartan: A sparse transformer world model attending to what matters. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
• ■ PreprintFlorent Draye, Anson Lei, Hsiao-Ru Pan, Ingmar Posner, and Bernhard Schölkopf. Intrinsically interpretable attention via sparse post-training, 2026. URL https://arxiv.org/abs/2512.05865.

Other methods in Interpretability-by-Design Methods for Intrinsic Interpretability

Weight-Sparse Training (Weight-Sparse Transformer) · Softmax Linear Units (SoLU) · Sparse Mixture of Linear Transforms (MOLT) · Kolmogorov–Arnold Networks (KANs)