Methods · Learning Dynamics and Mechanistic Interpretability
Per-Example Decomposition (POLCA)
Survey sections: §10.3.1 Per-Example Decomposition
Transitions invisible in every predefined metric can be recovered from the training data itself. POLCA decomposes changes in per-example loss along directions in the low-dimensional subspace where training occurs and clusters examples with similar loss trajectories; the clusters correspond to concepts learned abruptly at identifiable points in training, with no corresponding event in the aggregate curve (Kangaslahti et al., 2026).
Related methods
Key papers
Cited in §10.3.1.
- Sara Kangaslahti, Elan Rosenfeld, and Naomi Saphra. Hidden breakthroughs in language model training. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=eiEIKGuqaf.
- Michael Y. Hu, Angelica Chen, Naomi Saphra, and Kyunghyun Cho. Latent state models of training dynamics. Transactions on Machine Learning Research, 2023.
- Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? In Advances in Neural Information Processing Systems, volume 36, 2023. URL https://arxiv.org/abs/2304.15004.
- Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022. URL https://openreview.net/forum?id=yzkSU5zdwD.
- Zhengxiao Du, Aohan Zeng, Yuxiao Dong, and Jie Tang. Understanding emergent abilities of language models from the loss perspective. In Advances in Neural Information Processing Systems, volume 37, 2024. URL https://openreview.net/forum?id=35DAviqMFo.
- Rylan Schaeffer, Hailey Schoelkopf, Brando Miranda, Gabriel Mukobi, Varun Madan, Adam Ibrahim, Herbie Bradley, Stella Biderman, and Sanmi Koyejo. Why has predicting downstream capabilities of frontier AI models with scale remained elusive? In Proceedings of the 42nd International Conference on Machine Learning, 2025. URL https://arxiv.org/abs/2406.04391.
- Tung-Yu Wu and Pei-Yu Lo. U-shaped and inverted-u scaling behind emergent abilities of large language models. In International Conference on Learning Representations, 2025. URL https://arxiv.org/abs/2410.01692.
Other methods in Learning Dynamics and Mechanistic Interpretability
Trajectory-Based Attribution · Per-Token Learning Curves · Refined Local Learning Coefficients (Localising Change to Components) · Forked-Run Experiments and Linear Mode Connectivity · Loss-Curve Derivatives · Capability-Specific Metrics · Latent State Models · Progress Measures · Local Learning Coefficient (LLC) · Forecasting from Smaller and Partial Runs · Forecasting Safety-Relevant Behaviours · Internal Precursors as Early Warning · Pretraining Data Filtering (Deep Ignorance) · Gradient Routing · Monitoring and Steering During Fine-Tuning (Persona Vectors) · Train-Time Intervention Loop