Methods · Training Data Attribution
Eigenvalue-Corrected K-FAC (EK-FAC) and ASTRA
Survey sections: §11.2.3 Eigenvalue-Corrected K-FAC (EK-FAC) and ASTRA
EK-FAC (George et al., 2018) improved K-FAC by correcting its eigenvalues. Recent work has adopted this technique to scale the influence function method to modern LLMs (Grosse et al., 2023).
Intended applications
ASTRA (Wang et al., 2025a) further used EK-FAC as a preconditioner for Neumann series iterations, achieving higher-quality GGN-vector products with less hyperparameter tuning. These efforts make influence functions increasingly practical for modern language models with billions of parameters.
Related methods
Key papers
Cited in §11.2.3.
- Thomas George, César Laurent, Xavier Bouthillier, Nicolas Ballas, and Pascal Vincent. Fast approximate natural gradient descent in a kronecker-factored eigenbasis. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18, pp. 9573–9583, Red Hook, NY, USA, 2018. Curran Associates Inc.
- PreprintRoger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamilė Lukošiūtė, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. Studying large language model generalization with influence functions, 2023. URL https://arxiv.org/abs/2308.03296.
- Andrew Wang, Elisa Nguyen, Runshi Yang, Juhan Bae, Sheila A. McIlraith, and Roger Baker Grosse. Better training data attribution via better inverse hessian-vector products. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025a. URL https://openreview.net/forum?id=7LTTzYXyJ1.
Other methods in Training Data Attribution
Verbatim Data Attribution: OLMoTrace · Influence Functions (Gradient-Based Attribution) · Kronecker-Factored Approximate Curvature (K-FAC) · TRAK (Tracing with the Randomly-projected After Kernel) · TracIn