Methods · Training Data Attribution
Kronecker-Factored Approximate Curvature (K-FAC)
Survey sections: §11.2.3 Kronecker-Factored Approximate Curvature (K-FAC)
Martens & Grosse (2015) proposed Kronecker-Factored Approximate Curvature (K-FAC) to approximate a large Fisher information matrix (FIM) using a set of smaller matrices.
Assumptions and requirements
Although K-FAC approximates the FIM rather than the GGN, these matrices are equivalent for common loss functions under suitable probabilistic assumptions.
The approximation separates the expectation into two independent factors: activations and pseudo-gradients.
Intended applications
G. This shows that we can use K-FAC to approximate the GGN used in the PBRF influence score.
Related methods
Key papers
Cited in §11.2.3.
- James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 2408–2417, Lille, France, 07–09 Jul 2015. PMLR. URL https://proceedings.mlr.press/v37/martens15.html.
- PreprintRoger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamilė Lukošiūtė, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. Studying large language model generalization with influence functions, 2023. URL https://arxiv.org/abs/2308.03296.
Also discussed in: §11.2.3 Addressing the Computational and Memory Bottlenecks of GGN · §11.2.3 Eigenvalue-Corrected K-FAC (EK-FAC) and ASTRA
Other methods in Training Data Attribution
Verbatim Data Attribution: OLMoTrace · Influence Functions (Gradient-Based Attribution) · Eigenvalue-Corrected K-FAC (EK-FAC) and ASTRA · TRAK (Tracing with the Randomly-projected After Kernel) · TracIn