Methods · Training Data Attribution
Verbatim Data Attribution: OLMoTrace
Survey sections: §11.1 Verbatim Data Attribution: OLMoTrace and Maximal Matching Spans · §11.1 Limitations as a Mechanistic Interpretability Tool
Verbatim data attribution detects explicit memorisation by identifying exact or near-exact string matches between model outputs and the training corpus. These searches can scale to trillions of tokens using techniques such as suffix arrays and infini-gram models.
Assumptions and requirements
OLMoTrace (Liu et al., 2025a) scales this to trillions of tokens using a combination of suffix arrays, infini-gram models, and maximal matching span extraction (Liu et al., 2024a), making it less a data-attribution method in the influence-function sense and more a retrieval tool for identifying content the model may have memorised.
OLMoTrace identifies maximal matching spans by combining suffix-array search with infini-gram queries, adjusting boundaries for tokenisation, and retaining only the longest non-overlapping spans.
Intended applications
“Eurovision Song Contest with the song ’Ne” This process links language model output directly to the training corpus, enabling identification and attribution of memorised content.
UTF8gbsnSpan: “她喜欢阅读关于科技、心理学和历史的书籍，业余时间还会参加编程课”—matched in post-training (instruction-tuning) data This example demonstrates how OLMoTrace traces multilingual outputs across diverse training sources, linking model generations directly to their memorised origins in pretraining and post-training datasets.
Limitations
Verbatim attribution excels at factual memorisation but struggles with compositional or synthesised outputs.
For tasks like translation, poetry, or reasoning, OLMoTrace (and other verbatim matching methods) can identify spans of text that appear in the training data, but these matches do not explain how the model combined information from multiple sources to produce the output.
Related methods
Key papers
Cited in §11.1.
- Aleksandra Piktus, Christopher Akiki, Paulo Villegas, Hugo Laurençon, Gérard Dupont, Sasha Luccioni, Yacine Jernite, and Anna Rogers. The ROOTS search tool: Data transparency for LLMs. In Danushka Bollegala, Ruihong Huang, and Alan Ritter (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pp. 304–314, Toronto, Canada, July 2023a. Association for Computational Linguistics. . URL https://aclanthology.org/2023.acl-demo.29/.
- Aleksandra Piktus, Odunayo Ogundepo, Christopher Akiki, Akintunde Oladipo, Xinyu Zhang, Hailey Schoelkopf, Stella Biderman, Martin Potthast, and Jimmy Lin. Gaia search: Hugging face and pyserini interoperability for nlp training data exploration. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pp. 588–598, 2023b.
- Jiacheng Liu, Taylor Blanton, Yanai Elazar, Sewon Min, Yen-Sung Chen, Arnavi Chheda-Kothary, Huy Tran, Byron Bischoff, Eric Marsh, Michael Schmitz, et al. Olmotrace: Tracing language model outputs back to trillions of training tokens. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pp. 178–188, 2025a.
- Jiacheng Liu, Sewon Min, Luke Zettlemoyer, Yejin Choi, and Hannaneh Hajishirzi. Infini-gram: Scaling unbounded n-gram language models to a trillion tokens. In First Conference on Language Modeling, 2024a. URL https://openreview.net/forum?id=u2vAyMeLMm.
Also discussed in: §11 Training Data Attribution · §11.1 OLMoTrace Influential Samples Identified: · §11.2.7 Summary
Other methods in Training Data Attribution
Influence Functions (Gradient-Based Attribution) · Kronecker-Factored Approximate Curvature (K-FAC) · Eigenvalue-Corrected K-FAC (EK-FAC) and ASTRA · TRAK (Tracing with the Randomly-projected After Kernel) · TracIn