Methods · Learning Dynamics and Mechanistic Interpretability
Forecasting from Smaller and Partial Runs
Survey sections: §10.4.1 Forecasting from Smaller Runs and Partial Runs
Summary This approach forecasts a specific behaviour of a model before the full training run, such as whether a training sequence will be memorised, using smaller fully trained models or early checkpoints of the target run.
Biderman et al. (2023a) introduced the problem of forecasting whether a specific training sequence will be memorised by a model before that model is trained.
Assumptions and requirements
Scaling laws demonstrate that aggregate loss is predictable before training (Kaplan et al., 2020; Hoffmann et al., 2022), but phase transitions break this style of extrapolation for individual capabilities (Section 10.3). Forecasting therefore targets specific behaviours directly.
Evaluation considerations
They argued that recall is the metric to optimise (a false negative means training a full model that leaks the sequence, while a false positive only discards a cheap trial run).
To assess the reliability of a forecast, researchers can compare training runs with different random seeds.
Intended applications
They established baselines that forecast from smaller fully trained models and from partially trained checkpoints of the target run, together with equi-compute recommendations for dividing a fixed forecasting budget between model size and tokens.
Limitations
van der Wal et al. (2025) found that capabilities generally emerge at similar stages and representations develop consistently across training runs, although some runs differ substantially. These exceptions highlight the uncertainty of forecasts based on a single training run.
However, prospective forecasting for broader capability transitions relevant to AI safety remains underdeveloped.
Related methods
Key papers
Cited in §10.4.1.
- PreprintJared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. URL https://arxiv.org/abs/2001.08361.
- Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre. Training compute-optimal large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088.
- Stella Biderman, Usvsn Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raff. Emergent and predictable memorization in large language models. Advances in Neural Information Processing Systems, 36: 28072–28090, 2023a.
Also discussed in: §10.4.1 Run-to-Run Variance
Other methods in Learning Dynamics and Mechanistic Interpretability
Trajectory-Based Attribution · Per-Token Learning Curves · Refined Local Learning Coefficients (Localising Change to Components) · Forked-Run Experiments and Linear Mode Connectivity · Loss-Curve Derivatives · Capability-Specific Metrics · Per-Example Decomposition (POLCA) · Latent State Models · Progress Measures · Local Learning Coefficient (LLC) · Forecasting Safety-Relevant Behaviours · Internal Precursors as Early Warning · Pretraining Data Filtering (Deep Ignorance) · Gradient Routing · Monitoring and Steering During Fine-Tuning (Persona Vectors) · Train-Time Intervention Loop