EvoLM: In Search of Lost Training Dynamics for Language Model Reasoning

NeurIPSOral2025

Authors
Zhenting Qi, Fan Nie, Alexandre Alahi, James Zou, Himabindu Lakkaraju, Yilun Du, Eric P. Xing, Sham M. Kakade, Hanlin Zhang
Affiliation
Harvard University
Venue
NeurIPS 2025
Track
Oral

TL;DR

Modern language model (LM) training has been divided into multiple stages, making it difficult for downstream developers to evaluate the impact of design choices made at each stage. We present EvoLM, a model suite that enables systematic and transparent analysis of LMs' training dynamics across pre-training, continued pre-training, supervised fine-…

Opening excerpt from the authors’ abstract. source

Read the paper

Topics

language model pre-training reasoning

Related NeurIPS papers

← All NeurIPS 2025 Oral papers · Browse the whole archive