How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining
ICLROral2026
TL;DR
Use model weight average to enhance curriculum learning in LLM pretraining.
Opening excerpt from the authors’ abstract. source
Read the paper
Topics
pretraining llm rag