Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo

NeurIPSSpotlight2025

Authors: Zachary Charles, Gabriel Teston, Lucio M. Dery, J Keith Rush, Nova Fallen, Zachary Garrett, Arthur Szlam, Arthur Douillard
Venue: NeurIPS 2025
Track: Spotlight

TL;DR

As we scale to more massive machine learning models, the frequent synchronization demands inherent in data-parallel approaches create significant slowdowns, posing a critical challenge to further scal…

Opening excerpt from the authors’ abstract. source

Read the paper

OpenReview

Topics

language model scaling law efficient

← All NeurIPS 2025 Spotlight papers · Browse the whole archive