Data Mixing Can Induce Phase Transitions in Knowledge Acquisition

TL;DR

Large Language Models (LLMs) are typically trained on data mixtures: most data come from web scrapes, while a small portion is curated from high-quality sources with dense domain-specific knowledge…

Opening excerpt from the authors’ abstract. source