Overparametrization bends the landscape: BBP transitions at initialization in simple Neural Networks

TL;DR

We quantitatively analyze how overparametrization reshapes the high-dimensional loss landscape of a teacher–student setup in random positions, showing it can anticipate and qualitatively alter transitions between successful and failed signal recovery…

Opening excerpt from the authors’ abstract. source