Implicit Bias of Spectral Descent and Muon on Multiclass Separable Data

TL;DR

Different gradient-based methods for optimizing overparameterized models can all achieve zero training error yet converge to distinctly different solutions inducing different generalization properties…

Opening excerpt from the authors’ abstract. source