Generalized Linear Mode Connectivity for Transformers

TL;DR

We propose a unified framework for model merging that leverages multiple symmetry classes to enable low- and zero-loss interpolation between independently trained Transformer models, including Vision Transformers and GPT-2.

Opening excerpt from the authors’ abstract. source