Efficient Resource-Constrained Training of Transformers via Subspace Optimization

TL;DR

We propose a novel method that enables training vision transformer models within a low-rank subspace to optimize computational resources, making on-device learning practically feasible.

Opening excerpt from the authors’ abstract. source