A multiscale analysis of mean-field transformers in the moderate interaction regime

TL;DR

In this paper, we study the evolution of tokens through the depth of encoder-only transformer models at inference time by modeling them as a system of particles interacting in a mean-field way and studying the corresponding dynamics. More specifically, we consider this problem in the moderate interaction regime, where the number $N$ of tokens is la…

Opening excerpt from the authors’ abstract. source