Quantitative Bounds for Length Generalization in Transformers

TL;DR

We provide an upper bound on the length of training sequences required for a transformer to generalize to sequences of arbitrary lengths.

Opening excerpt from the authors’ abstract. source