Extrapolation by Association: Length Generalization Transfer In Transformers

TL;DR

Transformer language models have demonstrated impressive generalization capabilities in natural language domains, yet we lack a fine-grained understanding of how such generalization arises…

Opening excerpt from the authors’ abstract. source