Characterizing the Expressivity of Fixed-Precision Transformer Language Models

TL;DR

Transformer-based language models (LMs) have achieved widespread empirical success, but their theoretical expressive power remains only partially understood…

Opening excerpt from the authors’ abstract. source