Towards Interpretable and Efficient Attention: Compressing All by Contracting a Few

TL;DR

Attention mechanisms have achieved significant empirical success in multiple fields, but their underlying optimization objectives remain unclear yet…

Opening excerpt from the authors’ abstract. source