Tensor Product Attention Is All You Need

TL;DR

Scaling language models to handle longer input sequences typically necessitates large key-value (KV) caches, resulting in substantial memory overhead during inference…

Opening excerpt from the authors’ abstract. source