The emergence of sparse attention: impact of data distribution and benefits of repetition

NeurIPSOral2025

Authors: Nicolas Zucchet, Francesco D'Angelo, Andrew Kyle Lampinen, Stephanie C.Y. Chan
Affiliation: ETHZ - ETH Zurich
Venue: NeurIPS 2025
Track: Oral

TL;DR

We show that learning sparse attention is prone to emerging behaviors during training, and study (theoretically and empirically) how data and model design influence emergence speed.

Opening excerpt from the authors’ abstract. source

Read the paper

Topics

attention

← All NeurIPS 2025 Oral papers · Browse the whole archive