What are you sinking? A geometric approach on attention sink

NeurIPSSpotlight2025

Authors
Valeria Ruscio, Umberto Nanni, Fabrizio Silvestri
Venue
NeurIPS 2025
Track
Spotlight

TL;DR

Attention sink (AS) is a consistent pattern in transformer attention maps where certain tokens (often special tokens or positional anchors) disproportionately attract attention from other tokens…

Opening excerpt from the authors’ abstract. source

Read the paper

Topics

transformer attention geometric

Related NeurIPS papers

← All NeurIPS 2025 Spotlight papers · Browse the whole archive