Bigram Subnetworks: Mapping to Next Tokens in Transformer Language Models

TL;DR

In Transformer language models, activation vectors transform from current token embeddings to next token predictions as they pass through the model…

Opening excerpt from the authors’ abstract. source