What One Cannot, Two Can: Two-Layer Transformers Provably Represent Induction Heads on Any-Order Markov Chains
NeurIPSSpotlight2025
TL;DR
In-context learning (ICL) is a hallmark capability of transformers, through which trained models learn to adapt to new tasks by leveraging information from the input context…
Opening excerpt from the authors’ abstract. source
Read the paper
Topics
in-context learning transformer llm rag
← All NeurIPS 2025 Spotlight papers · Browse the whole archive