Intrinsic Entropy of Context Length Scaling in LLMs

TL;DR

We propose to use Intrinsic Entropy for understanding impact of context length on Language Modeling, and conduct experiments to validate theoretical assumptions and deductions with language and synthetic datasets.

Opening excerpt from the authors’ abstract. source