Efficient Prompt Compression with Evaluator Heads for Long-Context Transformer Inference

TL;DR

Although applications involving long-context inputs are crucial for the effective utilization of large language models (LLMs), they also result in increased computational costs and reduced performance…

Opening excerpt from the authors’ abstract. source