TokenSwap: A Lightweight Method to Disrupt Memorized Sequences in LLMs

TL;DR

As language models scale, their performance improves dramatically across a wide range of tasks, but so does their tendency to memorize and regurgitate parts of their training data verbatim…

Opening excerpt from the authors’ abstract. source