AdaSPEC: Selective Knowledge Distillation for Efficient Speculative Decoders

TL;DR

Speculative Decoding (SD) accelerates large language model inference by employing a small draft model to generate predictions, which are then verified by a larger target model…

Opening excerpt from the authors’ abstract. source