FP4 All the Way: Fully Quantized Training of Large Language Models

TL;DR

We demonstrate, for the first time, fully quantized training (FQT) of large language models (LLMs) using predominantly 4-bit floating-point (FP4) precision for weights, activations, and gradients on d…

Opening excerpt from the authors’ abstract. source