Mozart: Modularized and Efficient MoE Training on 3.5D Wafer-Scale Chiplet Architectures

TL;DR

Mixture-of-Experts (MoE) architecture offers enhanced efficiency for Large Language Models (LLMs) with modularized computation, yet its inherent sparsity poses significant hardware deployment challeng…

Opening excerpt from the authors’ abstract. source