UMoE: Unifying Attention and FFN with Shared Experts

TL;DR

Sparse Mixture of Experts (MoE) architectures have emerged as a promising approach for scaling Transformer models…

Opening excerpt from the authors’ abstract. source