Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Loss

TL;DR

Mixture-of-Experts (MoE) models lack explicit constraints to ensure the router's decisions align well with the experts' capabilities, which ultimately limits model performance. To address this, we propose expert-router coupling (ERC) loss, a lightweight auxiliary loss that tightly couples the router's decisions with expert capabilities.

Opening excerpt from the authors’ abstract. source