Fixed-Point RNNs: Interpolating from Diagonal to Dense

TL;DR

Linear recurrent neural networks (RNNs) and state-space models (SSMs) such as Mamba have become promising alternatives to softmax-attention as sequence mixing layers in Transformer architectures…

Opening excerpt from the authors’ abstract. source