Differentiable Sparsity via $D$-Gating: Simple and Versatile Structured Penalization

TL;DR

Structured sparsity regularization offers a principled way to compact neural networks, but its non-differentiability breaks compatibility with conventional stochastic gradient descent and requires eit…

Opening excerpt from the authors’ abstract. source