Depth-Bounds for Neural Networks via the Braid Arrangement

TL;DR

We contribute towards resolving the open question of how many hidden layers are required in ReLU networks for exactly representing all continuous and piecewise linear functions on $\mathbb{R}^d$. While the question has been resolved in special cases, the best known lower bound in general is still 2.

Opening excerpt from the authors’ abstract. source