ReLU Activation

tags
Neural Network

A simple activation function which clips the output of the internal operations to be positive.

\[ f(\mathbf{x}_i) = \begin{cases} \mathbf{x}_i, & \text{ for } \mathbf{x_i} \ge 0 \\\
0 & \text{ o.w.} \end{cases} \]

The main hallmark of this activation function is the smaller effect of the vanishing gradient problem in deep neural networks. This comes at a cost of stability and the greater risk of dead neurons.

chandar2019: Towards Non-saturating Recurrent Units for Modelling Long-term Dependencies

The study and use of non-saturating activation functions (i.e. ReLU Activations) for the non-linear transfer functions