ReLU Activation
- tags
- Neural Network
A simple activation function which clips the output of the internal operations to be positive.
\[ f(\mathbf{x}_i) = \begin{cases} \mathbf{x}_i, & \text{ for } \mathbf{x_i} \ge 0 \\ 0 & \text{ o.w.} \end{cases} \]
The main hallmark of this activation function is the smaller effect of the vanishing gradient problem in deep neural networks. This comes at a cost of stability and the greater risk of dead neurons.