# ReLU Activation

tags
Neural Network

A simple activation function which clips the output of the internal operations to be positive.

$f(\mathbf{x}_i) = \begin{cases} \mathbf{x}_i, & \text{ for } \mathbf{x_i} \ge 0 \\\ 0 & \text{ o.w.} \end{cases}$

The main hallmark of this activation function is the smaller effect of the vanishing gradient problem in deep neural networks. This comes at a cost of stability and the greater risk of dead neurons.