# ReLU Activation

- tags
- Neural Network

A simple activation function which clips the output of the internal operations to be positive.

\[ f(\mathbf{x}_i) = \begin{cases}
\mathbf{x}_i, & \text{ for } \mathbf{x_i} \ge 0 \\\

0 & \text{ o.w.}
\end{cases}
\]

The main hallmark of this activation function is the smaller effect of the vanishing gradient problem in deep neural networks. This comes at a cost of stability and the greater risk of dead neurons.