# ReLU Activation

- tags
- Neural Network

A simple activation function which clips the output of the internal operations to be positive.

\[ f(\mathbf{x}_i) = \begin{cases}
\mathbf{x}_i, & \text{ for } \mathbf{x_i} \ge 0 \\\

0 & \text{ o.w.}
\end{cases}
\]

The main hallmark of this activation function is the smaller effect of the vanishing gradient problem in deep neural networks. This comes at a cost of stability and the greater risk of dead neurons.

## Backlinks

## 1 Backlinks

### chandar2019: Towards Non-saturating Recurrent Units for Modelling Long-term Dependencies

The study and use of non-saturating activation functions (i.e. ReLU Activations) for the non-linear transfer functions