Implementing activation functions

ReLU

Rectified Linear unit is the most basic activation function. For each element in the input vector, it returns 0 if the value is negative, otherwise it returns the value itself.

$\text{ReLU}(x)=\max (0,x)$The forward message can be rewritten as $\max (0,x)=\{\begin{array}{l}xx\ge 0\\ 0\text{else}\end{array}=\frac{|x|+x}{2}.$The python code in numpy can therefore be either

def forward(self, X):
	self.X = X
	return X * (X > 0)

or

def forward(self, X):
	self.X = X
	return (np.abs(X) + X) / 2

However, first version will almost surely result in less operations, though it’s not as fancy. Notice that we need to store the input matrix X for the backward message. Or better, store only the boolean mask (X > 0).

For the backward message, activation function doesn’t have any parameters, but we still need a backward message for previous layers. For a single sample input, the derivative with respect to $x$ is given by $\frac{\partial \max (0,x)}{\partial x}=\{\begin{array}{l}1x>0\\ 0\text{else}\end{array}.$The full chain rule $\frac{\partial \mathcal{L}}{\partial x}=\frac{\partial \mathcal{L}}{\partial \text{ReLU}(x)}\cdot \frac{\partial \text{ReLU}(x)}{\partial x}$results in

def backward(self, dY):
	return dY * (self.X > 0)

In all the cases, the x > 0 is performed element-wise. So is the * operator. Beware that at 0, the gradient technically doesn’t exists. However, it is standard to set it to 0(as this implementation does).