NN: Activation Function

Activation function in neural networks acts as a non-linear transfer function to convert linear regression of inputs with weights to more complex system. Three non-linear activation functions are illustrated here, including Sigmoid, Tanh, and Relu, compared with linear activation function.

A neural network with linear activation function is nothing but a linear regression model.

With a sigmoid activation function, small and gradual changes in w and b can cause small and gradual changes in z, thereby producing similarly gradual changes in the neuron’s activation sigma. However, large negative or positive values of z illustrate an exception: at extreme z values, sigmoid neurons sigma will output 0’s (when z is negative) or 1’s (with z is positive). In these extreme cases, subtle updates to the weights and biases during training will have little to no effect on the output, and thus learning will stall, a situation called neuron saturation.

The shape of the tanh ‘tanch’ curve is similar to the sigmoid curve. The difference lies in their ranges, sigmoid function exists in the range of [0,1], whereas the tanh neuron’s output has the range [-1,1]. This 0-centered characteristics make neuron saturation less likely, thereby enabling the entire network to learn more efficiently.

The rectified linear unit is essentially two distinct linear functions combined (one at negative z values returning 0, and the other at positive z values returning z). This non-linearities permit deep learning models to approximate any continuous function. The incorporation of ReLu into AlexNet was one of the reasons for its popularity.

Different from ReLu, leaky ReLu changes the transfer function when z is negative to a linear function.

References:

Book: Deep Learning Illustrated: A Visual, Interactive Guide to Artificial Intelligence by Jon Krohn with Grant Beyleveld and Aglae Bassens https://www.amazon.com/Deep-Learning-Illustrated-Intelligence-Addison-Wesley/dp/0135116694/ref=sr_1_1?keywords=deep+learning+illustrated+a+visual&qid=1582417080&sr=8-1