Activation Functions: Which to use?

Published

This is a series where I answer the questions my tutoring students really ask – the ones textbooks tend to skip. I teach machine learning, deep learning, statistics, and math to learners of all levels.

The question. A student building her first network stared at the documentation and asked: “Sigmoid, tanh, ReLU, GELU, SiLU, softmax – are these interchangeable, or is there a logic to the zoo? And why did everyone stop using sigmoid?”

There is a logic, and you can see all of it in one number: the derivative. But first, the reason these functions exist at all.

Why nonlinearity is not optional

Stack two linear layers and you get a linear layer: W2 @ (W1 @ x) = (W2 @ W1) @ x, one matrix doing the work of two. A deep network with linear activations is just linear regression with extra steps. The activation function is the small nonlinearity applied after every linear map that lets a stack of layers bend, fold, and approximate functions a straight line never could. Choosing it is choosing how your network bends – and, more importantly for training, how gradients flow backward.

The zoo in one picture

Here are the seven you will actually meet, on the same input range:

The commonly used activation functions plotted on the same input range with a usage guide

And here is the same zoo as a worked example. Take three inputs, x = -1.5, 0.5, 2.0, and push them through each function:

function    f(-1.5)    f(0.5)    f(2.0)
sigmoid      0.182      0.622      0.881
tanh        -0.905      0.462      0.964
ReLU         0.000      0.500      2.000
Leaky ReLU  -0.015      0.500      2.000
GELU        -0.100      0.346      1.954
SiLU        -0.274      0.311      1.762

The formulas, in plain ASCII:

sigmoid(x)  = 1 / (1 + e^(-x))
tanh(x)     = (e^x - e^(-x)) / (e^x + e^(-x))
ReLU(x)     = max(0, x)
LeakyReLU(x)= x if x > 0 else 0.01 * x
GELU(x)     = x * Phi(x)          Phi = standard normal CDF
SiLU(x)     = x * sigmoid(x)      also called Swish
softmax(z)_i = e^(z_i) / sum_j e^(z_j)

Sigmoid and tanh both squash, but tanh is centered at zero, which keeps the next layer’s inputs balanced. ReLU and Leaky ReLU pass positive inputs untouched. GELU and SiLU look like ReLU with the corner sanded off – smooth, and slightly negative just left of zero, which turns out to matter.

Softmax is the odd one out: it does not act on one number but on a whole vector, turning scores into a probability distribution. Logits [2.0, 1.0, 0.1] become [0.659, 0.242, 0.099] – the biggest logit wins, the gaps are preserved, and everything sums to one. That is why it lives at the output of classifiers, and inside attention, where the scores over tokens must become weights.

The derivative is the whole story

Why did sigmoid disappear from hidden layers? Compute its derivative: sigmoid’(x) = sigmoid(x) * (1 - sigmoid(x)), which peaks at 0.25 and falls to nearly zero in the tails. During backpropagation, every layer multiplies the passing gradient by its local derivative, so a ten-layer sigmoid network shrinks the gradient by a factor of up to:

0.25^10 = 0.000001

One millionth of the signal reaches the early layers. Learning stalls long before the network is deep. This is the vanishing gradient problem, and the figure shows both halves of the story: the saturating curves whose derivatives die in the tails, and the ReLU family whose derivative is a clean 1 for every positive input, letting the gradient pass through unchanged.

Derivatives of the activation functions and the vanishing gradient across layers

Tanh suffers less – its derivative peaks at 1.0 – but it still saturates, which is why deep tanh networks fell out of favor too. ReLU’s weakness is the mirror image: a unit that receives only negative inputs outputs zero with zero gradient forever, the “dying ReLU.” Leaky ReLU patches that with a small 0.01 slope on the left. GELU and SiLU keep ReLU’s non-saturating right side while staying smooth everywhere, and smoothness makes optimization better behaved – one reason modern transformers settled on them.

So which one should you use?

The pragmatic answer, matching what the field actually does:

  • Hidden layers, default: ReLU. It is a single comparison, it does not saturate for positive inputs, and it trains fast.
  • Hidden layers, transformers: GELU (BERT, GPT-style models) or SiLU inside a SwiGLU block (LLaMA-style models).
  • If units keep dying: switch ReLU to Leaky ReLU before reaching for anything fancier.
  • Outputs: sigmoid for a single probability, softmax for a distribution over classes.
  • Gates: sigmoid, exactly where the LSTM post used it – when you need a number between 0 and 1 that means “how much.”

Sigmoid is not dead, in other words. It just retired from hidden layers and kept the jobs where squashing to (0, 1) is the whole point.

One thing to try

Pick x = 0.5 and compute SiLU by hand: sigmoid(0.5) = 0.622, so SiLU(0.5) = 0.5 * 0.622 = 0.311. Now do GELU at the same point with the normal CDF table: Phi(0.5) = 0.691, so GELU(0.5) = 0.346. Two “modern” functions, two slightly different answers to the same input – that gap, small as it looks, is the kind of detail architecture papers argue about. Then compute sigmoid’(8) = 0.0003 and imagine multiplying it by itself ten times.

The worked examples in this article are adapted from the teaching notebooks I use with my students, where every activation value is computed by hand before any library is allowed.

References

  1. Nair, V., & Hinton, G. E. (2010). “Rectified Linear Units Improve Restricted Boltzmann Machines.” ICML 2010.
  2. Hendrycks, D., & Gimpel, K. (2016). “Gaussian Error Linear Units (GELUs).” arXiv:1606.08415.
  3. Ramachandran, P., Zoph, B., & Le, Q. V. (2017). “Searching for Activation Functions.” arXiv:1710.05941.
  4. Glorot, X., & Bengio, Y. (2010). “Understanding the Difficulty of Training Deep Feedforward Neural Networks.” AISTATS 2010.

I tutor machine learning, deep learning, statistics, and math. If neural networks – or anything else – are not clicking for you, you can find me on Wyzant.