NN: Backpropagation, Gradient Descent

Suppose we have a neural network with three layers (input, hidden, output), the first node value of X before the sigmoid activation function, has the first input_1 multiplied by the weight W11, added to the second input_2 multiplied by the weight W21:

The following figure illustrates the propagation of errors backward, where the error 0.8 at the first output layer node being split proportionately into 0.32 and 0.48 across the two connected links which have weights 2.0 and 3.0. The recombined error at the first hidden layer node is the sum of the connected split errors, which here are 0.32 and 0.2, to give 0.42:

The following figure shows how the weight at the first hidden layer node is being updated. Here the feedforward propagation values from the first hidden layer node Oj=1 and the second hidden node Oj=2 are given:

References:

Book: Make Your Own Neural Network: A gentle journey through the mathematics of neural networks, and making your own using the Python computer language. By Tariq Rashid https://www.amazon.com/Make-Your-Own-Neural-Network-ebook/dp/B01EER4Z4G