Backpropagation

Backpropagation works out how much each weight contributed to the model's error. It does this by applying the chain rule backwards through the network, one layer at a time.

Without it, training a deep network would mean testing every weight separately, which is impossible at scale. Backpropagation gets the answer for every weight in a single backward sweep.

Think of it like this. Think of a project that shipped late, and tracing the delay back through each team to see who contributed what. You start at the outcome and work backwards, apportioning blame as you go.

How it works

• Forward pass first. Run the input through the network and compute the loss.
• Start at the output. Work out how much the loss changes with the final layer's output.
• Apply the chain rule backwards. Each layer converts the gradient arriving from above into the gradient for its own inputs and weights.
• Every weight gets a number. One backward sweep produces the gradient for all of them.
• Then the optimiser steps. Backpropagation stops here; gradient descent does the updating.

Common mistakes

• "Backpropagation trains the network." It computes gradients. The optimiser is what changes the weights.
• "It is how the brain learns." Almost certainly not. The name and the diagrams suggest a resemblance the neuroscience does not support.
• "It only works on simple feedforward networks." It works on any computation graph, including recurrent networks and transformers.
• "It is the only way to train." Evolutionary methods and other approaches exist. They are just far less efficient for this.