Layer Normalization

Layer normalization rescales values using the statistics of a single example, computed across its features. Unlike batch normalization, it does not look at other examples at all.

That independence is why transformers use it. Sequences vary in length and batches vary in size, and layer normalization behaves identically regardless, including on a batch of one.

Think of it like this. Think of grading each student against their own average across subjects, rather than against the class. It works the same whether the class has thirty students or one.

How it works

• Statistics come from one example, computed across its features.
• Normalize to roughly zero mean and unit variance.
• Scale and shift by learned parameters, as in batch normalization.
• Identical at training and inference, because there is no batch dependency and nothing to store.

When to use it

• Transformers. Effectively universal there, and the reason it matters at all.
• Recurrent networks, where sequence length varies between examples.
• Small or variable batch sizes, where batch statistics would be unreliable.
• Not usually for vision. Batch normalization generally still wins on convolutional networks.

Common mistakes

• "It is batch normalization with a different axis." True mechanically, and the consequences are large: no batch dependency, and identical behaviour in training and inference.
• "Use it everywhere." On convolutional vision models, batch normalization usually performs better.
• "Placement does not matter." Before or after the residual connection changes training stability considerably. Modern transformers put it before.