Dropout
Dropout randomly switches off a fraction of a network's units during training, so the model cannot depend on any single one and generalises better.
Machine Learning
Key takeaways
- Dropout randomly disables units during training so no single one becomes critical.
- It effectively trains an ensemble of thinner networks that share weights.
- It is disabled at inference. Leaving it on makes predictions random.
- Around 0.5 for fully connected layers, much lower for convolutional ones.
Dropout randomly switches off a fraction of units during training. Each batch sees a different partial network, so no single unit can be relied on.
That unreliability is the point. If any unit might vanish, the network has to spread what it learns across many, and redundant representations generalise better than fragile ones.
Think of it like this. Think of a team where a random few people are out sick each day. Nobody can be the only person who knows something, because knowledge that lives in one head is knowledge the team keeps losing.
How it works
- A fraction of units is zeroed each training step, chosen randomly.
- A different subset every batch, so the network is effectively an ensemble of many thinner networks.
- Remaining activations are scaled up so the total signal stays comparable.
- It is switched off at inference. Every unit participates when you actually use the model.
Typical values
- Fully connected layers: around 0.5, meaning half the units drop.
- Convolutional layers: much lower, roughly 0.1 to 0.2, because convolutions already share weights.
- Input layers: rarely above 0.2 if used at all.
- Transformers: commonly 0.1.
Common mistakes
- "Leave it on at inference." Predictions become random. Frameworks handle this with train and eval modes, and forgetting to switch is a classic bug.
- "More dropout is safer." Too much and the network cannot learn at all, which is underfitting.
- "Use it everywhere." With batch normalization present, heavy dropout often hurts. Pick one primary regulariser.