Bias-Variance Tradeoff
The bias-variance tradeoff explains why models fail in two opposite directions: too simple to learn the pattern, or too sensitive to the particular data they saw.
Key takeaways
- Bias is error from assumptions too simple. Variance is error from sensitivity to the training sample.
- Reducing one usually increases the other, which is why it is a tradeoff.
- Compare training error with validation error to tell which one you have.
- More data reduces variance and does nothing for bias.
The bias-variance tradeoff explains why models fail in two opposite ways. Bias is error from assumptions that are too simple. Variance is error from being too sensitive to the specific data you trained on.
The two pull against each other. Making a model more flexible cuts bias and raises variance. Constraining it does the reverse. The best model is the balance point, not either extreme.
Think of it like this. Think of archery. High bias is a tight cluster in the wrong place: consistent and consistently off. High variance is arrows scattered around the bullseye: right on average, unreliable individually.
How it works
- Error splits three ways: bias squared, plus variance, plus irreducible noise.
- Bias is systematic. Wrong assumptions, such as fitting a line to a curve. It causes underfitting.
- Variance is sensitivity. A different sample would give a noticeably different model. It causes overfitting.
- Irreducible error is noise. No model removes it, and pretending otherwise is how you end up chasing it.
How to spot it
- Both errors high: high bias. The model is too simple.
- Training low, validation high: high variance. The model memorised.
- Both low: you found the balance.
- You must compare the two. Training error alone cannot tell you which problem you have, which is the diagnostic mistake people make most often.
How to fix it
- Reduce bias: a more capable model, better features, less regularization.
- Reduce variance: more data, a simpler model, more regularization, or ensembling.
- More data only helps variance. It does nothing for bias, which is why a model that fails on its own training set will not be rescued by a bigger dataset.