Overfitting

Overfitting is when a machine learning model learns its training data too well. Instead of finding the general rule, it memorises the exact examples, including their random errors and quirks.

The result is a model that looks excellent and is not. It scores close to perfect on the data it was trained on, then does badly the moment it sees anything new. It never learned the rule. It learned the answers.

Think of it like this. A student memorises every question and answer in a practice book, word for word. On that practice paper they score full marks. The real exam asks the same ideas in different words, and they have nothing to fall back on.

Why it happens

• The model is too complex for the job. Too many parameters or layers for the amount of data available.
• There is not enough data. With only a few examples, memorising them is easier than finding the rule behind them.
• The data is noisy. Errors and irrelevant details get learned as if they were real rules.
• Training ran too long. After a point the model stops improving and starts fitting oddities in the sample.

How to spot it

• Training accuracy is very high, often close to perfect.
• Accuracy on new data is much lower. That gap is the signal.
• The two curves separate. Training error keeps falling while validation error starts rising. The point where they split is where memorising began.

How to fix it

• Get more data. The most reliable fix. Random noise averages out across a bigger sample, real patterns do not.
• Stop early. Halt training at the point validation error starts rising.
• Simplify the model. Fewer parameters, fewer features.
• Use regularisation. A penalty that stops the model leaning too hard on any single input.
• Cross validate. Test on several different splits, so one lucky split does not fool you.

One warning. Every fix here pushes the model toward being simpler. Push too far and you get the opposite problem, underfitting, where the model is too simple to learn anything useful.