Gradient Boosting

Gradient boosting trains models one at a time in a series. Each model fits the error residuals of the combined ensemble so far. The final prediction is the sum of all weak learners, scaled by a learning rate that controls how aggressively each correction is applied.

That sequential correction is what separates gradient boosting from parallel ensemble methods like random forest. Instead of training many independent trees and averaging them, gradient boosting treats learning as an optimization problem, nudging the model toward better accuracy step by step. In practice it often wins tabular competitions because it can model complex interactions without preprocessing.

Think of it like this. Think of a student solving practice problems one at a time. After each set, they identify the questions they got wrong, focus on those exact gaps, and try again. Each round is small, but the cumulative improvement is large.

How it works

The algorithm initializes with a simple prediction, usually the mean of the target. Each subsequent tree predicts the residuals of the current ensemble. Those residuals are scaled by the learning rate and added to the ensemble. Tree depth is typically kept shallow, between three and eight, so each learner remains weak and generalizes well.

Common mistakes

"More trees always improve performance." After a point additional trees overfit. "Deep trees work better." Shallow trees generalize better in boosting. "Gradient boosting always beats random forest." It often does on structured data, but random forest trains faster and is more robust to noisy labels.

Trade-offs

Very accurate on tabular data with minimal preprocessing, but requires careful tuning of learning rate, tree depth, and number of estimators. Training is sequential, so it does not parallelize as easily as random forest.