Regression

Regression is predicting a number. The model learns how the inputs relate to a continuous value and then estimates that value for new cases.

House prices, delivery times, tomorrow's demand. The answer sits on a scale rather than in a bucket, and being close counts for something, which is not true of categories.

Think of it like this. Think of estimating someone's height from a photograph. You will not be exactly right, and being off by a centimetre is very different from being off by a metre. Regression is scored on how close you got.

How it works

• Train on examples where the numeric answer is known.
• Fit a relationship between the features and that number.
• Minimise the error, usually the squared difference between predictions and truth.
• Predict for new inputs. The output is a number, not a choice.

Types

• Linear regression. Fits a straight line or flat plane. Simple, fast, and easy to explain.
• Polynomial and nonlinear. Fits curves when the relationship clearly bends.
• Regularised (Ridge, Lasso). Adds a penalty on large coefficients to control overfitting. Lasso can drive some to zero, which doubles as feature selection.
• Tree and ensemble based. Gradient boosted trees handle interactions and nonlinearity without you specifying them.

Common mistakes

• "Regression means linear regression." Linear is one kind. Most real relationships are not straight lines.
• "Logistic regression is regression." The name misleads. It predicts class probabilities and is used for classification.
• "Squared error is always the right measure." Squaring makes outliers dominate. Where that is wrong for your problem, absolute error is often the better choice.
• "A good R-squared means a good model." It measures fit on the data you have, and says nothing about how the model behaves on data you do not.