Regression

A company wants to estimate how many air conditioners it will sell next summer as a function of the maximum temperature in June. It has 10 years of data: for each year, the June maximum temperature and the number of air conditioners sold that month. Each year becomes one dot on a scatter plot.

The dots show a clear positive trend — hotter Junes mean more sales. But the trend is not perfect; there is scatter around it. The question is: is there a way to draw a single straight line through that cloud that captures the trend as well as possible, and can you then use that line to answer the practical question "if June 2027 has a maximum of 44°C, how many ACs should we stock?"

The answer is yes. The mathematical method is called least-squares regression. It produces the specific line that fits the data best in a precise sense, and once you have that line, you can plug in any x and read off the predicted y.

This article walks through where that line comes from, why the formula is what it is, and what the numbers mean once you have them.

What does "best fit" even mean?

Look at a scatter plot and draw any straight line through it. For every data point, there is a vertical gap between the actual point and the line — the point is either above the line, on it, or below it. Call the signed vertical gap the residual for that point:

where y_i is the actual y-value and \hat{y}_i is the value predicted by the line at x = x_i.

If you move the line up or rotate it slightly, the residuals change — some shrink, others grow. The "best" line should make the residuals small as a whole. But how do you measure "small as a whole" when some are positive and some are negative?

• Summing the residuals directly does not work: positive and negative errors cancel, and a wildly wrong line can still have \sum e_i = 0.
• Summing |e_i| avoids cancellation but is mathematically hard to work with (the absolute value has a corner).
• Summing e_i^2 avoids cancellation and is smooth — it is differentiable, which means you can find its minimum using calculus.

The last option is the one that wins, both for tradition and for pleasant mathematics. The least-squares line is the line that minimises

Every regression calculation in the rest of this article is a consequence of this one choice.

Deriving the regression line of y on x

Assume the line has equation \hat{y} = a + bx where a and b are unknown. The job is to choose a and b to minimise

S is a smooth function of two variables, and it has a minimum where both partial derivatives vanish. Set them up one at a time.

Derivative with respect to a. Differentiate S treating b as a constant:

Set this equal to zero:

Split the sum:

Divide by n:

That is the first normal equation. It says: the best-fit line passes through the point (\bar{x}, \bar{y}). The centre of the data is always on the line.

Derivative with respect to b. Differentiate S treating a as a constant:

Set this equal to zero:

Substitute a = \bar{y} - b \bar{x} from the first equation:

Use \sum x_i = n \bar{x} and \sum y_i = n\bar{y}:

Solve for b:

Factor a little more to get a cleaner form. Using the identities \sum (x_i - \bar{x})(y_i - \bar{y}) = \sum x_i y_i - n \bar{x}\bar{y} and \sum (x_i - \bar{x})^2 = \sum x_i^2 - n \bar{x}^2:

This is the slope of the best-fit line. It has a name.

The equation y - \bar{y} = b_{yx}(x - \bar{x}) just uses the fact that the line passes through (\bar{x}, \bar{y}) with slope b_{yx} — point-slope form of a line.

The connection to correlation

Compare the formula for b_{yx} with the formula for the correlation coefficient r from the previous article:

The numerators are identical. The denominators differ — and the ratio comes out to

where s_x = \sqrt{\frac{1}{n}\sum(x_i - \bar{x})^2} and s_y = \sqrt{\frac{1}{n}\sum(y_i - \bar{y})^2} are the standard deviations of the two variables. This is a very useful relationship: once you have the correlation and the standard deviations, the regression slope comes for free.

The other regression line

What if, instead of predicting y from x, you wanted to predict x from y? Run the same derivation with the roles swapped. You end up minimising the sum of squared horizontal distances from the points to a line, and you get a different line — the regression line of x on y.

Why are there two lines and not one? Because the definition of "best fit" is asymmetric. The line of y on x minimises vertical error — the distances measured parallel to the y-axis. The line of x on y minimises horizontal error — the distances measured parallel to the x-axis. Those are different optimisation problems, and they give different answers.

Both lines pass through the centre (\bar{x}, \bar{y}), so they always intersect at the mean point. They coincide exactly when |r| = 1 — when the data lies perfectly on a single line, both ways of measuring error give the same line. As the correlation weakens, the two lines spread apart.

A useful identity. Multiply the two regression coefficients:

So the product of the two regression coefficients is the square of the correlation coefficient. This gives you another way to compute r — compute both regression coefficients and take the square root of their product (with the sign chosen to match the common sign of b_{yx} and b_{xy}, which is always the same because both depend on the numerator \sum(x_i - \bar{x})(y_i - \bar{y})).

Worked examples

The prediction \hat{y} = 30.6 is what the regression line says sales will be given an ad spend of 7 lakh, assuming the relationship continues to hold. It is not a guarantee — actual sales in any specific month will differ from the line by some residual.

The two regression lines open up at the mean point like a pair of scissors. The angle between them is small when the correlation is strong and wider when the correlation is weak. At r = \pm 1 the two lines close together into a single line; at r = 0 they become perpendicular (one horizontal, one vertical).

Properties of regression coefficients

1. b_{yx} and b_{xy} have the same sign. Both are proportional to the same numerator \sum(x_i - \bar{x})(y_i - \bar{y}). The sign of that numerator is the sign of r, and both coefficients inherit it.

2. b_{yx} \cdot b_{xy} = r^2. Already derived above. This implies |b_{yx} \cdot b_{xy}| \leq 1 — the product of the two coefficients is always at most 1 in magnitude.

3. Unaffected by change of origin, affected by change of scale. Shifting all x_i by a constant does not change the slope of either line. Multiplying all x_i by a constant c multiplies b_{yx} by 1/c and multiplies b_{xy} by c.

4. The two regression lines intersect at (\bar{x}, \bar{y}). Always, for any dataset. This is a direct consequence of the derivation — both lines are written in point-slope form anchored at the mean point.

5. The arithmetic mean of b_{yx} and b_{xy} is at least |r|. This is the AM-GM inequality applied to the product identity: \frac{b_{yx} + b_{xy}}{2} \geq \sqrt{b_{yx} \cdot b_{xy}} = |r|.

Common confusions

• "You only ever need one regression line." The line depends on which variable you are predicting. Predicting sales from ad spend uses y on x; predicting ad spend from sales uses x on y. They are different lines and they give different numerical answers if you use the wrong one.

• "The regression line is the same as the correlation line." Correlation is a single number; it does not come with a line. The regression line is a line; it does not directly tell you the strength of the association. They are related (via b_{yx} = r \cdot s_y / s_x) but are different objects answering different questions.

• "Prediction far outside the observed data is as reliable as prediction within it." False. A regression line fit to data from x = 20 to x = 50 tells you very little about x = 200. The relationship might bend, flatten, or break down entirely outside the range you observed. Using the line far outside the observed range is called extrapolation and should be done with extreme caution.

• "A steeper line means a stronger correlation." The slope measures the rate of change; the correlation measures the tightness of the fit. A line with slope 100 fit to a very scattered cloud of points can have a weak correlation. A line with slope 0.1 fit to a tight cloud can have a strong one.

• "Residuals sum to zero, so the line fits perfectly." The residuals of a least-squares line always sum to zero — this falls out of the first normal equation. But having \sum e_i = 0 does not mean the line fits well; it just means the errors balance out. The sum of squared residuals, \sum e_i^2, is the actual measure of fit.

Going deeper

The geometry of least squares

Think of the n observations as a single point in n-dimensional space: \mathbf{y} = (y_1, y_2, \ldots, y_n). The n fitted values \hat{\mathbf{y}} = (\hat{y}_1, \ldots, \hat{y}_n) are another point — one that has to lie on a specific 2-dimensional plane inside the n-dimensional space, because every \hat{y}_i is of the form a + b x_i for some a and b.

What the least-squares method does is this: it drops a perpendicular from \mathbf{y} to that 2-dimensional plane. The foot of the perpendicular is the point \hat{\mathbf{y}} closest to \mathbf{y} in the ordinary Euclidean sense. The vector from \hat{\mathbf{y}} to \mathbf{y} is the vector of residuals \mathbf{e}, and by construction this residual vector is perpendicular to the plane.

This geometric picture is why least squares has so many pleasant properties. Orthogonal projection is the cleanest operation in linear algebra, and once you recognise regression as a projection, you inherit all the theorems about projections for free. This point of view is the starting point for multiple regression (more than one explanatory variable), generalised linear models, and most of modern statistics.

Coefficient of determination

You saw in the article on correlation that r^2 — the square of the correlation — has an interpretation as the fraction of variability in y that is "explained" by x. Now you can see exactly what that means.

Define three sums of squares:

• Total sum of squares: \text{TSS} = \sum(y_i - \bar{y})^2
• Regression sum of squares: \text{RSS} = \sum(\hat{y}_i - \bar{y})^2
• Error sum of squares: \text{ESS} = \sum(y_i - \hat{y}_i)^2

After some algebra (using the fact that the residuals are perpendicular to the fitted values — the geometric picture above), you get

Total variability splits into "variability captured by the line" plus "variability left over in the residuals." The fraction captured is

So r^2 is literally the fraction of the variance in y that the linear fit to x accounts for. An r of 0.9 means the line explains 81\% of the variability; an r of 0.5 means the line only explains 25\% — a huge difference from what the raw correlations suggest.

Assumptions and their cost

The least-squares line always exists as a mathematical object. Whether it is a useful summary of the data depends on several assumptions about how the data was generated. The basic ones:

1. The true relationship is approximately linear. If the real pattern is a parabola, a straight line fit to it will be misleading — even if the fit is numerically the best possible line. Look at the scatter plot before trusting the line.

2. The residuals have roughly constant spread. If the points scatter tightly around the line for small x and widely for large x, the regression is still computable, but predictions near one end will be more reliable than predictions at the other end.

3. No influential outliers. A single extreme point can pull the regression line a long way. If the scatter plot reveals one or two points far from the mass, the regression should be recomputed both with and without them to see how much they are driving the fit.

These assumptions are not part of the formula. The formula produces a line for any dataset. The assumptions are what determine whether using the line as a predictive tool is a good idea.

Where this leads next

You now have the machinery to fit a line to data and use it for prediction. The real world does not always hand you data where a straight line is the right model — and even when it does, you often want a more sophisticated analysis.

• Correlation — the measure of strength that pairs with regression. Every regression calculation can be restated in terms of correlations and standard deviations.
• Sampling — if the data is a sample from a population, the regression line you compute is an estimate of the true underlying line. Sampling theory tells you how much the estimate can wobble.
• Least Squares — the deeper theory of minimising squared errors, covering multiple predictors, weighted errors, and non-linear models.
• Linear Algebra Preview — regression as orthogonal projection onto a subspace, which generalises effortlessly to multiple predictors.
• Introduction to Inference — how to test whether a regression slope is "real" or could have arisen by chance from a sample.