Correlation and regression

Why this matters

Many analytical questions are about relationships. Do customers who are more satisfied recommend us more? Does spending more on advertising bring more sales? Does delivery time affect repeat purchases? Correlation measures how closely two variables move together, and regression describes the relationship with an equation that can be used to estimate one from the other.

Both are among the most used and most misused tools in analysis. The exam expects you to interpret them and to recognise their limits, especially the gap between correlation and causation. This lesson covers correlation coefficients, causation and confounding, simple linear regression, residuals, and the danger of extrapolation.

The lesson

Correlation coefficients and scatter plots

The starting point for any relationship between two numeric variables is a scatter plot: one variable on each axis, one point per record. It shows the direction, strength and form of the relationship, and any outliers, at a glance.

The correlation coefficient, usually Pearson's r, summarises the strength and direction of a linear relationship in one number between −1 and +1.

• +1 is a perfect positive relationship: as one variable rises, the other rises exactly in step.
• −1 is a perfect negative relationship: as one rises, the other falls.
• 0 means no linear relationship.

Rough conventions describe values above about 0.7 in magnitude as strong and below about 0.3 as weak, though what counts as strong depends on the field. In the practice survey, the correlation between the satisfaction score and the likelihood-to-recommend score is about 0.94: very strong, and positive.

Two limits matter. Pearson's r measures only straight-line relationships: a strong curved relationship, such as performance rising and then falling with workload, can have r near zero. And r is sensitive to outliers: a single extreme point can create or hide a correlation. Both are reasons to look at the scatter plot before trusting the number. Spearman's rank correlation, which works on the ranks of values rather than the values themselves, handles monotonic curves and outliers better.

Correlation against causation, and confounding variables

Correlation does not imply causation. Two variables can move together for reasons other than one causing the other.

• A confounding variable affects both. Ice cream sales and drowning incidents are correlated because hot weather increases both. Customers who receive more marketing emails may spend more because the marketing team emails its best customers most.
• Reverse causation: the direction is the opposite of what was assumed. Stores with more staff may have higher sales because busy stores are given more staff.
• Coincidence: with enough variables, some will correlate by chance, and a trend over time in two unrelated series produces a correlation too.

Establishing causation needs more than observed data. The strongest evidence comes from a randomised experiment, such as an A/B test, where the change is assigned at random so confounders balance out. Without one, analysts should describe relationships as associations, name plausible confounders, and avoid wording such as "drives" or "causes" that the data cannot support.

Simple linear regression: slope, intercept and R-squared

Simple linear regression fits a straight line through a scatter plot to predict one variable, the dependent or target variable, from another, the independent or predictor variable:

predicted y = intercept + slope × x

The slope is the change in the predicted value for a one-unit increase in the predictor. The intercept is the predicted value when the predictor is zero. The line is chosen by least squares: it minimises the sum of the squared vertical distances between the points and the line.

For the practice survey, predicting the recommend score (0 to 10) from satisfaction (1 to 5) gives a slope of about 1.91 and an intercept of about −0.11. Each extra point of satisfaction is associated with nearly two more points of likelihood to recommend, and a customer scoring 3 is predicted to give about 5.6.

R-squared (the coefficient of determination) is the proportion of the variation in the target explained by the model, between 0 and 1. For simple linear regression it is the square of r: here about 0.88, so satisfaction accounts for most of the variation in recommendation scores. A high R-squared shows the line fits these data well; it does not show causation, and it does not guarantee good predictions outside the data.

Multiple regression extends the idea to several predictors at once, which allows the effect of one variable to be estimated while holding others constant, a partial answer to the confounding problem.

Residuals, and what a pattern in them means

A residual is the difference between an actual value and the value the regression predicts. Positive residuals are points above the line; negative ones, below.

Residuals are the main diagnostic for whether a linear model is appropriate. A residual plot, residuals against predicted values, should look like random scatter around zero. Patterns reveal problems:

• a curve means the relationship is not linear, and a straight line is the wrong model;
• a funnel, with residuals spreading wider as values grow, means the variability is not constant, which undermines the model's confidence intervals;
• a few very large residuals are outliers worth investigating, and they may be pulling the line towards themselves.

Checking residuals is what separates fitting a line from understanding a relationship.

Predicting within the data against extrapolating beyond it

A regression describes the relationship within the range of the data it was fitted on. Using it inside that range is interpolation, and is reasonably safe if the model fits. Using it outside that range is extrapolation, and is risky, because nothing in the data says the relationship continues.

The survey model shows this in miniature. Its intercept of −0.11 is the predicted recommend score for a satisfaction of zero, which is off the survey's 1-to-5 scale and predicts an impossible negative score. The intercept is a mathematical part of the line, not a meaningful prediction.

Real extrapolation errors are more costly: projecting a sales growth rate years beyond the data it was measured on, or assuming a price change far larger than any observed will have a proportionate effect. When a forecast must go beyond the data, it should be labelled as an extrapolation, with the assumption that the relationship holds stated plainly.