Home / Courses / Introductory Statistics / Unit 9

Introductory Statistics Unit 9: Regression

Regression models the relationship between two quantitative variables with a line, and measures how well the line fits.

Covers: Scatterplots, correlation, least-squares regression, residuals, r-squared

Study guide

Regression describes relationships between two quantitative variables. Start with a scatterplot of the explanatory variable (x) against the response variable (y), and describe its direction (positive or negative), form (linear or curved), strength, and any unusual points.

The correlation r measures the strength and direction of a linear relationship. It ranges from −1 to 1, has no units, doesn't change if you swap x and y or change units, and is strongly affected by outliers. It only measures linear association, and correlation does not imply causation.

The least-squares regression line ŷ = a + bx minimizes the sum of squared residuals. Its slope is b = r(s_y/s_x), and it always passes through (x̄, ȳ). Interpret the slope as the predicted change in y for each one-unit increase in x, and the intercept as the predicted y when x = 0 (only if that makes sense). Predicting outside the range of the data is extrapolation and is unreliable.

A residual is actual − predicted, y − ŷ. A residual plot with no pattern means a linear model is appropriate; a curved pattern means it isn't. The standard deviation of the residuals, s, gives the typical prediction error. The coefficient of determination r² is the percent of the variation in y explained by the linear model with x.

Outliers have large residuals, while influential points (often with extreme x-values) substantially change the slope or correlation when removed.

To test for a linear relationship, use t = (b − β₀)/SE_b with df = n − 2. A confidence interval for the true slope β is b ± t*·SE_b, with b and SE_b read from computer output. Before slope inference, check that the form is linear, the observations are independent (with the 10% condition), the residuals are roughly normal, the residual spread is about equal across x, and the data come from a random sample or experiment.

A confidence interval for a mean response estimates the average y for all individuals with a given x. A prediction interval estimates one individual's y. It is always wider, because it adds the scatter of individuals around the line. Regression output lists the intercept and slope in the coefficient column, each with its standard error, t-statistic and p-value. It also reports s, the typical residual size, and r².

Key ideas

• Describe a scatterplot by direction, form, strength and unusual points, in context.
• Correlation r measures the strength and direction of a linear relationship. It runs from −1 to 1 and has no units.
• The least-squares line minimizes the sum of squared residuals.
• A residual plot with no pattern means a linear model is appropriate.
• r² is the percent of variation in y explained by the linear model.
• Don't extrapolate beyond the data, and correlation alone never proves causation.

Formulas and rules

Common mistakes

• Interpreting slope without "predicted": it's the predicted change in y per one-unit increase in x.
• Thinking a high r means the relationship is linear. Check the residual plot.

Words to know

Residual formula

Residual = actual − predicted = y − ŷ

Slope of the least-squares line

b = r·(s_y / s_x)

Coefficient of determination (r²)

The percent of variation in y explained by the linear model with x.

Residual plot

No pattern means a linear model is appropriate.

Correlation coefficient (r)

Strength and direction of a linear relationship; −1 to 1, no units.

Slope of a regression line (b)

The predicted change in y for each one-unit increase in x.

Extrapolation

Predicting outside the range of the data's x-values; unreliable.

Explanatory vs. response variable

x explains or predicts y.

Scatterplot

A graph of paired quantitative data.

Least-squares regression line

ŷ = a + bx, minimizing the sum of squared residuals.

Regression line passes through

(x̄, ȳ)

Standard deviation of residuals (s)

The typical prediction error.

Influential point

A point that substantially changes the slope or correlation when removed.

Outlier (regression)

A point with a large residual.

Sample practice questions

23 more questions like these, from level 1 to 5, are in the practice set.