Home / Courses / Statistics 2 / Unit 5

Statistics 2 Unit 5: Simple Linear Regression Inference

Regression inference asks whether a straight-line relationship seen in a sample reflects a real relationship in the population, and how precisely we can estimate it. You will test and estimate slopes, read regression output, check conditions, and build confidence and prediction intervals.

Covers: The population regression model, Least-squares estimates and residuals, t test and confidence interval for the slope, R², correlation and the regression ANOVA, Conditions and residual plots, Confidence vs prediction intervals

Study guide

The simple linear regression model says that for each x, the response y is normally distributed around a mean that falls on a line: μ_y = β₀ + β₁x, with the same standard deviation σ at every x. The least-squares line ŷ = b₀ + b₁x estimates that population line, with b₁ = r(s_y/s_x) and b₀ = ȳ − b₁x̄; it always passes through (x̄, ȳ). Residuals, y − ŷ, estimate the random errors, and their standard deviation s = √(SSE/(n − 2)) estimates σ. Two parameters are estimated, so inference uses n − 2 degrees of freedom.

The key question is usually whether β₁ = 0, which would mean the mean of y does not change with x. The test statistic is t = b₁/SE(b₁) with n − 2 df, and a confidence interval for the slope is b₁ ± t*·SE(b₁). Software reports both b₁ and SE(b₁); you rarely compute SE by hand, but its formula, SE(b₁) = s/(s_x√(n − 1)), explains a lot: the slope is estimated more precisely when residual scatter is small, n is large, and the x values are spread out. Interpret the slope in context and in the units of the data, as an average change in y per unit of x.

Regression also has an ANOVA table. Total variation SST splits into SSR, explained by the line, and SSE, left in the residuals. R² = SSR/SST is the fraction of variation in y explained by the linear relationship, and in simple regression R² = r². The F statistic MSR/MSE has (1, n − 2) df and equals t² for the slope, so the F test and the two-sided t test agree. The test of zero correlation, t = r√(n − 2)/√(1 − r²), is also the same test. With huge samples, tiny correlations become significant, so always look at R² and the size of the slope.

Inference requires the LINE conditions: the mean of y is Linear in x, errors are Independent, errors are Normal, and errors have Equal variance. Check them with a residual plot (residuals versus x or ŷ) and a histogram or normal plot of residuals. Curvature signals non-linearity; a fan shape signals unequal variance; patterns over time suggest dependence. Watch also for influential points: an observation with an extreme x value has high leverage and can drag the line toward itself, sometimes creating or hiding a significant slope. Fit with and without such points and report what changes.

To estimate the mean response at a particular x*, use a confidence interval ŷ ± t*·s√(1/n + (x* − x̄)²/((n − 1)s_x²)). To predict a single new observation, use a prediction interval, which adds 1 inside the square root to account for individual scatter: ŷ ± t*·s√(1 + 1/n + (x* − x̄)²/((n − 1)s_x²)). Both intervals are narrowest at x̄ and widen as x* moves away, because slope uncertainty matters more far from the center. The prediction interval never shrinks below about ±t*·s, however large n is. Avoid extrapolating beyond the range of observed x values.

Key ideas

• The model is μ_y = β₀ + β₁x with normal errors of constant SD σ.
• Inference on the slope uses t = b₁/SE(b₁) with n − 2 df.
• Spreading out x values, reducing noise and increasing n all shrink SE(b₁).
• R² = r² is the fraction of variation in y explained by the line.
• In simple regression, F = t² and the slope test equals the correlation test.
• Check LINE conditions with residual plots, and watch for influential points.
• Prediction intervals for individuals are always wider than confidence intervals for the mean response.

Formulas and rules

Common mistakes

• Using n − 1 instead of n − 2 degrees of freedom.
• Stating hypotheses about b₁ instead of β₁.
• Interpreting R² as the slope or as the correlation itself.
• Using a confidence interval for the mean when a prediction for one individual is wanted.
• Extrapolating far beyond the observed x values or ignoring an influential point.

Before the test, make sure you can

• State and test hypotheses about the population slope from regression output.
• Construct and interpret a confidence interval for β₁ in context.
• Compute residuals, s and R², and interpret each.
• Relate the slope t test to the regression ANOVA F and to the correlation test.
• Check LINE conditions using residual plots and identify influential points.
• Choose between and compute confidence and prediction intervals at a given x.
• Explain what factors make SE(b₁) smaller.
• Recognize the dangers of extrapolation and of over-reading significance with huge n.

Words to know

Population regression line

μ_y = β₀ + β₁x, the true mean of y at each x.

Least-squares line

The line ŷ = b₀ + b₁x that minimizes the sum of squared residuals.

Residual

Observed y minus predicted ŷ.

Regression standard error s

√(SSE/(n − 2)); the typical size of a residual and the estimate of σ.

SE of the slope

s/(s_x√(n − 1)), the standard error of b₁.

Slope t test

t = b₁/SE(b₁) with n − 2 df, testing H₀: β₁ = 0.

R²

SSR/SST, the fraction of variation in y explained by the linear model.

SSR

Regression sum of squares: variation in y explained by the fitted line.

Residual plot

A plot of residuals against x or ŷ used to check linearity and equal variance.

LINE conditions

Linear mean, Independent errors, Normal errors, Equal variance.

Leverage

How far an observation's x value is from x̄; high leverage gives it potential to move the line.

Influential point

An observation whose removal noticeably changes the slope or conclusions.

Confidence interval for mean response

An interval for the average y among all individuals with a given x.

Prediction interval

An interval for one new individual's y at a given x; wider than the CI for the mean.

Extrapolation

Predicting y for x values outside the range of the data.

Sample practice questions

30 more questions like these, from level 1 to 5, are in the practice set.