Home / Courses / Statistics 2 / Unit 6

Statistics 2 Unit 6: Multiple Regression

Multiple regression models a response with several predictors at once, letting you estimate the effect of one variable while holding others fixed. You will interpret coefficients, run overall and individual tests, compare models and spot multicollinearity.

Covers: Partial slopes and interpretation, Indicator (dummy) variables, Overall F test and individual t tests, R² and adjusted R², Multicollinearity, Interaction and polynomial terms, Nested models and model selection

Study guide

Multiple regression extends the simple model to μ_y = β₀ + β₁x₁ + … + β_kx_k with normal errors of constant SD σ. Each coefficient is a partial slope: the change in the mean of y for a one-unit increase in that predictor, holding all the other predictors constant. That phrase is the heart of the method. A partial slope can be much smaller than, or even opposite in sign to, the slope from a simple regression, because the simple slope also absorbs the effects of correlated variables left out of the model. Interpret coefficients in context, with units, as average associations.

Categorical predictors enter through indicator (dummy) variables equal to 1 for a category and 0 otherwise. A variable with c categories needs c − 1 indicators, with the omitted category serving as the baseline; each indicator's coefficient is the predicted difference from the baseline at fixed values of the other predictors. An interaction term, such as x·D, lets the slope on x differ between groups, so the fitted lines are no longer parallel. Polynomial terms such as x² capture curvature. With interactions or polynomials, the effect of x depends on other values and you cannot interpret its coefficient alone.

Inference happens at two levels. The overall F test, with (k, n − k − 1) df, tests H₀: all slopes are zero; F = MSR/MSE = (R²/k)/((1 − R²)/(n − k − 1)). Individual t tests, t = bⱼ/SE(bⱼ) with n − k − 1 df, ask whether a predictor adds explanatory power given all the others. To test several coefficients at once, compare nested models with F = [(SSE_reduced − SSE_full)/q]/MSE_full, where q is the number of coefficients dropped. Joint and individual tests can disagree when predictors overlap, which is why joint F tests exist.

R² = SSR/SST never decreases when you add a predictor, even pure noise, so it rewards complexity. Adjusted R² = 1 − (1 − R²)(n − 1)/(n − k − 1) includes a penalty and rises only when a new predictor helps more than chance would. Models with many predictors relative to n can overfit: they chase noise, report impressive R² values, and predict new data badly. Good practice combines subject knowledge, adjusted R² or similar criteria, residual checks, and ideally validation on fresh data.

Multicollinearity occurs when predictors are strongly correlated with each other. It does not bias predictions or the overall F, but it inflates the standard errors of the affected coefficients, making them unstable and their t tests weak. The variance inflation factor, VIF = 1/(1 − R²ⱼ), where R²ⱼ comes from regressing xⱼ on the other predictors, measures this; values above about 5 to 10 are warning signs. A classic symptom is a significant overall F with no significant individual t tests. Remedies include dropping a redundant predictor or combining correlated ones. Check residual plots against ŷ and against each predictor, including omitted ones, for patterns.

Key ideas

• Each multiple regression coefficient is a partial slope, holding the other predictors constant.
• A categorical variable with c categories uses c − 1 indicator variables.
• Interaction terms let slopes differ between groups; polynomial terms allow curvature.
• The overall F test asks whether any predictor helps; t tests ask about one predictor given the rest.
• Nested F tests compare a full model with a reduced one.
• R² never decreases as predictors are added; adjusted R² penalizes useless ones.
• Multicollinearity inflates standard errors and can make individual t tests fail even when F is significant.

Formulas and rules

Common mistakes

• Interpreting a coefficient without the phrase 'holding the other predictors constant'.
• Using n − 2 instead of n − k − 1 degrees of freedom.
• Choosing the model with the highest R² regardless of how many predictors it uses.
• Including an indicator for every category plus an intercept.
• Concluding a predictor has no effect because its t test fails in the presence of a correlated predictor.

Before the test, make sure you can

• Interpret partial slopes and indicator coefficients in context.
• Compute predicted values from a fitted multiple regression equation.
• Carry out the overall F test and individual t tests with the correct df.
• Compute R², adjusted R² and the overall F from an ANOVA table or R².
• Use a nested F test to decide whether a group of predictors is needed.
• Interpret interaction terms and find group-specific slopes.
• Diagnose multicollinearity using VIF and the pattern of F and t results.
• Explain overfitting and why a simpler model may be preferred.

Words to know

Multiple regression

A model predicting y from two or more explanatory variables.

Partial slope

The change in mean y per unit of one predictor with all other predictors held fixed.

Indicator variable

A 0/1 variable that represents membership in a category.

Baseline category

The category with no indicator; other categories' coefficients are differences from it.

Interaction term

A product of predictors that lets the effect of one depend on the value of another.

Overall F test

Tests whether all slope coefficients are zero.

Individual t test

Tests whether one coefficient is zero given the other predictors in the model.

Nested models

Two models where the reduced one is the full one with some coefficients set to zero.

Adjusted R²

R² penalized for the number of predictors.

Multicollinearity

Strong correlation among predictors, which inflates coefficient standard errors.

Variance inflation factor

1/(1 − R²ⱼ); how much collinearity inflates a coefficient's variance.

Overfitting

Fitting noise by using too many predictors relative to n, which hurts prediction.

Omitted-variable bias

Distortion of a coefficient when a relevant correlated predictor is left out.

Polynomial term

A power of a predictor, such as x², used to model curvature.

Sample practice questions

30 more questions like these, from level 1 to 5, are in the practice set.