Home / Courses / Statistics 2 / Unit 7

Statistics 2 Unit 7: Logistic Regression Basics

When the response is yes/no, logistic regression models the probability of 'yes' through the log-odds. You will move between probability, odds and log-odds, interpret coefficients as odds ratios, and test whether predictors matter.

Covers: Binary responses and why linear models fail, Probability, odds and log-odds, The logistic model and S-curve, Interpreting coefficients as odds ratios, Predicted probabilities, Wald tests, CIs and likelihood-ratio tests, Classification and multiple predictors

Study guide

Many outcomes are binary: a patient recovers or not, a customer buys or not, a loan defaults or not. Coding the response as 0 or 1 and fitting a straight line causes problems: predictions can fall below 0 or above 1, and the errors are neither normal nor constant in spread. Logistic regression solves this by modeling a transformed probability. Instead of p, it models the log-odds, log(p/(1 − p)), as a linear function of the predictors. Because the log-odds can be any real number, the line never produces impossible probabilities, and converting back gives an S-shaped curve bounded by 0 and 1.

Fluency with three scales is essential. Probability p runs from 0 to 1. Odds = p/(1 − p) run from 0 to infinity, with odds of 1 meaning a 50-50 chance; p = 0.8 gives odds 4. Log-odds run over all real numbers, with 0 meaning p = 0.5. To convert back, odds = e^(log-odds) and p = odds/(1 + odds). In the model log(p/(1 − p)) = β₀ + β₁x, the probability is p = e^(β₀ + β₁x)/(1 + e^(β₀ + β₁x)). The curve is steepest at p = 0.5, where x = −β₀/β₁, and its slope there is β₁/4.

Coefficients act on the log-odds scale. A one-unit increase in x adds β₁ to the log-odds, which multiplies the odds by e^(β₁), the odds ratio. A coefficient of 0.5 means each unit multiplies the odds by about 1.65, a 65% increase in odds. For a c-unit change, the odds ratio is e^(cβ₁). For an indicator variable, e^b compares the odds in the coded group with the baseline group. The change in probability is not constant: it is about β₁p(1 − p), large near p = 0.5 and small near 0 or 1. For rare outcomes, odds ratios approximate risk ratios; for common outcomes they exaggerate them.

Logistic models are fit by maximum likelihood, not least squares. Software reports each coefficient with a standard error and a Wald statistic z = b/SE, compared with the standard normal. A 95% CI for β is b ± 1.96·SE; exponentiate the endpoints to get a CI for the odds ratio, and check whether it contains 1. To test several predictors at once, compare nested models with a likelihood-ratio test: the drop in deviance follows approximately a χ² distribution with df equal to the number of added parameters. As in multiple linear regression, each coefficient is adjusted for the other predictors in the model.

Logistic models are often used to classify: predict 'yes' when p̂ exceeds a threshold such as 0.5. Sensitivity is the fraction of actual positives correctly flagged; specificity is the fraction of actual negatives correctly cleared. Raising the threshold trades sensitivity for specificity. Overall accuracy can be misleading when one outcome is rare, since always predicting the common outcome scores well. Watch for perfect separation, when a predictor splits the outcomes completely; maximum likelihood then fails to give finite estimates. And remember that observational data support associations, not causal claims, whatever the model.

Key ideas

• Logistic regression models the log-odds of a binary outcome as linear in the predictors.
• Odds = p/(1 − p) and p = odds/(1 + odds); log-odds of 0 means p = 0.5.
• e^(β₁) is the odds ratio for a one-unit increase in x; e^(cβ₁) for a c-unit increase.
• Effects on probability vary along the S-curve and are largest at p = 0.5.
• Coefficients are fit by maximum likelihood and tested with Wald z statistics.
• Likelihood-ratio tests compare nested models using the drop in deviance.
• Odds ratios approximate relative risks only for rare outcomes.

Formulas and rules

Common mistakes

• Treating a logistic coefficient as a change in probability.
• Confusing odds with probability (odds of 4 is p = 0.8, not 4).
• Forgetting to exponentiate the CI endpoints when reporting an odds ratio.
• Reading an odds ratio of 2 as 'doubles the risk' for a common outcome.
• Judging a classifier by overall accuracy when one class is rare.

Before the test, make sure you can

• Convert among probability, odds and log-odds.
• Compute predicted probabilities from a fitted logistic equation.
• Interpret coefficients as odds ratios, including for indicator variables and multi-unit changes.
• Find where the predicted probability equals 0.5.
• Carry out Wald tests and build CIs for odds ratios.
• Use a likelihood-ratio test to compare nested logistic models.
• Compute sensitivity and specificity for a classification rule.
• Explain why odds ratios are not risk ratios for common outcomes.

Words to know

Binary response

A variable with two outcomes, usually coded 1 (success) and 0 (failure).

Odds

p/(1 − p), the ratio of the chance an event happens to the chance it does not.

Log-odds (logit)

The natural log of the odds; can take any real value.

Logistic regression

A model in which the log-odds of a binary outcome are linear in the predictors.

Logistic curve

The S-shaped curve of p against x, bounded by 0 and 1.

Odds ratio

The factor by which odds change; e^(β₁) per unit of x.

Maximum likelihood

Estimation that picks parameters making the observed data most probable.

Wald test

A z test of a single coefficient: z = b/SE.

Deviance

A measure of lack of fit for a likelihood-based model; smaller is better.

Likelihood-ratio test

Compares nested models using the drop in deviance, referred to a χ² distribution.

Sensitivity

The proportion of actual positives that a classifier correctly identifies.

Specificity

The proportion of actual negatives that a classifier correctly identifies.

Perfect separation

When a predictor completely separates the outcomes, so estimates do not converge.

Relative risk

The ratio of two probabilities; approximated by the odds ratio when events are rare.

Sample practice questions

30 more questions like these, from level 1 to 5, are in the practice set.