Home / Courses / Probability / Unit 10

Probability Unit 10: Limit Theorems (LLN, CLT)

Limit theorems describe what happens when many independent random variables are added or averaged. The law of large numbers explains why averages settle down, and the central limit theorem lets you approximate sums with the normal distribution.

Covers: Markov's inequality, Chebyshev's inequality, Distribution of the sample mean, Weak law of large numbers, Central limit theorem for sums and means, Normal approximations with continuity correction, Sample size calculations

Study guide

Probability inequalities bound tail probabilities using only a few moments. Markov's inequality says that for nonnegative X, P(X ≥ a) ≤ E[X]/a. Applying it to (X − μ)² gives Chebyshev's inequality, P(|X − μ| ≥ kσ) ≤ 1/k², so at least 75% of any distribution lies within 2 standard deviations of its mean. These bounds hold for every distribution, which makes them useful when the distribution is unknown, but they are often loose: for a normal distribution the true probability outside 2 SDs is about 4.6%, far below Chebyshev's 25%.

Let X̄ₙ be the mean of n iid observations with mean μ and variance σ². Then E[X̄ₙ] = μ and Var(X̄ₙ) = σ²/n, so its standard deviation, the standard error, is σ/√n. The sum Sₙ has mean nμ and SD σ√n. Notice the square roots: averaging four times as many observations only halves the standard error. The weak law of large numbers follows from Chebyshev: P(|X̄ₙ − μ| ≥ ε) ≤ σ²/(nε²), which goes to 0. In words, the sample mean converges in probability to the population mean.

The central limit theorem says more: for large n, (Sₙ − nμ)/(σ√n) is approximately standard normal, whatever the shape of the population, as long as the variance is finite. Equivalently, X̄ₙ ≈ N(μ, σ²/n). To use it, find the mean and SD of the sum or mean, standardize, and read Φ. Insurance problems use it for total claims across many policies, and reliability problems use it for total lifetimes. For groups of different policy types, add the means and variances of all groups before standardizing.

When the sum is integer-valued, as with counts, dice totals or Poisson demand, use a continuity correction: P(S ≥ 380) becomes P(Y > 379.5). The CLT also explains the normal approximations to the binomial and to the Poisson with large mean. Sample-size questions reverse the calculation: set z × σ/√n equal to the allowed error and solve for n, always rounding up. Applied to logarithms, the CLT shows that products of many positive factors are approximately lognormal.

Key ideas

• Markov and Chebyshev bound tails using only the mean and variance.
• The sample mean has mean μ and SD σ/√n.
• Sums have SD σ√n, not nσ, for independent terms.
• The law of large numbers says X̄ₙ → μ in probability.
• The CLT makes sums and means approximately normal for large n.
• Use a continuity correction for integer-valued sums.
• For sample sizes, solve z σ/√n ≤ error and round up.

Formulas and rules

Common mistakes

• Using σ instead of σ/√n for the sample mean, or nσ instead of σ√n for a sum.
• Treating Chebyshev's bound as an exact probability.
• Saying the CLT makes the population normal rather than the sum or mean.
• Forgetting the continuity correction for counts and dice totals.
• Rounding a required sample size down.

Before the test, make sure you can

• Apply Markov's and Chebyshev's inequalities and interpret them as bounds.
• Find the mean and SD of a sum and of a sample mean.
• State the weak law of large numbers and explain why it follows from Chebyshev.
• Use the CLT to approximate probabilities for sums of claims or lifetimes.
• Apply continuity corrections to integer-valued sums.
• Compute probabilities for sample proportions.
• Solve for the sample size needed for a given precision.

Words to know

Markov's inequality

For X ≥ 0 and a > 0, P(X ≥ a) ≤ E[X]/a.

Chebyshev's inequality

P(|X − μ| ≥ kσ) ≤ 1/k² for any distribution with finite variance.

Sample mean

X̄ₙ = (X₁ + ··· + Xₙ)/n.

Standard error

The SD of the sample mean, σ/√n.

Weak law of large numbers

X̄ₙ converges in probability to μ as n → ∞.

Convergence in probability

P(|Yₙ − c| ≥ ε) → 0 for every ε > 0.

Central limit theorem

Standardized sums of iid finite-variance variables approach N(0, 1).

Convergence in distribution

The CDFs of Yₙ approach the CDF of a limit distribution.

Continuity correction

Shifting an integer boundary by 0.5 before using a normal approximation.

Normal approximation to the binomial

Binomial(n, p) ≈ N(np, np(1 − p)) for large n.

Normal approximation to the Poisson

Poisson(λ) ≈ N(λ, λ) for large λ.

Lognormal distribution

The distribution of Y when ln Y is normal, approximating products of many positive factors.

iid

Independent and identically distributed.

Sample practice questions

30 more questions like these, from level 1 to 5, are in the practice set.