Home / Courses / Probability / Unit 9

Probability Unit 9: Expectation, Variance and Covariance

This unit extends expectation and variance to several random variables. Covariance, conditional expectation and moment generating functions are the tools behind portfolio risk, compound claims and most limit theorems.

Covers: Expectations of functions of two variables, Covariance and correlation, Variance of sums and linear combinations, Conditional expectation and the law of total expectation, Law of total variance, Compound (random) sums, Moment generating functions

Study guide

For two random variables, E[g(X, Y)] is a double sum or double integral of g against the joint distribution. Linearity always holds: E[aX + bY] = aE[X] + bE[Y], with or without independence. Products are different: E[XY] = E[X]E[Y] holds when X and Y are independent but not in general. The covariance Cov(X, Y) = E[XY] − E[X]E[Y] measures how X and Y move together, and the correlation ρ = Cov/(σ_X σ_Y) rescales it to lie between −1 and 1. Covariance is bilinear and ignores added constants: Cov(aX + b, cY + d) = ac Cov(X, Y).

Variances of sums need covariances: Var(aX + bY) = a²Var(X) + b²Var(Y) + 2ab Cov(X, Y), and for many variables Var(ΣXᵢ) = ΣVar(Xᵢ) + 2Σ_{i<j}Cov(Xᵢ, Xⱼ). For independent variables the covariances vanish, so variances add, even for differences. Portfolio problems apply this directly, and diversification works because a correlation below 1 makes the combined variance smaller. Be careful with the converse: zero covariance does not imply independence, as Y = X² with X symmetric about 0 shows.

Conditional expectation E[X | Y] is itself a random variable, a function of Y. The law of total expectation says E[X] = E[E[X | Y]], which is the continuous form of averaging over cases. The law of total variance splits variability into two parts: Var(X) = E[Var(X | Y)] + Var(E[X | Y]), the average variance within groups plus the variance between group means. For a compound sum S = X₁ + ··· + X_N with N independent of the iid claims, E[S] = E[N]E[X] and Var(S) = E[N]Var(X) + Var(N)(E[X])², a key result in insurance.

The moment generating function M(t) = E[e^(tX)] encodes all moments: M⁽ⁿ⁾(0) = E[Xⁿ]. It also identifies distributions, since two variables with the same MGF near 0 have the same distribution. Learn the common MGFs: binomial (1 − p + peᵗ)ⁿ, Poisson e^(λ(eᵗ − 1)), exponential λ/(λ − t), and normal e^(μt + σ²t²/2). For independent variables, the MGF of a sum is the product of the MGFs, which is the fastest way to show that sums of independent Poissons are Poisson and sums of independent normals are normal.

Key ideas

• Expectation is always linear, but E[XY] = E[X]E[Y] needs independence or zero covariance.
• Covariance is bilinear and unaffected by added constants.
• Var(X ± Y) = Var(X) + Var(Y) ± 2Cov(X, Y).
• Zero covariance does not imply independence.
• E[X] = E[E[X | Y]] and Var(X) = E[Var(X | Y)] + Var(E[X | Y]).
• Compound sums combine claim count and claim size variability.
• MGF derivatives at 0 give moments, and MGFs of independent sums multiply.

Formulas and rules

Common mistakes

• Subtracting variances when finding Var(X − Y).
• Dropping the 2Cov term for dependent variables.
• Concluding independence from zero covariance.
• Using only E[Var(X | Y)] and forgetting Var(E[X | Y]) in the law of total variance.
• Dividing covariance by variances instead of standard deviations when finding correlation.

Before the test, make sure you can

• Calculate E[XY], Cov(X, Y) and ρ from a table or a joint density.
• Compute the variance of linear combinations, including portfolios.
• Explain with an example why zero covariance does not imply independence.
• Apply the laws of total expectation and total variance.
• Find the mean and variance of a compound sum.
• Find moments by differentiating an MGF.
• Identify a distribution from its MGF and use products of MGFs for sums.

Words to know

Covariance

Cov(X, Y) = E[(X − μ_X)(Y − μ_Y)] = E[XY] − E[X]E[Y].

Correlation

ρ = Cov(X, Y)/(σ_X σ_Y), a unitless measure of linear association between −1 and 1.

Uncorrelated

Cov(X, Y) = 0; implied by independence but not equivalent to it.

Bilinearity

Cov(aX + bY, Z) = aCov(X, Z) + bCov(Y, Z).

Variance of a sum

Var(ΣXᵢ) = ΣVar(Xᵢ) + 2Σ_{i<j}Cov(Xᵢ, Xⱼ).

Conditional expectation

E[X | Y], a function of Y giving the mean of X for each value of Y.

Law of total expectation

E[X] = E[E[X | Y]].

Law of total variance

Var(X) = E[Var(X | Y)] + Var(E[X | Y]).

Compound sum

S = X₁ + ··· + X_N with a random number N of iid terms.

Moment generating function

M(t) = E[e^(tX)], whose derivatives at 0 give the moments.

MGF uniqueness

If two MGFs agree near 0, the distributions are the same.

MGF of a sum

For independent X and Y, M_{X+Y}(t) = M_X(t)M_Y(t).

Portfolio variance

Var(wX + (1 − w)Y) = w²σ_X² + (1 − w)²σ_Y² + 2w(1 − w)Cov(X, Y).

Cauchy–Schwarz inequality

|Cov(X, Y)| ≤ σ_X σ_Y, which forces |ρ| ≤ 1.

Sample practice questions

30 more questions like these, from level 1 to 5, are in the practice set.