Home / Courses / Statistics 2 / Unit 9

Statistics 2 Unit 9: Experimental Design and Two-Way ANOVA

Good design decides what a study can conclude. This unit covers the principles of experiments, blocking, factorial designs with interactions, and the two-way ANOVA used to analyze them.

Covers: Principles of experimental design, Completely randomized and randomized block designs, Factorial designs and main effects, Interaction and interaction plots, Two-way ANOVA table and F tests, Placebos, blinding and confounding

Study guide

An experiment deliberately imposes treatments on experimental units and measures the response. Three principles make it trustworthy. Control means comparing treatments, including a control or placebo group, while holding other conditions as steady as possible. Randomization uses chance to assign units to treatments, which balances unmeasured variables on average and justifies causal conclusions. Replication means using enough units that chance variation can be estimated. Without random assignment, a confounding variable such as motivation in a volunteer study can explain the difference. With only one classroom or field per treatment, the treatment is confounded with that unit.

A completely randomized design assigns all units to treatments at random. When units differ in a way that strongly affects the response, such as starting weight, school, or field fertility, a randomized block design does better: group similar units into blocks and randomize treatments within each block. The analysis separates block-to-block variation from error, so MSE shrinks and the treatment F grows. With t treatments and b blocks, df are t − 1 for treatments, b − 1 for blocks, and (t − 1)(b − 1) for error. Matched pairs and crossover designs, in which each subject receives every treatment, are blocking with blocks of size 2 or one block per subject.

A factorial design studies two or more factors at once by running every combination of their levels; with a and b levels there are ab cells. Factorials are efficient, because each observation informs every factor, and they reveal interactions that one-factor-at-a-time studies miss. A main effect is the change in mean response across a factor's levels, averaged over the other factor. An interaction exists when the effect of one factor depends on the level of the other. On an interaction plot, parallel lines mean no interaction; non-parallel or crossing lines indicate interaction.

Two-way ANOVA with replication splits the total variation into SS_A, SS_B, SS_AB and SSE. With a and b levels and n replicates per cell, the df are a − 1, b − 1, (a − 1)(b − 1) and ab(n − 1). Each mean square is SS/df, and each F statistic is that mean square divided by MSE. Test the interaction first. If it is significant, main effects can mislead, because they average effects that differ; describe simple effects (the effect of one factor at each level of the other) instead. A crossing interaction can make both main effects look null even though each factor matters. Without replication, interaction and error cannot be separated.

Human studies need extra controls. A placebo lets the control group share the expectation of treatment, and blinding (single or double) keeps subjects' and evaluators' beliefs from shaping results. Identify the experimental unit, the thing actually randomized, and analyze at that level: assigning whole classrooms and then counting individual students as independent observations is pseudoreplication and overstates the evidence. Finally, design for power. Interaction tests often need more replication than main effects, so a non-significant interaction from a small study is inconclusive rather than proof that the factors act independently.

Key ideas

• Control, randomization and replication are the foundations of a sound experiment.
• Random assignment supports causal conclusions; self-selection invites confounding.
• Blocking removes known nuisance variation from the error, raising power.
• Factorial designs test all combinations of factor levels and can reveal interactions.
• Interaction means the effect of one factor depends on the level of another; non-parallel lines show it.
• When interaction is present, interpret simple effects rather than main effects.
• Analyze at the level of the experimental unit, the thing that was randomized.

Formulas and rules

Common mistakes

• Interpreting main effects when a strong interaction is present.
• Adding numbers of levels instead of multiplying them to count factorial cells.
• Ignoring blocks in the analysis, which inflates MSE.
• Treating individuals as independent units when groups were randomized (pseudoreplication).
• Concluding no interaction from a non-significant test in a small study.

Before the test, make sure you can

• Identify the experimental units, factors, levels and treatments in a study.
• Explain the roles of control, randomization, replication, placebos and blinding.
• Choose between completely randomized and randomized block designs and explain the benefit of blocking.
• Compute main effects and interaction contrasts from cell means.
• Read an interaction plot and describe what parallel or crossing lines mean.
• Complete a two-way ANOVA table with correct df and F statistics.
• Interpret two-way ANOVA results, including when to focus on simple effects.
• Spot confounding and pseudoreplication in a study description.

Words to know

Experimental unit

The object to which a treatment is randomly assigned.

Factor

An explanatory variable manipulated in an experiment.

Level

A specific value or category of a factor.

Treatment

A specific combination of factor levels applied to units.

Randomization

Using chance to assign units to treatments, balancing other variables on average.

Replication

Applying each treatment to several units so chance variation can be estimated.

Confounding

When the effects of two variables on the response cannot be separated.

Block

A group of similar units within which treatments are randomly assigned.

Randomized block design

A design that randomizes treatments separately within each block.

Factorial design

A design that includes every combination of the levels of two or more factors.

Main effect

The average effect of one factor across the levels of the others.

Interaction

When the effect of one factor depends on the level of another.

Simple effect

The effect of one factor at a single level of another factor.

Interaction plot

A plot of cell means showing one line per level of a factor; non-parallel lines indicate interaction.

Placebo

An inactive treatment given to the control group so expectations are equal.

Double-blind

Neither subjects nor evaluators know who receives which treatment.

Pseudoreplication

Treating non-independent measurements as independent replicates.

Sample practice questions

30 more questions like these, from level 1 to 5, are in the practice set.