Home / Courses / Introductory Statistics / Unit 2

Introductory Statistics Unit 2: Collecting data: sampling & experiments

How data is collected decides what you're allowed to conclude. Random sampling lets you generalize; random assignment lets you claim cause and effect.

Covers: Sampling methods, bias, experimental design, scope of inference

Study guide

How data are collected determines what conclusions you can draw. A population is the whole group of interest; a sample is the part actually measured. Sampling methods include the simple random sample (every group of size n equally likely), stratified random sampling (divide into similar groups called strata and sample from each), cluster sampling (randomly choose whole groups and use everyone in them), and systematic sampling (every kth individual).

Bias is a systematic error that favors certain outcomes. Convenience samples and voluntary response samples (people choose to respond) are biased. Undercoverage leaves part of the population out, nonresponse occurs when chosen individuals don't answer, and response bias comes from wording, interviewers or people misreporting. A bigger sample doesn't fix bias.

An observational study measures without intervening and can show association but not causation, because of possible confounding variables. An experiment imposes treatments on experimental units. Good experiments follow four principles: comparison of two or more treatments, random assignment of treatments, control of other variables, and replication on enough units. Blocking groups similar units and randomizes within each block to reduce variability; matched pairs is a special case. Placebos and blinding prevent expectations from affecting results.

Scope of inference: random sampling lets you generalize to the population, and random assignment lets you conclude cause and effect.

To describe random assignment, label the units 1 to N, use a random number generator to choose N/2 different labels (ignoring repeats) for one treatment, and give the remaining units the other treatment. Drawing well-mixed slips from a container also works. For a stratified random sample, name the strata, number the individuals within each stratum, and use a random number generator to take a simple random sample from every stratum. Good strata are similar within and differ from one another.

When identifying bias, state its likely direction: say whether the method would tend to overestimate or underestimate the parameter, and explain why. Random assignment tends to balance variables you didn't control across the treatment groups. Then a large difference in responses can be attributed to the treatments.

Key ideas

• Simple random sample, stratified (sample within groups), cluster (sample whole groups) and systematic sampling.
• Bias is a systematic error: undercoverage, nonresponse, response bias, voluntary response and convenience sampling.
• Observational studies watch; experiments impose a treatment.
• Good experiments use comparison, random assignment, control of other variables and replication.
• Blocking groups similar units before randomly assigning treatments within each block.

Formulas and rules

Common mistakes

• Confusing random sampling with random assignment.
• Claiming causation from an observational study.

Words to know

Stratified vs cluster sample

Stratified samples from every group; cluster randomly picks whole groups and uses everyone in them.

Random assignment

Lets you conclude cause and effect for the experimental units.

Random sampling

Lets you generalize results to the population sampled.

Voluntary response bias

People choose to respond, so those with strong opinions are overrepresented.

Four principles of experimental design

Comparison, random assignment, control, replication.

Blocking

Grouping similar units, then randomizing within groups, to reduce variability.

Population vs. sample

The whole group of interest vs. the part actually measured.

Simple random sample (SRS)

Every group of size n is equally likely to be chosen.

Stratified random sample

Divide into similar groups (strata) and take an SRS from each.

Cluster sample

Randomly choose whole groups and use everyone in them.

Undercoverage

Part of the population is left out of the sampling process.

Nonresponse

Chosen individuals can't be reached or refuse to answer.

Confounding variable

A variable tied to the explanatory variable that also affects the response.

Placebo effect

Improvement caused by expecting a treatment to work.

Double-blind

Neither subjects nor evaluators know who got which treatment.

Scope of inference

Random sampling allows generalizing; random assignment allows cause and effect.

Sample practice questions

13 more questions like these, from level 1 to 5, are in the practice set.