10  Sampling

Learning Objectives

By the end of this module you should be able to:

  1. Define a population, sample, parameter, and statistic.
  2. Explain why we sample instead of measuring the whole population.
  3. Describe the concept of sampling variability.
  4. Understand what a sampling distribution is and why it matters.
  5. State the Central Limit Theorem and explain its implications.
  6. Recognize sources of sampling bias.

10.1 Population and Sample

Two of the most important words in statistics. Please internalize the distinction.

A population is the complete set of things you want to draw conclusions about. All the canola fields in Saskatchewan in 2025. All the farmers in Canada. All the possible outcomes of a hypothetical experiment.

A sample is the subset you actually observe. 200 randomly selected canola fields. 500 farmers who answered your survey. Ten repetitions of the experiment.

We measure things in the sample and use them to make inferences about the population. Nearly every statistical procedure in this book — and most you will ever encounter — is some version of this.

Two more words:

A parameter is a number describing the population. The mean canola yield in Saskatchewan in 2025. The average farm size. The proportion of organic operations. We usually don’t know parameters directly.

A statistic is a number computed from the sample. The mean of the 200 fields you sampled. The average farm size of the 500 respondents. The proportion of organic operations in your sample. Statistics are known; parameters are (usually) not.

The core job of statistics is to use the known statistic to estimate the unknown parameter, with honest accounting for uncertainty.

10.2 Why Sample?

It would be nice to measure the entire population. Why don’t we?

  • Cost. Counting every canola field in Saskatchewan costs millions of dollars. Counting a random sample of 200 costs a small fraction.
  • Speed. A full census takes years. A sample takes weeks.
  • Destruction. Some measurements destroy the thing being measured. You cannot measure the moisture content of every kernel of grain in the world.
  • Impossibility. For hypothetical populations (“all possible repetitions of this experiment”), there is no way to measure the whole thing.

Even when a census is feasible — Statistics Canada does one every few years — it is often not better than a well-designed sample, because the scale introduces errors of its own (data entry mistakes, missing observations, lag).

For most practical purposes, a good sample is as good as a census. The key word is “good.”

10.3 Sampling Variability

Here is the key insight of the rest of this course. When you take a sample, you get one specific set of observations. If you had taken a different sample — drawn different random fields — you would have gotten different observations, and probably a different sample mean. The sample statistic is itself random. Sampling variability is the name for this.

To make this concrete, let’s simulate. Pretend the population of canola yields is \(N(45, 8^2)\). Take a sample of 30, compute its mean.

set.seed(42)
population_mean <- 45
population_sd <- 8

sample_1 <- rnorm(30, mean = population_mean, sd = population_sd)
mean(sample_1)
# result might be, say, 43.7

Now do it again:

sample_2 <- rnorm(30, mean = population_mean, sd = population_sd)
mean(sample_2)
# maybe 46.1

And again, a thousand times:

many_means <- replicate(1000, mean(rnorm(30, mean = 45, sd = 8)))
hist(many_means, breaks = 30)
mean(many_means)   # close to 45
sd(many_means)     # close to 8 / sqrt(30) ≈ 1.46

What do we see?

  1. The means are not all the same. They vary from sample to sample. This is sampling variability.
  2. The means are centered around the true population mean (45). On average, the sample mean is the right answer.
  3. The spread of the sample means (standard deviation ≈ 1.46) is much smaller than the spread of the individual values in the population (which is 8). Averaging reduces variability.

That distribution of sample means — the histogram you just made — is called the sampling distribution of the mean. It is not a distribution of your data; it is a distribution of the statistic.

10.4 The Central Limit Theorem

The sampling distribution of the mean has three remarkable properties:

  1. Its mean equals the population mean \(\mu\). On average, sample means are unbiased.
  2. Its standard deviation equals \(\sigma / \sqrt{n}\), where \(\sigma\) is the population standard deviation and \(n\) is the sample size. This is called the standard error of the mean (SE). Larger samples have smaller standard errors.
  3. Its shape is approximately normal for large enough \(n\), regardless of the shape of the population distribution.

That third point is the Central Limit Theorem (CLT) and it is the most important theorem in applied statistics. It says: even if the underlying data is highly skewed, has weird shapes, has outliers — the distribution of the sample mean will be approximately normal, as long as the sample is large enough.

How large is “large enough”? Depends on the shape. For roughly symmetric populations, \(n = 30\) is usually plenty. For highly skewed populations (like farm incomes), you might need \(n = 100\) or more. But the theorem guarantees convergence as \(n\) grows.

This is why the normal distribution is everywhere in inferential statistics. Even when the raw data is not normal, the sample mean is. And most of our inferences are about means.

10.5 Standard Error vs Standard Deviation

A common source of confusion. They are different things:

  • Standard deviation (\(s\) or \(\sigma\)): measures the spread of the individual values in a sample or population. Has units of the variable.
  • Standard error (\(SE\)): measures the spread of the sampling distribution of a statistic, usually the mean. For the mean of a sample, \(SE = s / \sqrt{n}\). Has the same units.

A dataset has one standard deviation. A statistic (like the mean) has a standard error that depends on the sample size.

When you see “\(45 \pm 1.5\)” in a report, ask: is that a standard deviation (showing the spread of the data) or a standard error (showing the precision of the estimate)? They answer different questions, and confusing them is a major source of statistical malpractice.

10.6 Sampling Bias

Sampling variability is about the random fluctuation around the true answer. Sampling bias is about being systematically wrong.

Examples:

  • Selection bias. Your sample is not representative of the population. You survey farmers about the new program by calling them during daytime business hours — but the farmers who answer are systematically different from the ones who are out in the field working.
  • Self-selection bias. You post a survey online and whoever wants to can respond. The people who respond are the ones who care strongly about the topic — who are usually not representative of everyone.
  • Survivorship bias. You measure yields from all the fields that were harvested. You miss the fields that failed and were never harvested. Your average yield is biased upward.
  • Recall bias. You ask people to report their pesticide use last year. They remember some things and forget others, and the remembering is not random.

Bias cannot be fixed by a larger sample size. A bigger biased sample just gives you the wrong answer with more precision. The only fix for bias is better study design.

The gold standard for unbiased sampling is the simple random sample: every member of the population has an equal chance of being selected, and selections are independent. This is an ideal that is sometimes hard to achieve in practice. More sophisticated designs (stratified sampling, cluster sampling) exist for specific situations; they are the subject of a course of their own.

For AREC 261, the takeaway is: always ask “how was this sample drawn, and does it plausibly represent the population I care about?” If the answer is unclear, so are the conclusions.

10.7 A Thought Experiment: Election Polls

Before an election, polling companies survey maybe 1500 voters and report which candidate leads. 1500 people out of many millions of voters. How is that enough?

The answer: because of the Central Limit Theorem and the \(1/\sqrt{n}\) formula. The standard error of a proportion from a sample of 1500 is about \(\sqrt{0.25 / 1500} \approx 0.013\) — about 1.3 percentage points. A 95% “margin of error” is about twice that, so around \(\pm 2.6\) points.

That’s honest enough for most practical purposes. If candidate A is leading candidate B by 10 points, they are probably really leading. If they are leading by 1 point, the poll cannot distinguish that from a tie.

Notice what sample size did not help with: selection bias. If the sample was drawn in a way that systematically misses certain types of voters (famously, “shy Trump voters” in the 2016 U.S. election, who didn’t admit their preference to pollsters), then no amount of \(n\) fixes that. The poll is systematically wrong regardless of the math.

10.8 Test Bank Sample

  1. (Definitions.) What is the difference between a parameter and a statistic?
  2. (CLT.) State the Central Limit Theorem in your own words. Why does it matter?
  3. (Standard error.) You sample 100 fields and the sample mean yield is 45 with sample standard deviation 8. What is the standard error of the mean?
  4. (Bias.) You run an online survey on a farming website to estimate the average age of farmers. Why might this give a biased estimate?
  5. (Sample size.) If you want to cut your standard error in half, by what factor do you need to increase your sample size?

10.9 Practice Exercises

  1. Write an R simulation that takes 1000 samples of size 30 from a skewed distribution and verifies that the sample means are approximately normal (CLT).
  2. Compute the standard error for a sample of [TBD] yields and write a sentence interpreting it.
  3. Identify a news article that reports a survey result and evaluate its methodology.
  4. [TBD: a designed sampling exercise.]