Sampling, bias and confidence intervals

Why this matters

Most data describes a sample, not everyone. A survey reaches some customers, a test runs on some users, an audit checks some invoices. The analyst's claim is nearly always about the larger group, and whether that claim holds depends on how the sample was chosen and how big it was. A large sample chosen badly is worse than a small one chosen well, because it gives a precise-looking wrong answer.

The exam asks about sampling methods, the biases that undermine them, and how uncertainty is expressed. This lesson covers populations and samples, the main sampling methods, common biases, margins of error and confidence intervals, and what sample size can and cannot fix.

The lesson

Population against sample

The population is the entire group a question is about: all customers, all transactions this year, every product in the warehouse. A sample is the subset actually measured.

A number that describes a population is a parameter, such as the true average satisfaction of all customers. A number calculated from a sample is a statistic, such as the average satisfaction of the 200 customers who answered the survey. Statistics are used to estimate parameters, and the estimate is always uncertain, because a different sample would give a slightly different answer.

Sometimes the whole population is available: every order in the database, for example. Then there is no sampling uncertainty about what happened, though there may still be questions about what the data will look like next year, and the data may still have quality problems. Knowing which situation you are in, a census or a sample, changes what the numbers mean.

Random, stratified, cluster and convenience sampling

How a sample is chosen decides whether it represents the population.

• Simple random sampling: every member of the population has an equal chance of selection. It is the foundation of sampling theory, and it needs a complete list of the population to draw from.
• Systematic sampling: every kth member is chosen from a list, such as every tenth invoice. It is easy, and it fails if the list has a pattern that repeats at the same interval.
• Stratified sampling: the population is divided into groups, or strata, such as regions, and a random sample is drawn from each. It guarantees every group is represented, and is especially useful when a small group matters, such as a region with few customers.
• Cluster sampling: the population is divided into clusters, such as stores or schools, some clusters are chosen at random, and everyone in them is measured. It is cheaper when travelling to each member is costly, and it is less precise if clusters differ from each other.
• Convenience sampling: whoever is easiest to reach, such as customers who happen to visit a web page. It is quick and cheap and almost always biased, because the people easiest to reach differ from those who are not.

The first four are probability sampling methods: each member has a known chance of selection, which is what allows uncertainty to be calculated. Convenience and volunteer samples are non-probability samples, and their results describe the sample more safely than the population.

Selection, survivorship and response bias

Bias is a systematic error that pushes results in one direction. Unlike random error, it does not shrink as the sample grows.

• Selection bias arises when the way a sample is chosen makes some members more likely to be included. A satisfaction survey sent only to customers who have ordered in the last month excludes those who stopped ordering, who are likely the least satisfied.
• Survivorship bias arises from looking only at the cases that survived a process. Studying only successful products to find what makes products succeed ignores the failed ones that shared the same features.
• Non-response bias arises when those who do not respond differ from those who do. If unhappy customers are more likely to ignore a survey, the responses overstate satisfaction.
• Response bias arises from how questions are asked or answered: leading wording, a desire to give the socially acceptable answer, or questions people cannot answer accurately.

The course's survey data is a useful example: it contains 200 responses, and nothing in it says how many customers were asked. Without the response rate and some knowledge of who did not respond, its average satisfaction describes respondents, not customers.

Margin of error and confidence intervals

Because a sample statistic varies from sample to sample, it is reported with a measure of that uncertainty.

A confidence interval is a range, calculated from the sample, that is likely to contain the true population value. A 95% confidence interval is built by a method that, over many repeated samples, captures the true value 95% of the time. The margin of error is the half-width of that interval.

For a mean, a 95% interval is approximately the sample mean plus or minus 1.96 standard errors, where the standard error is the standard deviation divided by the square root of the sample size. The practice survey's satisfaction scores have a mean of 3.575 and a standard deviation of about 1.35 across 200 responses, giving a margin of error of about 0.19 and a 95% interval of roughly 3.39 to 3.76.

For a proportion, the largest margin of error occurs at 50%. For a simple random sample, it is about ±9.8 percentage points with 100 respondents, ±4.9 with 400 and ±3.1 with 1,000.

Two cautions matter. A confidence interval does not mean there is a 95% probability that the true value lies in this particular interval; the true value is fixed, and the 95% describes the method. And the interval covers random sampling error only. It says nothing about bias: a biased sample produces a narrow, confident interval around the wrong answer.

Sample size, and what a small sample can and cannot say

Larger samples give narrower intervals, but with diminishing returns: the margin of error shrinks with the square root of the sample size, so halving it needs four times the sample. Going from 400 to 1,600 respondents cuts a ±4.9-point margin to about ±2.5.

Small samples have specific limits. Their intervals are wide, so they can detect only large differences. A single extreme value moves their statistics substantially. And subgroups are smaller still: a survey of 400 split across four regions and two segments leaves around 50 per group, with margins near ±14 points, too wide to compare groups meaningfully.

What sample size cannot fix is bias. A million responses from a biased sample are a million precise measurements of the wrong group. Choosing the sample well matters more than making it large, and a report should state both the sample size and how the sample was drawn, so readers can judge each.