Hypothesis testing and statistical significance

Why this matters

Every comparison in data raises the same question: is this difference real, or could it have happened by chance? A new checkout page converts 3.6% of visitors against the old page's 3.0%. Sales in the North are higher than in the South this quarter. Before anyone acts on a difference, it helps to know how easily random variation alone could have produced it.

Hypothesis testing is the standard way to answer that, and its vocabulary, null hypotheses, p-values and significance, appears throughout analytical work and on the exam. This lesson covers the framework, what significance does and does not mean, the common tests, the two kinds of error, and how an A/B test runs from design to decision.

The lesson

Null and alternative hypotheses

A hypothesis test starts with two competing statements.

The null hypothesis (H0) is the default of "no effect" or "no difference": the new page converts at the same rate as the old one; average spend is the same in both regions.

The alternative hypothesis (H1 or Ha) is what you suspect instead: the new page converts at a different rate. An alternative can be two-sided ("the rates differ") or one-sided ("the new page is better"). Two-sided tests are the safer default, because they do not assume the direction in advance.

The test then asks how surprising the observed data would be if the null hypothesis were true. If the data would be very surprising, that is evidence against the null. If it would not, the test has found no convincing evidence of a difference, which is not the same as showing there is none.

The hypotheses, the test and the threshold for significance should all be decided before looking at the results. Choosing them afterwards, to fit what the data happens to show, invalidates the test.

p-values, significance levels and what 'significant' does not mean

The p-value is the probability of seeing a result at least as extreme as the one observed, assuming the null hypothesis is true. A small p-value means the data would be unusual if there were really no difference.

The significance level, alpha (α), is the threshold set in advance, most often 0.05. If the p-value is below alpha, the result is called statistically significant and the null hypothesis is rejected. If not, the null is not rejected, which is deliberately weaker wording than "accepted".

Three misreadings are common enough to be exam material:

• The p-value is not the probability that the null hypothesis is true. It is a probability about the data, calculated by assuming the null is true.
• Significant does not mean important. With a very large sample, a difference too small to matter commercially can be highly significant.
• Not significant does not mean no difference. With a small sample, a real and important difference can fail to reach significance, because the test had too little data to detect it.

The checkout example shows the last two points together. If each page had 5,000 visitors, the conversion rates of 3.0% and 3.6% give a p-value of about 0.09: not significant at 0.05. The same rates with 20,000 visitors per page give a p-value below 0.001. The difference did not change; the evidence about it did. Reporting the size of the effect, with a confidence interval, alongside the p-value avoids both misreadings.

t-tests and chi-square tests, and when each applies

The right test depends on the type of data and the question.

A t-test compares means of a numeric variable.

• A one-sample t-test compares a sample mean with a fixed value: is the average delivery time different from the promised 3 days?
• An independent two-sample t-test compares the means of two separate groups: do North and South customers spend different amounts on average?
• A paired t-test compares two measurements on the same subjects: did the same stores sell more after a layout change than before?

t-tests assume roughly normal data or reasonably large samples, and the two- sample version is usually run in a form that does not assume equal variances.

A chi-square test works on counts in categories.

• The chi-square test of independence asks whether two categorical variables are related: does the proportion of customers who churn differ by segment?
• The goodness-of-fit test asks whether observed counts match expected ones: are orders spread evenly across the days of the week?

Comparing two proportions, such as two conversion rates, can be done with a z-test for proportions or a chi-square test, which give equivalent answers for two groups. For more than two group means, ANOVA (analysis of variance) extends the t-test. When data is clearly not normal and samples are small, non-parametric alternatives such as the Mann-Whitney test compare groups without the normality assumption.

Type I and Type II errors

A test's conclusion can be wrong in two ways.

A Type I error is a false positive: rejecting the null hypothesis when it is actually true, concluding the new page is better when it is not. The probability of a Type I error is alpha, which is why alpha is set low.

A Type II error is a false negative: failing to reject the null when it is false, missing a real improvement. Its probability is called beta, and power is 1 − beta: the probability of detecting an effect of a given size if it exists.

The two trade off. Lowering alpha to reduce false positives makes real effects harder to detect, raising false negatives. The way to reduce both is more data: larger samples increase power without loosening alpha.

Multiple testing makes Type I errors more likely. Testing twenty independent comparisons at alpha 0.05 gives a good chance that at least one is "significant" by chance alone. Analyses that test many metrics or many segments should adjust for it, for example by using a stricter threshold, or treat isolated significant results as leads to confirm rather than findings.

A/B tests from design to decision

An A/B test is a controlled experiment: users are randomly assigned to version A (the control) or version B (the change), and a chosen metric is compared. Randomisation is the key, because it makes the two groups alike in everything except the change, so a difference in outcome can be attributed to it.

A sound A/B test follows a sequence:

1. State the hypothesis and the primary metric before starting, such as "the new checkout increases conversion rate".
2. Decide the minimum effect worth detecting and calculate the sample size needed to detect it with adequate power, often 80%.
3. Randomise assignment and check that the groups are balanced.
4. Run for the planned duration, including full weekly cycles, and do not stop early the moment the result looks significant. Repeatedly checking and stopping at the first significant reading inflates false positives.
5. Analyse with the planned test, and report the effect size with a confidence interval as well as the p-value.
6. Decide using both the statistics and the business context: a significant but tiny lift may not justify the cost of the change.