Run the same A/B test at two sample sizes

short · 35 min · Objective 3.2

Task

Calculate a two-proportion z-test by hand for an A/B test in which the new page converts 3.6% of visitors and the old page 3.0%, first with 5,000 visitors per page and then with 20,000. The rates never change; watch what the p-value does.

Steps

  1. Write lab/ab/ztest.py taking conversions and visitors for each page. It calculates the pooled proportion, the standard error sqrt(p * (1 - p) * (1/n1 + 1/n2)), the z statistic, and the two-sided p-value as math.erfc(abs(z) / math.sqrt(2)).
  2. Run it for 150 of 5000 against 180 of 5000, and record small test: z=Z p=P in lab/ab/results.txt, z to two and p to three decimal places.
  3. Run it for 600 of 20000 against 720 of 20000, and record large test: z=Z p=P.
  4. Record in lab/ab/decision.txt the decision at a 0.05 significance level for each test, as small: significant or small: not significant, and likewise for large.

Verify

These checks run in a POSIX shell with Python 3: Terminal on macOS or Linux, and on Windows Git Bash (it comes with Git for Windows) or WSL. On Windows, python3 is often a Microsoft Store placeholder that prints “Python was not found”; run alias python3=py in Git Bash first.

grep -E '^(small|large) test: ' lab/ab/results.txt
grep -E '^(small|large): ' lab/ab/decision.txt

The small test gives z=1.68 and p=0.093, not significant; the large test gives z=3.36 and p=0.001, significant. The difference between the pages is identical in both. A non-significant result from a small test is not evidence that the pages perform the same -- only that 5,000 visitors each was too few to tell.

This is an independent study companion for CompTIA Data+ DA0-002 and is not produced by or endorsed by CompTIA.