Y

Statistics · Intermediate

Designing an A/B test that you can trust

3 min read

Why it matters

Product companies such as Flipkart, Swiggy, Razorpay and CRED run experiments on checkout flows, notifications and pricing every week. Product analyst and data scientist interviews usually include an A/B testing case: "we changed the button, conversion went up 3%, should we ship?". The strong answer starts with the design, not the result.

The concept

An A/B test randomly splits users into a control group (A) and a treatment group (B), changes one thing for B, and compares a pre-chosen primary metric.

Before you start, decide:

  1. Hypothesis and primary metric: "the new checkout raises order conversion".

  2. Minimum detectable effect (MDE): the smallest change worth detecting, for example +1 percentage point.

  3. Significance level (usually 5%) and power (usually 80%): how often you accept a false positive, and how often you want to catch a real effect of the MDE size.

  4. Sample size from those three numbers and the baseline rate.

  5. Guardrail metrics that must not get worse: refunds, page load time, support tickets.

  6. Duration: at least one full weekly cycle, even if the sample fills sooner.

Randomisation is what lets you say "B caused the difference". It balances everything else between the groups, including the things you did not think of.

Worked example: how many users do you need?

Say the baseline conversion is 11.7% (the subscription rate in the UCI Bank Marketing calls) and you want to detect a rise to 13.0%.

sample_size.py
from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

baseline, target = 0.117, 0.130
effect = proportion_effectsize(target, baseline)
n = NormalIndPower().solve_power(effect_size=effect, alpha=0.05, power=0.8, ratio=1)
print(round(n))          # about 10,048 per group

small = proportion_effectsize(baseline * 1.05, baseline)   # a 5% relative lift
print(round(NormalIndPower().solve_power(effect_size=small, alpha=0.05, power=0.8)))   # about 48,405 per group

Detecting a 1.3 point lift needs about 10,000 users per group. Detecting a 5% relative lift (11.7% to 12.3%) needs about 48,000 per group. Small effects are expensive to detect. That is why teams agree the MDE before the test, not after.

With 5,000 eligible users a day split evenly, the first design runs about 4 days of traffic. You still run 7 days to cover a full week, because weekday and weekend users behave differently.

Common mistakes

  • Peeking. Checking the p-value every day and stopping when it dips below 0.05 inflates false positives badly. Fix the sample size, or use a sequential method designed for peeking.

  • Changing the metric after seeing results. Pick the primary metric first and write it down.

  • Sample ratio mismatch. If you asked for 50/50 and got 52/48 with large numbers, the assignment is broken. Check with a chi-squared test before reading any result.

  • Novelty effects. Users click new things because they are new. Look at the effect over time.

  • Randomising the wrong unit. Randomise users, not sessions, when the same user can see both versions.

Interview angle

For "conversion went up 3%, ship it?", walk through: was the test pre-registered with a primary metric and sample size, did it run a full week, is the split ratio healthy, what is the confidence interval of the lift, did any guardrail get worse, and is 3% relative or absolute. Then give a recommendation.

Practice task

  1. Compute sample sizes for MDEs of 0.5, 1 and 2 percentage points at a 5% baseline. Plot sample size against MDE.

  2. Write a one-page test plan for a new "cash on delivery" badge on product pages: hypothesis, metric, guardrails, MDE, sample size, duration.

  3. Simulate 1,000 A/A tests (no real difference) where you peek daily and stop at p below 0.05. What share of tests "win"?

Further reading