Stop Peeking: Why Your A/B Test Tool Is Lying to You

Imagine you're a product manager. You launch an A/B test on Monday, and by Tuesday afternoon you see a 15% lift in conversion for the variation. Your impulse is to shout, 'Ship it!' But if you stop the test now, you've just committed a classic statistical sin. The tool you're using—if it's a fixed-horizon frequentist tool—is not designed for you to peek. Every time you check the results and consider stopping early, you risk inflating your false-positive rate from a nominal 5% to as high as 30% (Optimizely). That's not a typo. You could be shipping a losing change.

I'm the editor here, and I've seen too many teams fall into this trap. The fix is either discipline or a smarter tool. And here's my strong opinion: if you can't trust yourself to pre-commit to a sample size and end date, you need to switch to a sequential testing method that lets you peek safely. Let's bust some myths.

What's the big deal with peeking, anyway?

Peeking means checking your test results before the predetermined sample size is reached. The problem is that if you repeatedly test for significance as data rolls in, you're more likely to see a false positive. Evan Miller, who's written extensively on this, shows that if you peek at an ongoing experiment ten times, what you think is 1% significance is actually just 5% significance (Evan Miller). In a worst-case scenario, where you stop as soon as you see 5% significance after every observation, the actual false-positive rate can climb to 26.1% (Evan Miller). So your 'significant' result might be pure noise.

But doesn't a p-value of 0.03 mean there's a 97% chance the variation is better?

No, and this is a common misconception. A p-value is the probability of observing data as extreme as what you saw, assuming the treatment has no effect. A p-value of 0.03 means such a result would occur by chance 3% of the time under the null hypothesis (Optimizely). It does not mean there's a 97% chance the treatment works. It's a measure of surprise, not a probability of truth. So, even if you don't peek, a p-value doesn't give you what you probably want—a direct probability that one version is better. That's where Bayesian methods come in, which output probabilities like 'there's a 92% probability that Variant B is better' (Optimizely).

So should I just use Bayesian testing?

Bayesian testing has advantages. It allows continuous monitoring without inflating false positives (Optimizely), and it can incorporate prior knowledge (VWO). VWO claims that Bayesian approaches can provide actionable results almost 50% faster while focusing on statistical significance, and they can directly state the probability that one variation has a lower conversion rate than another (VWO). But Bayesian methods aren't magic—they require prior choices, and they can be computationally intense. If you're using a frequentist tool, you can still be safe if you follow the rules.

What's the safest way to run a frequentist test?

Pre-commit to a sample size and end date. The sample size is based on your baseline conversion rate and the minimum detectable effect (MDE) you care about (Optimizely sample size guide). For a simple rule of thumb, Evan Miller suggests n = 16 times (sigma-squared / delta-squared), where delta is the minimum effect you want to detect and sigma-squared is the expected sample variance (Evan Miller). For a binomial conversion rate, variance is p(1-p). So, say your baseline conversion is 10% (p=0.1), and you want to detect a 20% relative lift (delta=0.02). Then variance = 0.1*0.9 = 0.09, so n = 16*(0.09/0.0004) = 3600 per variation. That's a lot of visitors, but it's honest.

But what if I can't wait that long?

Then you need a sequential testing method. Sequential testing lets you peek at any time and still get valid results (Optimizely Stats Engine). Evan Miller's simple sequential procedure is a great start: choose a sample size N, assign subjects 50/50, and stop when the treatment-minus-control success count reaches 2 times the square root of N (declaring a winner) or the total success count reaches N (declaring no winner). In some cases, this can reduce the number of observations needed by 50% or more (Evan Miller). Tools like Optimizely's Stats Engine, built with Stanford statisticians, use sequential testing and can call a winner or loser up to 2.5 times as fast for experiments over 50,000 visitors (Optimizely Stats Engine).

Isn't sequential testing less reliable?

No, it's more reliable in practice because it accounts for continuous monitoring. The always-valid p-values and confidence intervals from sequential analysis let you take advantage of data as it becomes available while providing valid inference whenever you decide (arXiv - Always Valid Inference). When Optimizely re-ran 48,000 historical experiments, fixed-horizon statistics declared a winner or loser in 36% of tests, while the sequential Stats Engine did so in 22%—uncovering 39% fewer conclusive results (Optimizely Stats Engine). That's not less reliable; that's fewer false positives.

What about A/A tests and other quality checks?

A/A tests—where you test two identical versions—are a great way to validate your setup. Optimizely recommends running A/A calibration tests quarterly (Optimizely A/A testing glossary). If such a test reports a statistically significant difference (greater than 95% significance), that's a red flag that your implementation is flawed (Optimizely A/A testing glossary). Also, watch for Sample Ratio Mismatch (SRM), where the observed sample ratio differs from expected—a symptom of data quality issues (KDD 2019). Ignoring an SRM can lead to shipping a bad change that appears good (KDD 2019).

So here's my takeaway: stop peeking with fixed-horizon tools. Either pre-commit to a sample size and end date, or switch to a sequential testing tool that lets you peek safely. If you're using a tool that doesn't support sequential testing, you're flying blind. And if you're tempted to stop early, remember Evan Miller's warning: repeated significance testing always increases false positives. Your optimization tool isn't broken; you're using it wrong. But the good news is, smarter tools exist.

Sources

• Optimizely - https://www.optimizely.com/optimization-glossary/ab-testing/
• Evan Miller - https://www.evanmiller.org/how-not-to-run-an-ab-test.html
• Optimizely Stats Engine - https://www.optimizely.com/insights/blog/statistics-for-the-internet-age-the-story-behind-optimizelys-new-stats-engine/
• Optimizely sample size guide - https://www.optimizely.com/insights/blog/how-to-calculate-sample-size-of-ab-tests/
• arXiv - Always Valid Inference - https://arxiv.org/abs/1512.04922
• Optimizely A/A testing glossary - https://www.optimizely.com/optimization-glossary/aa-testing/
• KDD 2019 - Diagnosing Sample Ratio Mismatch - https://www.kdd.org/kdd2019/accepted-papers/view/diagnosing-sample-ratio-mismatch-in-online-controlled-experiments-a-taxonom