Stop Trusting Your A/B Test: The 26.1% False-Positive Trap

If you peek at your A/B test results every day and stop as soon as you see significance, you're not running an experiment—you're playing a game of chance you'll lose. In the worst case, where you stop the moment 5% significance appears, the actual false-positive rate can hit 26.1% (Evan Miller). That means over a quarter of your 'winners' are pure noise. But you don't have to fall into this trap.

Why does peeking inflate my false positives?

Classic statistics assumes you'll look at the data once—at a pre-planned end time. Every time you peek and consider stopping, you're running a separate hypothesis test on the same data. Each test has a 5% chance of a false positive, but those chances compound. Repeated significance testing always increases the rate of false positives, so the more you peek, the more your reported significance levels are off (Evan Miller). If you peek ten times, what you think is 1% significance is actually just 5% (Evan Miller).

Can I peek at all?

Yes, but with a strict rule: look, don't touch. Peeking at the data is acceptable as long as you restrain yourself from stopping an experiment before it has run its course. The recommended practice is to decide on a sample size in advance (Evan Miller). In other words, you can sneek a peek to check that your tracking is working, but you cannot call a winner early based on significance.

What's the alternative to fixed-horizon tests?

Sequential testing. It lets you see results that are always valid any time you peek, without needing to guess a minimum detectable effect in advance (Optimizely Stats Engine). Sequential sampling allows you to stop a trial early if the treatment appears to be a winner, addressing the peeking problem of eager experimenters who misuse fixed-sample methods (Evan Miller). Netflix uses sequential canary tests that test differences across the entire distribution rather than only the mean, so a streaming-quality metric such as PlayDelay (the time for a title to start once play is pressed) with a heavier treatment tail is flagged as a severe performance regression while the false-detection probability is strictly controlled (arXiv - Netflix canary testing).

Does sequential testing really work?

Yes, and it can save you time and money. In some circumstances the sequential procedure can reduce the number of observations required for a successful experiment by 50% or more, and it works extremely well with low conversion rates (Evan Miller). Optimizely's Stats Engine, developed with Stanford statisticians, began powering results for all Optimizely customers on January 21, 2015; for experiments over 50,000 visitors it can call a winner or loser up to 2.5 times as fast as fixed-horizon statistics (Optimizely Stats Engine). When Optimizely re-ran 48,000 historical experiments, fixed-horizon statistics declared a winner or loser in 36% of tests while the sequential Stats Engine declared winners or losers in 22%, uncovering 39% fewer conclusive results (Optimizely Stats Engine). That's not a bug—it's the tool telling you that many of your old 'conclusive' tests were actually false positives.

What about Bayesian methods?

Bayesian testing outputs probabilities such as 'there is a 92% probability that Variant B is better' and allows continuous monitoring without inflating false positives (Optimizely). VWO reports that the Bayesian approach can provide actionable results almost 50% faster while focusing on statistical significance, and it can directly state the probability that one variation has a lower conversion rate than another (VWO). If you're tired of the p-value circus, Bayesian might be your cup of tea. But beware: it still requires careful prior specification and doesn't magically fix bad experiment design.

Why is my A/A test showing a significant difference?

If your A/A test (two identical versions) reports a statistically significant difference (greater than 95% significance), that's a red flag that your testing implementation is broken (Optimizely A/A testing glossary). You should check your tracking, randomization, and segmentation. Also watch for Sample Ratio Mismatch (SRM)—when the observed sample ratio differs from the expected ratio. An SRM is a symptom of a variety of data quality issues (KDD 2019). Ignoring an SRM without knowing its root cause may result in a bad product modification appearing to be good and being shipped to users, or vice versa (KDD 2019).

How do I choose a sample size?

Before you launch, set your desired statistical power (usually 80%) and your minimum detectable effect (MDE) (Optimizely). Sample size calculations are based on the control group's baseline conversion rate and the MDE; power analysis then considers the significance level (usually 0.05), the power level (often 0.8), and the expected effect size (Optimizely sample size guide). A rule-of-thumb formula is n = 16 times (sigma-squared / delta-squared), where delta is the minimum effect you wish to detect and sigma-squared is the expected sample variance; for a binomial conversion rate the variance is p(1-p) (Evan Miller). For example, if your baseline conversion is 10% (variance 0.09) and you want to detect a 1% absolute lift (delta = 0.01), you'd need n = 16 * 0.09 / 0.0001 = 14,400 visitors per variation. That's a lot of traffic—so plan accordingly.

What about multiple metrics?

Testing many goals and variations at once increases the chance of incorrectly declaring a winning or losing variation by over 5x (Optimizely Stats Engine). Re-analyzing 48,000 historical experiments, Optimizely found roughly 20% fewer variations with a false discovery rate below 0.1 than with a false positive rate at the same level, illustrating the multiple-comparisons cost of testing many metrics (Optimizely Stats Engine). So pick your primary metric and stick to it.

Bottom line

Stop peeking at your A/B test and stopping early. Pre-commit to a sample size, or better yet, use sequential testing. Your future self—and your conversion rate—will thank you.

Sources

• Evan Miller - How Not to Run an A/B Test - https://www.evanmiller.org/how-not-to-run-an-ab-test.html
• Optimizely Stats Engine - https://www.optimizely.com/insights/blog/statistics-for-the-internet-age-the-story-behind-optimizelys-new-stats-engine/
• Optimizely A/A testing glossary - https://www.optimizely.com/optimization-glossary/aa-testing/
• KDD 2019 - Diagnosing Sample Ratio Mismatch - https://www.kdd.org/kdd2019/accepted-papers/view/diagnosing-sample-ratio-mismatch-in-online-controlled-experiments-a-taxonom
• Optimizely sample size guide - https://www.optimizely.com/insights/blog/how-to-calculate-sample-size-of-ab-tests/
• VWO A/B testing guide - https://vwo.com/ab-testing/