Why Your A/B Test Is Lying: The 26.1% False-Positive Trap

You’ve been running an A/B test for a week. You peek at the dashboard—your variation is up 12% with a p-value of 0.04. You stop the test, ship it, and it flops. What went wrong? The answer: you peeked. And peeking can turn a harmless 5% false-positive rate into a 26.1% disaster.

Here’s the hard truth: if you check results daily and stop the moment significance appears, you’re not running a 5% significance test—you’re running a 26.1% false-positive machine. That’s not a guess; it’s a calculation from Evan Miller, who showed that in a worst-case scenario where you test after every observation and stop at 5% significance, the actual false-positive rate hits 26.1% (Evan Miller - How Not to Run an A/B Test).

This article answers one question: how do you run an A/B test that you can actually trust? The answer is sequential testing—a method that lets you peek without inflating your error rates. I’ll walk you through why peeking breaks fixed-horizon tests, what sequential testing does differently, and exactly how to implement it.

Why Peeking Breaks Your A/B Test

Classic A/B testing relies on a fixed-horizon design. You pre-commit to a sample size, run the test, and analyze the data once. That’s the theory. In practice, you’re human. You check results after a day, then again after two days, and if something looks good, you stop early.

That behavior—peeking—is a statistical sin. Repeated significance testing always increases the rate of false positives, so the more you peek, the more your reported significance levels are off (Evan Miller - How Not to Run an A/B Test). If you peek ten times, what you think is 1% significance is actually just 5% significance. That’s a massive gap.

Optimizely puts it bluntly: when paired with classical statistics, intuitive practices such as peeking and testing many goals at once can increase the chance of incorrectly declaring a winner by over 5x (Optimizely Stats Engine). So if you’re relying on fixed-horizon p-values and peeking, you’re not getting 95% confidence—you’re getting something closer to 70%.

The root cause is that fixed-horizon tests assume you look at the data exactly once. Every additional look is an opportunity for random noise to cross the significance threshold, and the math doesn’t account for that.

How Sequential Testing Fixes This

Sequential testing is the antidote. Instead of forcing you to pick a sample size and wait, it lets you peek at any time and still get valid p-values. That’s the promise of “always-valid inference” (arXiv - Always Valid Inference).

Evan Miller’s simple sequential procedure is a good example. You choose a sample size N, assign subjects randomly 50/50, and stop when the treatment-minus-control success count reaches 2 times the square root of N (declaring a winner) or the total success count reaches N (declaring no winner). In some circumstances, this method can reduce the number of observations needed by 50% or more, and it works extremely well with low conversion rates (Evan Miller - Simple Sequential A/B Testing).

Why does it work? Because the stopping rule is baked into the analysis. You’re not using a fixed threshold that assumes one look; you’re using a boundary that adjusts for every look you might take. That’s what makes it valid.

Netflix uses sequential canary tests for this exact reason. They test differences across the entire distribution, not just the mean, so a metric like PlayDelay—the time for a title to start once play is pressed—can flag a severe regression even if the mean doesn’t change (arXiv - Netflix canary testing). They don’t wait for a fixed end date; they monitor continuously and stop when the evidence is clear.

What the Numbers Say

If you’re still skeptical, look at the data. Optimizely re-ran 48,000 historical experiments and compared fixed-horizon statistics to their sequential Stats Engine. Fixed-horizon declared a winner or loser in 36% of tests, while the sequential engine declared one in only 22%—uncovering 39% fewer conclusive results (Optimizely Stats Engine). That’s not because sequential testing is weaker; it’s because fixed-horizon tests were crying wolf.

That 39% reduction means thousands of experiments that looked conclusive were actually false positives. If you’ve ever shipped a test result that didn’t hold up, this is why.

Sequential testing also makes you faster when there is a real effect. Optimizely’s Stats Engine can call a winner or loser up to 2.5 times as fast as fixed-horizon statistics for experiments over 50,000 visitors (Optimizely Stats Engine). That’s a huge operational advantage.

Here’s a comparison to make it concrete:

The false-positive rate is the killer. With fixed-horizon and peeking, you’re not at 5%—you’re at 26.1% (Evan Miller). That’s one in four tests being wrong. Would you ship a feature with a 25% chance of doing nothing?

How to Implement Sequential Testing Today

You don’t need to build your own math. Tools like Optimizely’s Stats Engine already use sequential testing. If you’re using Optimizely, you’re covered—just make sure you’re not using a fixed-horizon mode if you have the choice.

If you’re doing it yourself, Evan Miller’s simple procedure is a good start. Here’s the step-by-step:

• Choose a sample size N (you can use the rule of thumb n = 16 * sigma^2 / delta^2, where delta is the minimum effect you care about).
• Assign visitors randomly 50/50.
• Track the difference in success counts (treatment minus control).
• Stop when that difference reaches 2 * sqrt(N) — declare a winner.
• Stop when the total success count reaches N — declare no winner.

That’s it. No p-values, no peeking anxiety. You can check the dashboard daily; the rule stays valid.

Warning: Don’t use a fixed-horizon calculator and then peek anyway. That’s the worst of both worlds. If you’re going to peek, use a sequential method.

What I’d Actually Do

Stop using fixed-horizon p-values for any test where you might look at the data before the planned end. That’s most tests, because you’re human. Switch to sequential testing. If you’re on Optimizely, enable Stats Engine. If you’re rolling your own, implement Evan Miller’s simple procedure. It’s free, it’s simple, and it keeps your false-positive rate at 5% no matter how often you peek.

I’d also run A/A tests quarterly (Optimizely recommends this) to make sure your implementation isn’t broken. And always check for Sample Ratio Mismatch (SRM)—if the observed sample ratio differs from the expected, your data is bad and your results are suspect (KDD 2019).

The bottom line: your A/B test is lying to you if you’re peeking with fixed-horizon stats. The fix isn’t more discipline—it’s better math. Sequential testing is that better math.

Sources

• Evan Miller - How Not to Run an A/B Test: https://www.evanmiller.org/how-not-to-run-an-ab-test.html
• Evan Miller - Simple Sequential A/B Testing: https://www.evanmiller.org/sequential-ab-testing.html
• Optimizely Stats Engine: https://www.optimizely.com/insights/blog/statistics-for-the-internet-age-the-story-behind-optimizelys-new-stats-engine/
• arXiv - Always Valid Inference: https://arxiv.org/abs/1512.04922
• arXiv - Netflix canary testing: https://arxiv.org/abs/2205.14762
• KDD 2019 - Diagnosing Sample Ratio Mismatch: https://www.kdd.org/kdd2019/accepted-papers/view/diagnosing-sample-ratio-mismatch-in-online-controlled-experiments-a-taxonom