Stop Peeking, Start Sequenti: Why Fixed-Horizon A/B Tests Are a Trap

If you're still running classic fixed-horizon A/B tests, you're doing it wrong. The standard approach — pick a sample size, run the test, and declare victory at the end — is a statistical time bomb. It's the reason so many test results are wrong. I'm not talking about bad test design or low traffic. I'm talking about the method itself.

The truth is, sequential testing is the only sane way to run experiments. It solves the peeking problem, cuts your sample size, and gives you valid results even if you check your dashboard every hour. Don't believe me? Let's put the two approaches head-to-head.

The Fixed-Horizon Fallacy

Fixed-horizon testing is the classic textbook method. You decide ahead of time how many visitors you need, run the test, and then and only then look at the p-value. The math is clean, but the reality is messy. No one actually waits. You check the dashboard on day two and see a 95% significance. You stop. You ship. You're wrong.

The problem is peeking. Checking results daily and stopping as soon as significance appears can inflate the false-positive rate from a nominal 5% to as high as 30% (Optimizely). Repeated significance testing always increases false positives — the more you peek, the more your significance levels are off (Evan Miller). If you peek ten times, what you think is 1% significance is actually just 5%. In the worst case, stopping as soon as 5% significance appears after every observation yields a true false-positive rate of 26.1% (Evan Miller).

That's not an edge case. That's the rule. And it's not just a statistical curiosity. It's a business disaster. You're shipping bad features based on false confidence.

The Sequential Alternative

Sequential testing is the antidote. It lets you peek as often as you like without inflating your error rate. Always-valid p-values and confidence intervals mean you can make a decision at any time, and the math holds (arXiv). You don't need to pre-commit to a fixed sample size, and you don't need to guess a minimum detectable effect in advance (Optimizely Stats Engine).

This isn't just theory. Netflix uses sequential canary tests to check streaming-quality metrics across the entire distribution, not just the mean, so they catch regressions like PlayDelay spikes while strictly controlling the false-detection probability (arXiv). That's the real world using sequential methods to make better decisions.

Sample Size: Sequential Wins Big

Fixed-horizon sample size calculations are a pain. You need to know your baseline conversion rate, your minimum detectable effect, your significance level, and your power. The rule-of-thumb formula is n = 16 * (sigma^2 / delta^2) (Evan Miller). For a binomial rate, variance is p(1-p). So if your baseline is 10%, you need about 16 * (0.09 / 0.01) = 144 per variation to detect a 10% relative lift. That's just a rule of thumb, but it's a start.

Sequential methods can cut that sample size dramatically. In some circumstances, a simple sequential procedure can reduce the number of observations needed by 50% or more, especially with low conversion rates (Evan Miller). And when Optimizely re-ran 48,000 historical experiments, their sequential Stats Engine declared a winner or loser in 22% of tests, while fixed-horizon statistics did so in 36% — meaning fixed-horizon was calling 39% more conclusions that sequential testing didn't support (Optimizely Stats Engine). That's not just a sample size win; that's a correctness win.

False Positives: The Real Cost

Fixed-horizon testing is designed to control the false-positive rate at your chosen alpha, say 5%. But that's only true if you never peek. The moment you peek, your error rate balloons. Sequential testing keeps the false-positive rate at the nominal level no matter how often you look.

That's the whole point. With sequential methods, you can monitor your test continuously without worrying about inflating Type I errors. The always-valid p-values from Johari et al. are designed for continuous monitoring (arXiv). You get valid inference whenever you decide to stop.

Speed: Sequential Isn't Slower, It's Faster

You might think that because sequential tests are more conservative, they take longer. Wrong. Optimizely's Stats Engine, for experiments over 50,000 visitors, can call a winner or loser up to 2.5 times as fast as fixed-horizon statistics (Optimizely Stats Engine). And VWO reports that Bayesian approaches can provide actionable results almost 50% faster (VWO). Sequential methods can stop early when there's a clear winner, saving time and resources.

So you're not sacrificing speed for correctness. You're getting both.

Head-to-Head: Sequential vs. Fixed-Horizon

I'm not saying fixed-horizon is always wrong. If you have a well-defined sample size, a clear minimum detectable effect, and you absolutely won't peek, it can work. But that's rare. In practice, you'll peek. And when you do, you're fooling yourself.

Sequential testing is the safer, faster, and more honest way to run experiments. It's not just for big tech. Tools like Optimizely's Stats Engine and Evan Miller's simple sequential procedure make it accessible to anyone.

Who Should Use What

• Use sequential if you're a product manager or growth marketer who wants to check results as they come in without losing sleep over false positives.
• Use sequential if you're running low-traffic tests where sample size is a concern.
• Use fixed-horizon only if you have a rigid process that guarantees no peeking and you can afford the sample size.

But honestly, there's no reason to stick with fixed-horizon. The tools are there, the evidence is clear, and the cost of getting it wrong is too high. Stop peeking, start sequential.

Sources

• Optimizely - A/B Testing: https://www.optimizely.com/optimization-glossary/ab-testing/
• Evan Miller - How Not to Run an A/B Test: https://www.evanmiller.org/how-not-to-run-an-ab-test.html
• Evan Miller - Simple Sequential A/B Testing: https://www.evanmiller.org/sequential-ab-testing.html
• Optimizely Stats Engine: https://www.optimizely.com/insights/blog/statistics-for-the-internet-age-the-story-behind-optimizelys-new-stats-engine/
• arXiv - Always Valid Inference (Johari et al.): https://arxiv.org/abs/1512.04922
• arXiv - Netflix canary testing (Lindon et al.): https://arxiv.org/abs/2205.14762