Imagine you're two days into a new A/B test on your checkout page. You glance at the dashboard and see a 10% lift in conversion for the variant. Your pulse quickens. You're tempted to declare victory and ship it. But if you do, you're walking into a trap.
That's the peeking problem, and it's the single most common statistical error in A/B testing. The more you peek, the more likely you are to see a significant result that's actually just noise. The fix isn't willpower—it's choosing the right statistical method from the start. In this article, I'll argue that sequential testing is the only sane way to run A/B tests in the real world, and I'll show you why fixed-horizon tests, the old standard, are a dangerous illusion.
Why Peeking Breaks Your Test
Here's the thing about classic, fixed-horizon statistics: they assume you'll collect a predetermined sample size, then analyze it once. The p-value you get at the end is valid only if you didn't peek at intermediate results. But in practice, everyone peeks. It's human nature. And as Optimizely points out, checking results daily and stopping as soon as significance appears can inflate the false-positive rate from a nominal 5% to as high as 30%. That's a six-fold increase in the chance you'll ship a change that actually does nothing.
Let that sink in. You think you're being data-driven, but you're actually being fooled by randomness. The more you peek, the worse it gets. Evan Miller quantified this: if you peek at an ongoing experiment ten times, what you think is 1% significance is actually just 5% significance. And in the worst case—stopping as soon as 5% significance appears, checking after every observation—the actual false-positive rate can reach 26.1%. That's not an edge case; that's a disaster.
The conventional advice is to pre-commit to a sample size and not peek at all. But that's easier said than done, especially in a fast-moving company. And even if you do resist, fixed-horizon tests have another problem: they often take longer than necessary.
The Hidden Cost of Fixed-Horizon Tests
Before you run any test, you're supposed to calculate a sample size based on your baseline conversion rate and the minimum effect you want to detect. The standard approach, as described by Evan Miller, uses a rule-of-thumb formula: n = 16 times (sigma-squared divided by delta-squared), where delta is the minimum effect and sigma-squared is the variance. For a conversion rate, that's p(1-p). You also set your statistical power, usually at 80%, and your significance level, usually 0.05, as Optimizely notes.
But here's the catch: that sample size is a commitment. You must collect that many visitors no matter what. If the true effect is huge, you still wait. If the effect is zero, you still wait. Fixed-horizon tests are rigid. They don't adapt to the data.
In contrast, sequential testing lets you stop early if the evidence is clear. Evan Miller describes a simple sequential procedure: pick a sample size N, assign subjects 50/50, and stop when the treatment-minus-control success count reaches 2 times the square root of N (declaring a winner) or the total success count reaches N (declaring no winner). This procedure can reduce the number of observations needed by 50% or more in some cases, and it works extremely well with low conversion rates.
That's a huge practical advantage. If you're testing a new headline, you might be able to get a definitive answer in days instead of weeks. And because the thresholds are designed for continuous monitoring, you can peek as often as you like without worrying about false positives.
What Sequential Testing Actually Is
Sequential testing isn't a single technique; it's a family of methods that allow valid inference at any stopping time. The most famous is the Sequential Probability Ratio Test (SPRT), developed during World War II. But modern implementations, like the one in Optimizely's Stats Engine, go further. They use always-valid p-values and confidence intervals, which let you take advantage of data as it arrives while maintaining statistical validity whenever you decide to stop.
The key idea is that the thresholds for significance grow as you collect more data, so that the overall false-positive rate stays controlled no matter how often you peek. This is fundamentally different from fixed-horizon tests, where the threshold is fixed at, say, p
Netflix uses sequential canary tests to detect regressions in streaming quality before they affect all users. As described in a 2022 paper, these tests compare the entire distribution of metrics like PlayDelay—the time for a title to start once play is pressed—rather than just the mean. This catches subtle performance regressions that a mean-only test would miss, while strictly controlling the false-detection probability.
So sequential testing isn't just about avoiding false positives; it's about making better decisions with the data you have, faster.
But Wait, There's a Trade-off
If sequential testing is so great, why isn't everyone using it? The main reason is inertia. Most people learned classic hypothesis testing in school, and tools like Excel and many legacy platforms default to fixed-horizon methods. There's also a misconception that sequential testing is more complex or requires special software.
But the math is actually simpler than you might think. Evan Miller's procedure is just a few lines of code. And modern platforms like Optimizely have built sequential testing into their Stats Engine, which has been powering results for all customers since January 21, 2015. For experiments over 50,000 visitors, it can call a winner or loser up to 2.5 times as fast as fixed-horizon statistics.
There is a cost, though: sequential tests might require a slightly larger sample size to detect a small effect if you don't stop early. But that's a conservative trade-off. In practice, the ability to stop early and the validity under peeking more than compensate.
For example, let's say you're testing a new button color. Your baseline conversion rate is 5%, and you want to detect a 10% relative lift (that's a 0.5 percentage point absolute increase). Using Evan Miller's rule of thumb, you'd need roughly 16 * (0.05*0.95) / (0.005)^2 ≈ 30,400 visitors per variation. That's a big test. With a fixed-horizon design, you'd have to wait for all 60,800 visitors. But with sequential testing, if the true effect is larger than 10%, you might stop at half that. And if the effect is zero, you'll know not to ship, but you'll have spent more time than necessary—that's the price of rigor.
There's also the issue of multiple metrics. If you're testing many different goals or variations, you inflate your false-positive rate even with sequential testing. Optimizely found that re-analyzing 48,000 historical experiments, roughly 20% fewer variations had a false discovery rate below 0.1 than with a false positive rate at the same level. That's the multiple-comparisons cost, and it applies whether you peek or not. So sequential testing isn't a magic bullet; you still need to pre-specify your primary metric and avoid cherry-picking.
The Verdict: Make the Switch
I've made my case, but let me be clear about what I'm recommending. If you're running A/B tests and you care about making correct decisions, you should switch to sequential testing. Not just for new tests, but for your entire testing program. Stop using fixed-horizon tests. They're a trap.
The evidence is overwhelming. Peeking is unavoidable, and fixed-horizon tests can't handle it. Sequential testing, on the other hand, is designed for the way we actually work: we look at data as it comes in, and we want to make decisions quickly. It's not just about avoiding false positives; it's about being more efficient and more confident in your results.
If you're using a platform that supports sequential testing, such as Optimizely, enable it. If you're running your own tests, implement Evan Miller's simple sequential procedure. It's free, it's easy, and it will save you from embarrassing mistakes.
One caveat: sequential testing isn't a license to ignore other statistical issues. You still need to run A/A tests to validate your setup, as Optimizely recommends doing quarterly. If you see a significant difference between two identical versions, something is wrong with your implementation. And you still need to watch for Sample Ratio Mismatch, which KDD 2019 describes as a symptom of data quality issues that can lead to shipping bad changes. But those are separate problems. For the core question of whether your variation beats the control, sequential testing is the only method I trust.
So the next time you're tempted to peek at your dashboard, go ahead. With sequential testing, you can peek as much as you want. Just remember to let the test run its course—or better yet, let it stop when it's ready.
Sources
- Optimizely - https://www.optimizely.com/optimization-glossary/ab-testing/
- Evan Miller - https://www.evanmiller.org/how-not-to-run-an-ab-test.html
- Evan Miller - https://www.evanmiller.org/sequential-ab-testing.html
- Optimizely Stats Engine - https://www.optimizely.com/insights/blog/statistics-for-the-internet-age-the-story-behind-optimizelys-new-stats-engine/
- arXiv - Always Valid Inference - https://arxiv.org/abs/1512.04922
- arXiv - Netflix canary testing - https://arxiv.org/abs/2205.14762
Comments (0)
Please sign in to post a comment.
Don't have an account? Create one
No comments yet. Be the first to comment!