Peeking vs. Sequential Testing: Why Your A/B Test Lies

You're checking your A/B test daily, and on day three, the variation hits 95% significance. You stop it and ship. Sound familiar? Here's the problem: that 'significant' result is likely a lie. Peeking at your data and stopping early can inflate your false-positive rate from 5% to as high as 30% (Optimizely). That's a 6x increase in the chance you're shipping a change that does nothing. So, what do you do? Stop peeking and pick a method that respects the math. In this head-to-head, I compare the two main statistical approaches: fixed-horizon testing (the traditional way) and sequential testing (the modern fix). I'll tell you which wins and when.

What Are Your Options?

You've got two practical choices when you design an A/B test.

Option 1: Fixed-horizon testing. This is the classic frequentist approach. You decide your sample size upfront, run the test until you hit that number, and then look at the p-value. If p

Option 2: Sequential testing. This is the modern approach, popularized by tools like Optimizely's Stats Engine and methods like the Sequential Probability Ratio Test (SPRT). You set a stopping rule that adjusts for multiple looks. You can peek at your results as often as you want, and your error rates stay controlled. It's designed for the real world where nobody wants to wait.

There's also a third option: Bayesian testing. It outputs probabilities like 'there is a 92% probability that Variant B is better' and allows continuous monitoring without inflating false positives (Optimizely). But Bayesian methods can be more complex to implement, and for many teams, frequentist sequential testing is the easier upgrade.

Criteria 1: False-Positive Control

This is the non-negotiable. A false positive means you ship a change that actually hurts you or does nothing—wasted effort, wasted money.

Fixed-horizon testing, when done correctly, controls the false-positive rate at your chosen alpha (usually 5%). But the moment you peek, that control breaks. Repeated significance testing always increases false positives. If you peek ten times, what you think is 1% significance is actually just 5% (Evan Miller). In a worst-case scenario—stopping as soon as 5% appears—your actual false-positive rate can hit 26.1% (Evan Miller). That's a disaster.

Sequential testing, on the other hand, is built for peeking. Always-valid p-values let you look at your data as often as you want and still get valid inference whenever you stop (arXiv - Always Valid Inference). Optimizely's Stats Engine, which uses sequential testing, found 39% fewer conclusive results when re-running 48,000 historical experiments compared to fixed-horizon statistics (Optimizely Stats Engine). That's not because it's less powerful—it's because it's not fooled by random noise.

Winner: Sequential testing. It keeps your false positives at the level you expect, even if you peek.

Criteria 2: Speed and Efficiency

Time is money. You don't want to run a test for six weeks if you can get a reliable answer in three.

Fixed-horizon tests require you to pre-commit to a sample size based on your desired power (usually 80%) and minimum detectable effect (MDE). If your MDE is small, you need a huge sample, which means a long test. And you can't stop early even if the effect is huge, because that would be peeking.

Sequential testing often gets you to a decision faster. Evan Miller's simple sequential procedure can reduce the number of observations required by 50% or more in some cases (Evan Miller - Simple Sequential A/B Testing). Optimizely's Stats Engine can call a winner or loser up to 2.5 times faster than fixed-horizon statistics for experiments over 50,000 visitors (Optimizely Stats Engine). That's a massive advantage when you're iterating quickly.

But there's a trade-off. Sequential tests can sometimes take longer if the true effect is small, because they're waiting for enough evidence. However, in practice, the speed gains on large effects make it a clear win.

Winner: Sequential testing. It's faster on the big wins, which are the ones you want to catch.

Criteria 3: Ease of Use and Interpretation

If your team can't understand the method, they won't use it correctly.

Fixed-horizon testing is straightforward to explain. You set a sample size, run, and check the p-value. The p-value is a standard concept: the probability of seeing data as extreme as what you saw, assuming no effect (Optimizely). But the problem is that it's easy to misuse—people peek because they're eager, and they misinterpret what a p-value means. A p-value of 0.03 doesn't mean a 97% chance the treatment works; it just means the data is surprising under the null (Optimizely).

Sequential testing is a bit more complex to set up. You need to choose a stopping rule or use a tool that handles it for you. But the interpretation is just as simple: if the sequential p-value crosses the threshold, you have a winner. And you can explain it like this: 'We can look at the data as often as we want, and the error rate stays controlled.' That's easier for stakeholders to accept than 'we can't peek because it's naughty.'

Also, sequential methods don't require you to guess an MDE in advance (Optimizely Stats Engine). That's a huge practical advantage because MDE is often a guess anyway.

Winner: Sequential testing. It's slightly more technical to implement, but it's more user-friendly in practice because it fits how people actually behave.

Criteria 4: When Fixed-Horizon Is Your Friend

Don't throw out fixed-horizon testing entirely. There are cases where it's the right choice.

If you have a very small team and you're running a simple test with a large expected effect, a fixed-horizon test can work fine—as long as you have the discipline not to peek. If you're doing a one-off analysis and you can pre-register your sample size, fixed-horizon is simpler and well-understood.

Also, if your organization is already using a tool that only does fixed-horizon, it's better to use it correctly than to switch tools and deal with a learning curve. But the key is to set your sample size upfront and stick to it. Evan Miller's advice: decide on a sample size in advance and don't stop early (Evan Miller - How Not to Run an A/B Test).

However, the reality is that most teams can't resist peeking. If you're human, you'll peek. And if you peek with fixed-horizon, your results are garbage. So unless you have iron discipline, sequential is safer.

The Verdict

Here's my recommendation: If you're running A/B tests on a website or app, use sequential testing. It's the industry's move toward better statistics. Optimizely, Netflix, and others have adopted it for good reason. Netflix uses sequential canary tests to detect performance regressions like PlayDelay, and they control the false-detection probability strictly (arXiv - Netflix canary testing).

But don't just switch methods—also check your data quality. An SRM (sample ratio mismatch) can invalidate your test regardless of method. Ignoring an SRM can make a bad change look good (KDD 2019). So before you trust any result, make sure your test is clean.

If you're a small team with a simple test and you can commit to a fixed sample size, fixed-horizon can still work. But if you want to be efficient and avoid the peeking trap, sequential is the way.

The Bottom Line

Stop peeking with fixed-horizon tests. Switch to sequential testing. It's the single most important thing you can do to improve the reliability of your A/B tests. Your future self—and your revenue—will thank you.

Sources

• Optimizely - https://www.optimizely.com/optimization-glossary/ab-testing/
• Evan Miller - How Not to Run an A/B Test - https://www.evanmiller.org/how-not-to-run-an-ab-test.html
• Optimizely Stats Engine - https://www.optimizely.com/insights/blog/statistics-for-the-internet-age-the-story-behind-optimizelys-new-stats-engine/
• Evan Miller - Simple Sequential A/B Testing - https://www.evanmiller.org/sequential-ab-testing.html
• arXiv - Always Valid Inference - https://arxiv.org/abs/1512.04922
• KDD 2019 - Diagnosing Sample Ratio Mismatch - https://www.kdd.org/kdd2019/accepted-papers/view/diagnosing-sample-ratio-mismatch-in-online-controlled-experiments-a-taxonom