A/B Testing: Why Peeking at Your Results Is Sabotaging Your Data

Peeking at Your A/B Test: The Habit You Need to Break

You've launched an experiment. Two days later, you're refreshing the dashboard like it's a social media feed. The temptation to look is real. But here's the thing: peeking isn't just a bad habit—it's a data integrity killer. If you stop your test the moment you see a significant p-value, you're not being impatient; you're actively corrupting your results. Checking daily and stopping early can inflate your false-positive rate from 5% to as high as 30% (Optimizely). That's not a minor error; that's a broken compass.

So what's a practical marketer to do? The solution isn't to lock yourself in a room until the test ends. It's to switch from the classic fixed-horizon approach to a sequential testing method that lets you peek safely. In this article, I'll compare the two dominant approaches: the fixed-horizon method (what most of us learned) and the sequential method (what forward-thinking teams are adopting). I'll break them down on three concrete criteria: flexibility, speed, and reliability. And I'll tell you which one wins—and when sticking with the old way might still make sense.

Option 1: Fixed-Horizon Method (The Old Guard)

This is the classic A/B testing playbook. You pick a sample size before you start, based on your baseline conversion rate, minimum detectable effect (MDE), and desired power—typically 80% (Optimizely). You run the test until you hit that sample size, then you analyze with a p-value threshold of 0.05. It's clean, simple, and the default in most tools.

But here's the catch: this method assumes you won't look at the data until the end. The moment you peek, you're violating the statistical assumptions. Evan Miller, in his classic rant "How Not to Run an A/B Test," shows that if you peek at an ongoing experiment ten times, what you think is 1% significance is actually 5% significance (Evan Miller). And if you stop as soon as 5% significance appears, with a test run after every observation, your actual false-positive rate can hit 26.1% (Evan Miller). That's not a hypothetical edge case; that's a real-world failure mode.

Who is this for? Teams with ironclad discipline, a pre-committed sample size, and no temptation to check early. It's also the default when using basic A/B testing tools that don't offer sequential methods. But let's be honest: most of us lack that discipline. We're human. We see a 2% lift on day three and start planning the celebration.

Option 2: Sequential Method (The New Sheriff in Town)

Sequential testing flips the script. Instead of fixing your sample size in advance, you use a procedure that lets you stop early if the result is already convincing, while still maintaining valid statistical guarantees. Evan Miller's simple sequential procedure is a great example: you choose a sample size N, assign subjects randomly 50/50, and stop when the treatment-minus-control success count reaches 2 times the square root of N (declaring a winner) or the total success count reaches N (declaring no winner) (Evan Miller). The beauty is that this method allows you to peek as often as you like, and your false-positive rate stays under control.

This approach is gaining traction. Optimizely's Stats Engine, built with Stanford statisticians, uses sequential testing to let users see results that are always valid any time they peek, without needing to guess a minimum detectable effect in advance (Optimizely). The benefits are real: in some circumstances, sequential procedures can reduce the number of observations needed for a successful experiment by 50% or more, especially with low conversion rates (Evan Miller). Netflix even uses sequential canary tests to detect performance regressions like PlayDelay, testing across the entire distribution rather than just the mean (arXiv).

Who is this for? Teams that want to move fast without waiting for a fixed sample size, especially when traffic is limited or the experiment is risky. It's also ideal for teams that struggle with the discipline of not peeking—because you can peek all you want.

Head-to-Head: Three Criteria That Decide the Winner

Let's pit these two methods against each other on three practical criteria: flexibility, speed, and reliability.

As the table shows, sequential testing wins on flexibility, speed, and reliability. But there's a catch: sequential methods can sometimes require a larger maximum sample size than fixed-horizon, because they're designed to handle early stops. And if you're running a test that truly cannot stop early—say, you need to observe a full week of user behavior—sequential methods might not give you a big advantage.

Still, the evidence is compelling. Optimizely re-ran 48,000 historical experiments and found that fixed-horizon statistics declared a winner or loser in 36% of tests, while the sequential Stats Engine declared winners or losers in only 22%—uncovering 39% fewer conclusive results (Optimizely). That might sound like a drawback, but it means the fixed-horizon method was declaring false positives at a higher rate. Sequential testing is more conservative, but it's more honest.

What I'd Actually Do

Here's my recommendation: if your A/B testing tool supports sequential methods (like Optimizely's Stats Engine or Evan Miller's simple sequential procedure), use them. Period. The ability to peek without penalty is a game-changer for teams that want to iterate quickly. It's not just about avoiding false positives; it's about making better decisions faster.

• Adopt sequential testing for most experiments, especially when you're testing low-stakes UI changes or content tweaks.
• Set a maximum sample size based on your traffic and MDE, but plan to stop early if the evidence is overwhelming.
• Run A/A calibration tests quarterly to ensure your setup is sound (Optimizely).

But don't completely abandon fixed-horizon. If you're running a high-stakes test where the cost of a false positive is enormous—like a pricing change or a major feature launch—the conservative nature of sequential methods might be worth the extra time. And if you're using a tool that doesn't support sequential testing, you can still implement Evan Miller's procedure manually. The key is to stop treating your A/B test like a black box that you peek into at your peril. Embrace the method that lets you look, and you'll sleep better at night.

Sources

• Optimizely - https://www.optimizely.com/optimization-glossary/ab-testing/
• Evan Miller - How Not to Run an A/B Test - https://www.evanmiller.org/how-not-to-run-an-ab-test.html
• Optimizely - https://www.optimizely.com/insights/blog/statistics-for-the-internet-age-the-story-behind-optimizelys-new-stats-engine/
• Evan Miller - Simple Sequential A/B Testing - https://www.evanmiller.org/sequential-ab-testing.html
• arXiv - Always Valid Inference - https://arxiv.org/abs/1512.04922