Stop Peeking or Stop Testing: Fixed-Horizon A/B Tests Are a Lie

Why does my A/B test keep lying to me?

You check your dashboard at noon. The variation is up 3%. You check again at 5 PM—still up. By Friday, the p-value dips below 0.05. You call it a win, ship the feature, and move on. But here's the ugly truth: that result might be pure noise. Peeking at your data and stopping the moment significance appears is a statistical sin. It inflates your false-positive rate from the nominal 5% to as high as 26.1% (Evan Miller). That's a one-in-four chance you're shipping a feature that does nothing—or worse, hurts your users.

I'm not here to tell you to stop peeking. I'm here to tell you to stop using fixed-horizon tests entirely. Switch to sequential testing. It's the only honest way to run an experiment when you—like every human—can't resist a glance at the numbers.

The fixed-horizon lie

Classic A/B testing tells you to pick a sample size, run the test, and don't look until the end. That's the theory. In practice, nobody does that. You have a boss asking for results, a roadmap to hit, and a dashboard that's always on. So you peek. And every peek is a repeated significance test, which always increases your false-positive rate (Evan Miller). Check ten times, and your 1% significance is actually 5% (Evan Miller). Worst case—if you stop the moment you see 5% significance—your real false-positive rate is 26.1% (Evan Miller). That's not a rounding error. That's a broken process.

The fixed-horizon approach also forces you to guess a minimum detectable effect and sample size in advance. Get those wrong, and you're either underpowered (missing real effects) or overpaying for a test that takes forever. And even if you follow the rules, you're stuck waiting for the full sample to accrue, even if the result is obvious early.

Sequential testing is the fix

Sequential testing flips the script. Instead of fixing a sample size, you let the data accumulate and check at any time. The math adjusts your thresholds so that your false-positive rate stays controlled no matter how often you peek. This is what Optimizely's Stats Engine does, and it's backed by Stanford statisticians (Optimizely). The always-valid p-values from sequential testing let you make decisions as fast as data comes in, without penalizing you for looking (arXiv).

Here's a concrete example: suppose you run a test on your checkout page. With fixed-horizon stats, you'd need, say, 10,000 visitors per variation to detect a 1% lift. But with a simple sequential procedure, you can stop as soon as the treatment-minus-control success count reaches 2 times the square root of N, or when total successes hit N (Evan Miller). In some cases, this can cut the required observations by 50% or more (Evan Miller). That means faster decisions, less opportunity cost, and fewer false alarms.

But what about the false positives from sequential testing?

You might think: “Sequential testing is just a way to cheat the p-value. Isn't it less rigorous?” No. It's more rigorous. Fixed-horizon tests are only valid if you never peek. Sequential tests are valid even if you peek every hour. The trade-off is that sequential tests may require a slightly larger sample size in the worst case, but they save you from the peeking penalty. And in practice, they often save resources. When Optimizely re-ran 48,000 historical experiments, fixed-horizon stats declared a winner or loser in 36% of tests, while their sequential engine declared in 22% (Optimizely). That's 39% fewer conclusive results—meaning the fixed-horizon tests were calling winners that were likely false positives.

So yes, sequential testing is stricter. That's a feature, not a bug.

How to switch today

You don't need to build your own math. Use a tool that supports sequential testing. Optimizely's Stats Engine has it built in (Optimizely). Evan Miller's sequential procedure is simple enough to implement manually for a quick check (Evan Miller). The key is to stop pre-committing to a fixed sample size and instead let the data guide you.

• Pick a minimum effect you care about, but don't treat it as a hard stop.
• Use sequential p-values that are always valid.
• Run A/A tests quarterly to make sure your setup isn't broken (Optimizely).

Quick tip: If your tool only offers fixed-horizon stats, resist the urge to peek. Set a calendar reminder for the end date and don't open the dashboard until then. But if you can't trust yourself, switch tools.

Bottom line

Fixed-horizon A/B testing is a lie. Peeking turns a 5% false-positive rate into 26.1% (Evan Miller). Sequential testing lets you peek safely and get answers faster. Switch today.

Sources

• Evan Miller - How Not to Run an A/B Test - https://www.evanmiller.org/how-not-to-run-an-ab-test.html
• Evan Miller - Simple Sequential A/B Testing - https://www.evanmiller.org/sequential-ab-testing.html
• Optimizely Stats Engine - https://www.optimizely.com/insights/blog/statistics-for-the-internet-age-the-story-behind-optimizelys-new-stats-engine/
• arXiv - Always Valid Inference - https://arxiv.org/abs/1512.04922
• Optimizely A/A testing glossary - https://www.optimizely.com/optimization-glossary/aa-testing/