Why Peeking at Your A/B Test Is Sabotaging Your Results

Peeking Is the Silent A/B Test Killer

Let me tell you something that took me way too long to learn: checking your A/B test daily and stopping the moment it hits 95% significance is a recipe for disaster. I've done it. I've shipped changes that did absolutely nothing because I couldn't resist peeking. The numbers are brutal. Peeking and stopping early can push your false-positive rate from the intended 5% to as high as 30% (Optimizely). That means you're celebrating wins that are pure noise. The fix isn't more willpower; it's picking the right statistical method before you even start. Let's compare the two main contenders: fixed-horizon testing and sequential testing. Spoiler: sequential wins for most teams, but there are times when the old-school method is still worth it.

The Contenders: Fixed-Horizon vs Sequential

Fixed-horizon testing is the classic. You decide your sample size upfront, run the test for that exact number of visitors, and only then look at the results. The math is straightforward: you plug in your significance level (usually 0.05) and power (often 80%), and you get a sample size (Optimizely). It's in every A/B testing guide, and it's what you learned in Stats 101. Sequential testing, on the other hand, lets you peek as often as you like. The catch is that it adjusts the significance thresholds on the fly, so your false-positive rate stays controlled (Evan Miller). Methods like the Sequential Probability Ratio Test (SPRT) or Optimizely's Stats Engine are built for this. They don't force you to guess a minimum detectable effect in advance, and you get valid results whenever you look (Optimizely Stats Engine).

False-Positive Rate: The Make-or-Break Criterion

This is where it all falls apart for fixed-horizon. If you peek every day and stop at the first sign of significance, you're fooling yourself. Evan Miller's analysis shows that if you peek ten times, what you think is 1% significance is actually just 5% significance. And in the worst case—stopping at the first hint of 5%—your actual false-positive rate can hit 26.1% (Evan Miller). That's over five times the 5% you intended. Sequential testing completely sidesteps this. Methods like always-valid p-values let you monitor continuously without inflating false positives (Johari et al.). Optimizely's Stats Engine was built for this exact problem. When they re-ran 48,000 historical experiments, it declared a winner or loser in only 22% of tests, compared to 36% with fixed-horizon statistics (Optimizely Stats Engine). That's 39% fewer false conclusions. If your team can't resist peeking (and let's be honest, most can't), sequential is the only sane choice.

Speed to Decision: When Time Is Money

Speed matters—especially when you're iterating fast. Sequential testing can be dramatically faster. In some cases, it cuts the number of observations needed by 50% or more, particularly with low conversion rates (Evan Miller). Optimizely's Stats Engine, for experiments with over 50,000 visitors, can call a winner or loser up to 2.5 times faster than fixed-horizon statistics (Optimizely Stats Engine). That's not a marginal gain. Fixed-horizon testing forces you to wait the full duration, even if the effect is obvious early on. You're stuck with your initial sample size estimate, which might be wrong if your baseline conversion rate or effect size assumptions were off. Sequential testing adapts. It stops early if there's a clear winner, and it can keep going if results are borderline. For fast-moving teams, that agility is gold.

Complexity and Ease of Implementation: The Catch

Here's where fixed-horizon has an edge. It's dead simple. You calculate your sample size based on baseline conversion rate and minimum detectable effect (Optimizely sample size guide). You set a calendar reminder to check results after that many visitors. No fancy math. Sequential testing, on the other hand, requires more sophisticated tools. You can't just use a standard t-test with a p-value threshold. You need dedicated software like Optimizely's Stats Engine or a custom implementation of an SPRT. That complexity can be a barrier for small teams without strong data science support. But the good news is that many modern A/B testing platforms, including Optimizely, have built-in sequential methods (Optimizely Stats Engine). If you're using a platform that supports it, the complexity is handled for you. If you're rolling your own, it's harder. But the cost of complexity is trivial compared to the cost of shipping a false positive.

Flexibility with Multiple Metrics and Goals

Real-world tests often have multiple success metrics. You might be tracking conversion rate, revenue, and user engagement simultaneously. Fixed-horizon testing struggles here because multiple comparisons inflate your false-positive rate. Optimizely found that when they re-analyzed 48,000 experiments, about 20% fewer variations were declared winners with a false discovery rate below 0.1 than with a false positive rate at the same level (Optimizely Stats Engine). That's the cost of testing many metrics. Sequential methods, especially those that control the false discovery rate, are better equipped to handle this. Netflix uses sequential canary tests that examine the entire distribution of a metric, not just the mean, so they can catch regressions in tail performance without increasing false detections (Lindon et al.). If your tests are simple—one primary metric, clear winner—fixed-horizon can work. But if you're like most teams, with multiple goals and stakeholders, sequential is the safer bet.

Who Wins? It Depends on Your Situation

My recommendation: default to sequential testing. The peeking problem is so pervasive that I'd argue it's irresponsible to run a fixed-horizon test without strict discipline. If you have a team that can commit to a pre-decided sample size and never look at the data until the end, fixed-horizon is fine—it's simpler and cheaper to implement. But that's rare. I've seen too many teams peek, rationalize, and ship garbage. Sequential testing is the pragmatic choice for most businesses. It's built into major platforms, it's faster, and it protects you from yourself. Here's a quick comparison table:

Quick tip: If you're using a platform that doesn't support sequential testing, the best defense against peeking is to set a sample size in advance and hide the results dashboard until the test is done (Evan Miller).

The Takeaway

Stop lying to yourself. The traditional fixed-horizon A/B test is a relic of a pre-digital world where you couldn't peek at data every hour. In today's fast-paced environment, sequential testing is the only way to get reliable results without locking yourself in a dark room. It's faster, it's safer, and it's available in most modern tools. If you're not using it, you're leaving money on the table and shipping changes that don't work. Make the switch.

Sources

• Optimizely - https://www.optimizely.com/optimization-glossary/ab-testing/
• Optimizely Stats Engine - https://www.optimizely.com/insights/blog/statistics-for-the-internet-age-the-story-behind-optimizelys-new-stats-engine/
• Evan Miller - https://www.evanmiller.org/how-not-to-run-an-ab-test.html
• Evan Miller - https://www.evanmiller.org/sequential-ab-testing.html
• arXiv - https://arxiv.org/abs/1512.04922
• arXiv - https://arxiv.org/abs/2205.14762