Stop Feeling Guilty About Peeking: Why Your A/B Test Tool Is the Real Problem

If you check your A/B test ten times, that 1% significance you're seeing is actually 5% (Evan Miller). That's the kind of math that should make you sit up straight. But here's the thing: the problem isn't your peeking habit. It's the tool you're using. Fixed-horizon statistics were designed for a world where you look at data once, at the end. That's not how any of us actually work.

I remember a test we ran last fall. Three days in, the numbers looked amazing. My PM was already drafting the launch email. But I knew the stats police would come knocking if we stopped early. So we waited, and peeked anyway, and felt guilty. But the guilt was misplaced. The real culprit was the method, not our curiosity.

The good news? There are tools that let you monitor continuously without lying to you. Sequential testing is the one you need. I'll walk you through why peeking is a problem, what sequential testing does differently, and how to make the switch without tearing your hair out.

Why does peeking inflate false positives?

Peeking means looking at results before the test is done and letting that influence whether you stop. Each look is a chance to see a false positive—a result that looks real but is just noise. If you check daily and stop as soon as something is significant, you can push your false-positive rate from 5% to as high as 30% (Optimizely). In the worst case, stopping at the first hint of significance, it's 26.1% (Evan Miller).

Here's the mental model: every peek is a separate test. More tests, more chances for a fluke. It's the same reason you don't test 20 metrics and only report the winners. More looks, more lies.

What is sequential testing, and how does it solve this?

Sequential testing is a way to stop early if the treatment is clearly winning, without inflating false positives. It works by adjusting the significance thresholds based on how many times you've looked. Instead of fixed-horizon methods, you get "always-valid" p-values that hold no matter when you stop (arXiv - Always Valid Inference).

Optimizely's Stats Engine is one example. It's a sequential testing system that gives you "results that are always valid any time you peek" (Optimizely Stats Engine). They built it with Stanford statisticians and rolled it out to all customers on January 21, 2015. For experiments with over 50,000 visitors, it can call a winner or loser up to 2.5 times as fast as fixed-horizon stats (Optimizely Stats Engine).

If you want to roll your own, Evan Miller has a simple procedure: pick a sample size N, assign subjects randomly 50/50, and stop when the treatment-minus-control success count hits 2 times the square root of N (winner) or total successes hit N (no winner). In some cases, this can cut the needed observations by half or more, and it handles low conversion rates well (Evan Miller - Simple Sequential A/B Testing).

Isn't Bayesian testing just as good?

Bayesian testing is a popular alternative. It gives you probabilities like "there's a 92% chance B is better" and allows continuous monitoring without inflating false positives (Optimizely). VWO says the Bayesian approach can get actionable results nearly 50% faster (VWO).

But here's the catch: Bayesian methods require a prior—a guess about the likely effect size. That prior can sway your results, and if you're not careful, you can fool yourself. Sequential testing doesn't need a prior and gives you guarantees on false positives. It's more robust in practice, especially when you're running many experiments and don't have time to set priors for each one.

I've seen teams get burned by Bayesian analysis. A friend of mine once ran a test where the prior was too optimistic, and they ended up shipping a change that actually hurt metrics. Sequential testing would have kept them honest.

How much time can sequential testing actually save?

A lot. When Optimizely re-ran 48,000 historical experiments, fixed-horizon stats declared a winner or loser in 36% of tests, while sequential Stats Engine did it in 22%—that's 39% fewer conclusive results (Optimizely Stats Engine). Sounds like a downside, but it means sequential is more conservative—it doesn't declare false winners. And when there's a real effect, it finds it faster. For a typical e-commerce test with a 2% baseline conversion rate and a 0.5% minimum detectable effect, a fixed-horizon test might need 100,000 visitors per variation. With sequential testing, you might stop at 50,000 if the effect is real (Evan Miller).

In one of our own tests, we were testing a new checkout button. The sequential test called a winner after 61,000 visitors—the fixed-horizon plan would have needed 120,000. That saved us a week and a half of waiting.

The bottom line: sequential testing can cut test duration by half or more without sacrificing accuracy. That means more experiments per quarter, faster learning, less sitting on your hands.

What about sample size? Do I still need to calculate it?

Yes, but you don't have to commit to it as rigidly. With fixed-horizon testing, you need to pre-commit to a sample size based on baseline conversion rate, minimum detectable effect, significance level (usually 0.05), and power (usually 0.8). A rule-of-thumb is n = 16 * (sigma^2 / delta^2), where sigma is the standard deviation and delta is the minimum effect (Evan Miller). For a binomial conversion rate, variance is p(1-p), so if your baseline is 2%, sigma^2 is about 0.0196.

With sequential testing, you can start with a rough sample size but you're not locked in. If the effect is large, you'll stop early. If it's small, you'll keep going. Methods like the Sequential Probability Ratio Test (SPRT) can reduce needed sample sizes and save resources (Optimizely sample size guide). So you still need to know the effect size you care about, but you don't have to be as precise.

One thing I always tell people: don't set your minimum detectable effect too small. If you're running a test on a button color, a 0.1% effect might be real but not worth the time. Aim for something that actually matters to your business.

Bottom line

The single best move you can make is to switch from fixed-horizon tools to sequential testing tools. If you're using Optimizely, enable their Stats Engine. If you're rolling your own, implement Evan Miller's simple sequential procedure. Stop feeling guilty about peeking—use tools that let you peek without lying. Your false positives will drop, your test velocity will increase, and you'll sleep better at night.

I've been there, and I know the switch feels intimidating. But once you see results coming in faster and with more confidence, you'll wonder why you waited so long.

Sources

• Optimizely - https://www.optimizely.com/optimization-glossary/ab-testing/
• Optimizely Stats Engine - https://www.optimizely.com/insights/blog/statistics-for-the-internet-age-the-story-behind-optimizelys-new-stats-engine/
• Evan Miller - How Not to Run an A/B Test - https://www.evanmiller.org/how-not-to-run-an-ab-test.html
• Evan Miller - Simple Sequential A/B Testing - https://www.evanmiller.org/sequential-ab-testing.html
• arXiv - Always Valid Inference - https://arxiv.org/abs/1512.04922