Stop Trusting Your A/B Test Results: The Hidden Peeking Problem

Imagine you're two days into a week-long A/B test. You glance at the dashboard and see that Variant B is crushing it—95% significance! You stop the test, ship B, and celebrate. But here's the kicker: that significance is a lie. You've just fallen for the most common trap in A/B testing: peeking.

If you've ever felt that itch to check results early, you're not alone. But the data is clear: peeking can wreck your test's validity. The solution isn't to ignore your results—it's to design your test so you can peek safely. Let's bust some myths.

What's the big deal about peeking, really?

When you peek at your A/B test and stop as soon as you see significance, you're inflating your false-positive rate. In fact, checking results daily and stopping as soon as significance appears can push your false-positive rate from a nominal 5% to as high as 30% (Optimizely). That means up to 1 in 3 of your "winners" could be false alarms. And if you peek ten times, what you think is 1% significance is actually just 5% (Evan Miller). The more you peek, the more you're fooling yourself.

Can't I just use a higher significance level to protect myself?

No, that's a band-aid, not a fix. Raising your significance threshold (say, to 99%) might reduce false positives, but it also reduces power—your ability to detect a real effect. You'd need a much larger sample size to maintain power. The real problem is the repeated testing itself. Every time you look at the data and consider stopping, you're implicitly running another hypothesis test, and that always increases false-positive rates (Evan Miller). The only way to avoid this is to pre-commit to a fixed sample size or use a sequential testing method.

So, should I just set a sample size and never look until it's done?

That's one option, and it's the classic approach. Pre-committing to a sample size—based on your baseline conversion rate and minimum detectable effect—is the gold standard for fixed-horizon tests (Optimizely sample size guide). But it has a downside: you might run longer than necessary, wasting time and resources. And if you're tempted to peek, you're back to square one. The better option for many teams is sequential testing, which lets you peek safely.

What is sequential testing, and why should I care?

Sequential testing allows you to stop early if the treatment is a clear winner, without inflating false positives. It's like having a safety net for your impatience. Evan Miller's simple sequential procedure is a great starting point: pick a sample size N, assign users 50/50, and stop when the treatment-minus-control success count reaches 2 times the square root of N (declaring a winner) or the total success count reaches N (declaring no winner) (Evan Miller). This approach can reduce the number of observations needed by 50% or more in some cases, and it works especially well with low conversion rates (Evan Miller).

Optimizely's Stats Engine uses sequential testing too, and it can declare a winner or loser up to 2.5 times faster than fixed-horizon statistics for experiments over 50,000 visitors (Optimizely Stats Engine). The beauty is that you get always-valid p-values and confidence intervals, so you can peek any time and trust the results (arXiv - Always Valid Inference).

But what about A/A tests? Are those just a waste of time?

Absolutely not. A/A tests—where you test two identical versions—are a crucial sanity check for your testing setup. If your A/A test shows a statistically significant difference (greater than 95% significance), something is wrong with your implementation (Optimizely A/A testing glossary). I recommend running an A/A test quarterly to catch any data quality issues before they poison your real experiments. Think of it as a calibration check for your experimentation platform.

What about sample ratio mismatch? Is that a myth?

No, Sample Ratio Mismatch (SRM) is a real red flag. It's when the observed ratio of users in your control vs. variation differs from the expected 50/50 split. SRM is like a fever—it's a symptom of underlying issues like data loss, tracking bugs, or even a bug in your randomization (KDD 2019). Ignoring SRM can lead you to ship a bad variation or kill a good one. Always check for SRM before you trust any result.

So, what's the bottom line for designing reliable A/B tests?

Stop pretending you have willpower. Design your test for the reality that you'll peek. Use sequential testing methods, run A/A checks, and watch for SRM. The days of the fixed-horizon, don't-peek-or-else approach are over—embrace the safety of sequential analysis.

Quick tip: If you're using a classic fixed-sample test, set your sample size in advance and commit to it. If you can't resist peeking, switch to a sequential method.

Sources

• Optimizely - A/B testing: https://www.optimizely.com/optimization-glossary/ab-testing/
• Optimizely Stats Engine: https://www.optimizely.com/insights/blog/statistics-for-the-internet-age-the-story-behind-optimizelys-new-stats-engine/
• Evan Miller - How Not to Run an A/B Test: https://www.evanmiller.org/how-not-to-run-an-ab-test.html
• Optimizely sample size guide: https://www.optimizely.com/insights/blog/how-to-calculate-sample-size-of-ab-tests/
• Optimizely A/A testing glossary: https://www.optimizely.com/optimization-glossary/aa-testing/
• KDD 2019 - Diagnosing Sample Ratio Mismatch: https://www.kdd.org/kdd2019/accepted-papers/view/diagnosing-sample-ratio-mismatch-in-online-controlled-experiments-a-taxonom