Stop Letting Your A/B Test Tool Peek for You

Your Tool Is Lying to You

I'm going to say something that might get me unfriended by every optimization platform out there: the "peeking" feature in your A/B testing tool is the enemy of good experimentation. You've been told that checking results early is fine, that the tool will handle it, that you can stop as soon as it says "significant." That's nonsense. The data is clear: peeking at your test results and stopping early can inflate your false-positive rate from a nominal 5% to as high as 30% (Optimizely). That's not a minor tweak; that's a disaster. You're shipping changes that do nothing, or worse, hurt your conversion rate, all because you couldn't wait for the test to finish.

Imagine You're a Growth Marketer at a SaaS Startup

Picture this: you're running a landing page test. Your baseline conversion rate is 2%, and you want to detect a 20% relative lift. You set up the test in your favorite tool, and it tells you to run it for two weeks. But on day three, you see that the variation is crushing it—it's already at 2.6% conversion, and the tool's p-value is 0.04. Tempting, right? You're tempted to call it a winner and move on. But here's the kicker: that result is not reliable. The tool is using fixed-horizon statistics, which assume you'll look at the data only once, at the end. But you peeked, and you stopped early. You've just committed the classic peeking sin.

The Math Behind the Madness

Why is peeking so dangerous? Because it's a multiple-comparisons problem. Every time you look at the data and consider stopping, you're effectively running another hypothesis test. Evan Miller, who has written extensively on this, points out that if you peek ten times, what you think is 1% significance is actually just 5% significance (Evan Miller). In the worst case, if you stop as soon as you see 5% significance and you check after every observation, the actual false-positive rate can reach 26.1% (Evan Miller). That means one in four of your "wins" is a fluke. And here's the thing: your tool isn't protecting you from this. It's enabling it.

Why Your Tool's "Guarantee" Is a Myth

Most A/B testing tools use frequentist statistics. They give you a p-value and a confidence interval, and they tell you to wait until you reach significance. But they don't stop you from peeking. They even make it easy by showing you real-time results. They might have a "sequential testing" option, but it's often buried in settings, and the default is still the classic approach. The result? You're left to your own devices, and you'll peek. We all do. It's human nature. The solution isn't to train yourself to resist; it's to use a tool that makes peeking safe.

Sequential Testing: The Antidote

Sequential testing is designed for this. It allows you to look at the data as often as you want, without inflating your false-positive rate. Optimizely's Stats Engine, for example, is built on sequential testing and can call a winner or loser up to 2.5 times as fast as fixed-horizon statistics for experiments over 50,000 visitors (Optimizely). Evan Miller offers a simple sequential procedure that you can implement yourself: pick a sample size N, assign subjects randomly 50/50, and stop when the treatment-minus-control success count reaches 2 times the square root of N (declaring a winner) or the total success count reaches N (declaring no winner). In some cases, this can reduce the number of observations needed by 50% or more (Evan Miller). That's a game-changer for a startup with limited traffic.

The Reality Check: What Sequential Testing Gives You

But here's the catch: sequential testing doesn't give you a fixed sample size upfront. It gives you always-valid p-values and confidence intervals, so you can check at any time and make a decision that's statistically valid (arXiv). That means you can't pre-calculate your test duration as neatly. You have to be comfortable with a bit of uncertainty. However, the trade-off is worth it. When Optimizely re-ran 48,000 historical experiments, fixed-horizon statistics declared a winner or loser in 36% of tests, while the sequential Stats Engine declared winners or losers in only 22%, uncovering 39% fewer conclusive results (Optimizely). That's not because sequential testing is less powerful; it's because it's more honest. It avoids false positives, and it saves you from shipping bad changes.

What I'd Actually Do

My recommendation is simple: stop using the default fixed-horizon setting in your A/B testing tool. Switch to sequential testing, whether that's Optimizely's Stats Engine, Evan Miller's simple procedure, or any tool that supports always-valid inference. If your tool doesn't support it, consider switching tools. And if you can't switch, at least commit to a fixed sample size before you start, and don't peek. It's painful, but it's better than shipping a dud. Also, run A/A tests quarterly to make sure your implementation is sound (Optimizely). If an A/A test shows a significant difference, your setup is broken. And watch for sample ratio mismatch, which can indicate data quality issues that invalidate your results (KDD). Your optimization tool is a means to an end, not a crystal ball. Use it wisely.

Sources

• Optimizely - https://www.optimizely.com/optimization-glossary/ab-testing/
• Optimizely Stats Engine - https://www.optimizely.com/insights/blog/statistics-for-the-internet-age-the-story-behind-optimizelys-new-stats-engine/
• Evan Miller - How Not to Run an A/B Test - https://www.evanmiller.org/how-not-to-run-an-ab-test.html
• Evan Miller - Simple Sequential A/B Testing - https://www.evanmiller.org/sequential-ab-testing.html
• arXiv - Always Valid Inference (Johari et al.) - https://arxiv.org/abs/1512.04922
• KDD 2019 - Diagnosing Sample Ratio Mismatch - https://www.kdd.org/kdd2019/accepted-papers/view/diagnosing-sample-ratio-mismatch-in-online-controlled-experiments-a-taxonom