Stop Peeking at Your A/B Test: Use Sequential Tools Instead

The Peeking Trap: Why You Can't Trust Your 'Significant' Result

You've been told to calculate a sample size, set a fixed end date, and never peek. But that's not how real optimization works. You have a boss breathing down your neck, a product launch looming, or you're just naturally curious. You peek. And then you stop the test the moment you see 95% significance. Here's the kicker: that 95% is a lie. Checking results daily and stopping as soon as significance appears can inflate the false-positive rate from a nominal 5% to as high as 30% (Optimizely). In a worst-case scenario where you stop the moment 5% significance appears, the actual false-positive rate can reach 26.1% (Evan Miller). That's a one-in-four chance you're shipping a change that does nothing—or worse, hurts your metrics.

The core issue is that traditional fixed-horizon statistics assume you look at the data once, at a pre-determined time. Every time you peek, you're performing a separate hypothesis test, and the more you peek, the more your reported significance levels are off (Evan Miller). If you peek ten times, what you think is 1% significance is actually just 5% significance (Evan Miller). So you're not just being risky—you're being statistically blind.

But there's a better way: sequential testing. Sequential methods like the Sequential Probability Ratio Test (SPRT) let you stop early if a winner is clear, without inflating false positives (Optimizely sample size guide). In some circumstances, they can reduce the number of observations needed by 50% or more, and they work extremely well with low conversion rates (Evan Miller). This isn't just theory—Optimizely's Stats Engine, which uses sequential testing, has been powering results for all Optimizely customers since January 21, 2015, and for experiments over 50,000 visitors it can call a winner or loser up to 2.5 times as fast as fixed-horizon statistics (Optimizely Stats Engine).

How to Run a Sequential A/B Test Without Losing Your Mind

Here's a practical walkthrough for anyone who's tired of waiting weeks for a test to 'finish' and wants to make decisions faster without sacrificing correctness. This is for you if you're running experiments on a website, app, or any digital product, and you've felt the urge to peek at your dashboard mid-test. I'm going to give you a simple, concrete process that uses sequential testing to keep you honest.

First, pick a sequential testing tool or implement a simple procedure. If you're using a platform like Optimizely, look for their Stats Engine—it's built on sequential methods and gives you always-valid p-values and confidence intervals (Optimizely Stats Engine). If you're rolling your own, Evan Miller's simple sequential procedure is a great starting point: choose a sample size N, assign subjects randomly 50/50, and stop when the treatment-minus-control success count reaches 2 times the square root of N (declaring a winner) or the total success count reaches N (declaring no winner) (Evan Miller). This procedure is straightforward and doesn't require complex math.

Second, set your parameters: baseline conversion rate, minimum detectable effect (MDE), and desired power. Most people set power at 80% and significance at 5% (Optimizely). The MDE is the smallest effect you care about—anything smaller is noise (Evan Miller). For example, if your baseline conversion is 5%, and you want to detect a 10% relative lift (i.e., a 0.5 percentage point increase), your MDE is 0.5%. Use Evan Miller's sample size calculator to get a rough N. But remember, sequential testing doesn't require you to guess the MDE in advance—that's one of its advantages (Optimizely Stats Engine). You can start with a reasonable N and let the sequential method adapt.

Third, monitor your test regularly, but trust the sequential boundaries. You can peek as often as you like—that's the whole point. But don't stop just because the p-value dips below 0.05. In a sequential test, the stopping rule is what protects you. Follow the rule you set at the start. If you're using Optimizely's Stats Engine, it will tell you when a result is conclusive, and you can trust that result because it's always valid (Optimizely Stats Engine).

Fourth, run an A/A test first. This is non-negotiable for me. Optimizely recommends running A/A calibration tests quarterly (Optimizely). An A/A test validates your setup: if you see a 'significant' difference between two identical versions, something is broken in your implementation (Optimizely). It's a data quality check that catches issues before they poison your real tests.

What Can Go Wrong: Sample Ratio Mismatch and Novelty Effects

Even with sequential testing, you can't ignore data quality. One common problem is Sample Ratio Mismatch (SRM)—when the observed ratio of visitors in each variation differs from the expected 50/50. SRM is a symptom of a variety of data quality issues, and ignoring it can lead you to ship a bad change as a good one, or vice versa (KDD 2019). Always check for SRM before declaring a winner.

Another trap: novelty and primacy effects. Novelty is the tendency for users to engage with something new just because it's new, while primacy is growing engagement as users learn the new feature (arXiv). These effects can make a new variation look great at first, then fade. If you stop your test too early, you might capture the novelty spike and make a wrong decision. Sequential testing helps because it lets you run longer if needed, but you should still be aware of these effects and consider running tests for at least a week or two to let them settle.

Quick Tip: Don't Stop at Significance

Here's a warning: don't stop your test the moment it reaches 'significance' in a traditional sense. In sequential testing, the stopping rule is your friend. If you're using a tool, let it tell you when to stop. If you're doing it manually, follow Evan Miller's procedure. Trust the process, not your gut.

Also, think about the business context. Longer tests may conflict with business cycles or seasonal changes, and running one large test has an opportunity cost versus running multiple smaller tests (Optimizely sample size guide). So choose your tests wisely.

Takeaway

Peeking is human, but it's also the quickest way to turn a 5% false-positive rate into a 26.1% one. Stop using fixed-horizon statistics for your A/B tests. Switch to sequential testing tools like Optimizely's Stats Engine or implement a simple SPRT. You'll get valid results faster, you can peek without guilt, and you'll avoid shipping changes that are actually no-ops. Run an A/A test first, watch for SRM, and let the sequential method tell you when to stop. Your conversion rate—and your sanity—will thank you.

Sources

• Optimizely (A/B testing) - https://www.optimizely.com/optimization-glossary/ab-testing/
• Optimizely Stats Engine - https://www.optimizely.com/insights/blog/statistics-for-the-internet-age-the-story-behind-optimizelys-new-stats-engine/
• Evan Miller - How Not to Run an A/B Test - https://www.evanmiller.org/how-not-to-run-an-ab-test.html
• Evan Miller - Simple Sequential A/B Testing - https://www.evanmiller.org/sequential-ab-testing.html
• Optimizely A/A testing glossary - https://www.optimizely.com/optimization-glossary/aa-testing/
• KDD 2019 - Diagnosing Sample Ratio Mismatch - https://www.kdd.org/kdd2019/accepted-papers/view/diagnosing-sample-ratio-mismatch-in-online-controlled-experiments-a-taxonom