Stop Peeking at Your A/B Test: Sequential Testing Is the Fix

Imagine you're checking your A/B test dashboard on day three. The variation is up 12% and the p-value just dipped below 0.05. You're tempted to call it a win, ship it, and move on. But here's the catch: you peeked. And peeking is the quiet killer of reliable A/B tests.

If you stop the experiment now, your 'significant' result is likely a lie. The more you peek, the more you inflate your false-positive rate, and that 5% significance you're celebrating? It's not what it seems. So what's the solution? Stop using fixed-sample statistics that assume you'll only look once, and switch to sequential testing methods that let you peek anytime without breaking the math.

This article answers one specific question: how do you run an A/B test when you know you're going to peek? The answer isn't 'willpower.' It's sequential testing, and I'll show you why it's the only honest choice for real-world optimization.

The Peeking Problem: Why Your Dashboard Lies

Classical A/B testing is built on a simple premise: you pick a sample size in advance, run the test, and analyze the data exactly once. That's the fixed-horizon approach. The math works beautifully if you follow the rules. But nobody follows the rules. You peek. I peek. Everyone peeks.

And the cost is real. Checking results daily and stopping as soon as significance appears can inflate the false-positive rate from a nominal 5% to as high as 30% (Optimizely). That's not a rounding error; that's a broken experiment. Evan Miller, who's done the math on this, shows that if you peek at an ongoing experiment ten times, what you think is 1% significance is actually just 5% significance (Evan Miller). In a worst-case scenario where you stop as soon as 5% significance appears and run a significance test after every observation, the actual false-positive rate can reach 26.1% (Evan Miller).

Let me put that in context. Say you're testing a new checkout button. You decide to peek daily. After three days, you see a p-value of 0.04 and stop. But because you peeked, the true false-positive rate is somewhere between 5% and 26%, meaning there's a good chance you're shipping a change that does absolutely nothing. That's not optimization; that's gambling.

Why Fixed Sample Sizes Aren't the Answer

The textbook fix is to pre-commit to a sample size and never peek until the end. You calculate your sample size based on your baseline conversion rate and the minimum detectable effect you care about, set your power to 80%, and run the test (Optimizely sample size guide). That works if you have iron discipline. But real life gets in the way: a stakeholder asks for results, a bug surfaces, or you just get curious. And the moment you peek, you've violated the assumptions.

Even if you have the discipline, fixed-sample tests are inefficient. You're locked into a predetermined duration, which might be too long (costing you time and money) or too short (missing small effects). Evan Miller's rule-of-thumb sample size formula is n = 16 times (sigma-squared / delta-squared), where delta is the minimum effect you want to detect (Evan Miller). For a conversion rate of 10%, that means you'd need thousands of visitors to detect a 1% lift. That's a lot of traffic for a test that might be inconclusive.

And here's the kicker: even if you pre-commit, you're still vulnerable to the temptation to peek. The only way to make peeking safe is to use a method that accounts for it.

Sequential Testing: The Always-Valid Solution

Sequential testing is a different framework. Instead of a fixed sample size, you collect data continuously and can stop at any time, with valid statistical guarantees. The key idea is 'always-valid' p-values and confidence intervals, which let you peek as often as you like without inflating your false-positive rate (arXiv - Always Valid Inference).

Optimizely's Stats Engine, developed with Stanford statisticians, uses sequential testing. It began powering results for all Optimizely customers on January 21, 2015 (Optimizely Stats Engine). For experiments over 50,000 visitors, it can call a winner or loser up to 2.5 times as fast as fixed-horizon statistics (Optimizely Stats Engine). And when Optimizely re-ran 48,000 historical experiments, fixed-horizon statistics declared a winner or loser in 36% of tests while the sequential Stats Engine declared winners or losers in 22%, uncovering 39% fewer conclusive results (Optimizely Stats Engine). That might sound like a downside, but it means the sequential method is less likely to declare a false positive.

Evan Miller offers a simple sequential procedure that you can implement yourself: choose a sample size N, assign subjects randomly 50/50, and stop when the treatment-minus-control success count reaches 2 times the square root of N (declaring a winner) or the total success count reaches N (declaring no winner) (Evan Miller). In some circumstances, this can reduce the number of observations needed by 50% or more, and it works extremely well with low conversion rates (Evan Miller).

Comparing Fixed-Sample vs. Sequential Testing

Practical Recommendations: What to Do Now

So, what should you do? Here's my straight-talking advice:

• Stop peeking with fixed-sample tools. If you're using a tool that only offers fixed-horizon statistics, resist the urge to check daily. Pre-commit to a sample size and duration.
• Switch to a tool that supports sequential testing. Optimizely's Stats Engine is one option, but there are others. You can also implement Evan Miller's simple sequential procedure yourself.
• Run A/A tests to validate your setup. Optimizely recommends running A/A calibration tests quarterly (Optimizely A/A testing glossary). If an A/A test reports a significant difference between two identical versions, your implementation is broken.

Let me give you a concrete example. Say you're testing a new headline. Your baseline conversion rate is 5%, and you want to detect a 1% absolute lift (a 20% relative improvement). Using Evan Miller's rule of thumb, you'd need about 16 * (0.05*0.95) / (0.01^2) = 76,000 visitors per variation. That's a huge test. With sequential testing, you might stop after 38,000 visitors if the effect is real, or you might continue if it's not. You can peek every day, and your false-positive rate stays at 5%.

One more thing: sequential testing isn't just for conversion rates. Netflix uses sequential canary tests to monitor streaming quality, testing differences across the entire distribution rather than just the mean (arXiv - Netflix canary testing). They can flag a regression in PlayDelay (the time for a title to start once play is pressed) even if the mean looks fine, because the tail is worse. That's the power of sequential methods.

Takeaway

Peeking is inevitable. Stop pretending you won't do it. Instead, use sequential testing methods that let you peek anytime without lying to yourself. The math is on your side: always-valid p-values, lower false-positive rates, and faster decisions when the effect is real. Your optimization program will be more honest, and you'll ship fewer bad features. That's a win you can measure.

Sources

• Optimizely - https://www.optimizely.com/optimization-glossary/ab-testing/
• Evan Miller - https://www.evanmiller.org/how-not-to-run-an-ab-test.html
• Optimizely Stats Engine - https://www.optimizely.com/insights/blog/statistics-for-the-internet-age-the-story-behind-optimizelys-new-stats-engine/
• arXiv - Always Valid Inference - https://arxiv.org/abs/1512.04922
• Optimizely sample size guide - https://www.optimizely.com/insights/blog/how-to-calculate-sample-size-of-ab-tests/