Stop Babysitting Your A/B Test: Why Fixed Sample Sizes Are Your Friend

The 30% False Positive Problem

Here's a number that should scare you: checking results daily and stopping as soon as significance appears can inflate your false-positive rate from a nominal 5% to as high as 30% (Optimizely). That means nearly one in three of your "winning" variations is actually a loser. Yet most A/B testing tools make it ridiculously easy to peek—they show you a colorful graph with a "95% significance" badge the moment it appears. And we all know what happens next: we see that badge and we stop the test, high-fiving our team. But we're fooling ourselves.

My Thesis: Fixed Sample Size or Sequential Testing—No Exceptions

I'm going to make a controversial claim: if you're not using a pre-committed sample size or a sequential testing method, you're not actually running an A/B test—you're running a placebo that gives you false confidence. The entire edifice of statistical inference in A/B testing rests on the assumption that you decide how many observations you'll collect before you look at any data. The moment you peek and let that influence when you stop, you've corrupted the p-value. It's not a matter of discipline; it's a matter of math.

The fix is simple: choose a sample size in advance, based on the baseline conversion rate and the minimum effect you care about, and then don't stop until you've reached that number. Or, if you can't resist the allure of peeking, use a sequential method that lets you look at the data any time without inflating your error rates. But you have to pick one strategy and stick to it.

What Peeking Actually Does to Your p-Values

Let's get into the weeds. A p-value is the probability of seeing data as extreme as what you observed, assuming the treatment has no effect. A p-value of 0.03 means such a result would occur by chance 3% of the time (Optimizely). But if you peek at your data ten times, that 0.03 is a lie. Evan Miller, a statistician who has written extensively on this, notes that repeated significance testing always increases the rate of false positives. In fact, if you peek ten times, what you think is 1% significance is actually just 5% significance (Evan Miller). And in a worst-case scenario, where you stop as soon as 5% significance appears after every observation, the actual false-positive rate can reach 26.1% (Evan Miller). That's not a rounding error—that's a broken process.

Why does this happen? Because each time you look at the data and compute a p-value, you're essentially running a separate hypothesis test. The more tests you run, the higher the chance that at least one of them will randomly cross the 5% threshold. It's like playing the lottery: buy enough tickets, and you'll eventually win—but that doesn't mean your strategy is sound.

The Counter-Argument: "But I'm Just Checking In"

I can hear the pushback already: "I'm not stopping early—I'm just monitoring to make sure nothing is broken." That's a reasonable instinct, but it's dangerous. Even if you don't intend to stop, peeking can subconsciously influence your decisions. You might decide to extend the test, or kill it because it looks hopeless, or start planning a rollout based on a trend that's not yet significant. And the math doesn't care about your intentions. As Evan Miller puts it, peeking at the data is acceptable only if you restrain yourself from stopping before the experiment has run its course (Evan Miller). But can you really restrain yourself? I know I can't. That's why I prefer methods that make peeking harmless.

There's also a common belief that Bayesian methods let you peek safely. And it's true that Bayesian testing outputs probabilities like "there is a 92% probability that Variant B is better" and allows continuous monitoring without inflating false positives (Optimizely). But Bayesian methods come with their own baggage—they require priors, and the interpretation of the probability is different. If you're not comfortable with the philosophy, you might still be better off with a frequentist approach done correctly.

Sequential Testing: The Best of Both Worlds

The good news is that you don't have to choose between disciplined waiting and the freedom to peek. Sequential testing methods give you valid results no matter when you look at the data. The idea is to use a stopping rule that adjusts the significance threshold based on how many times you've checked the data. Evan Miller's simple sequential procedure, for example, chooses a sample size N, assigns subjects randomly 50/50, and stops when the treatment-minus-control success count reaches 2 times the square root of N (declaring a winner) or the total success count reaches N (declaring no winner) (Evan Miller). In some circumstances, this can reduce the number of observations required by 50% or more, and it works extremely well with low conversion rates (Evan Miller).

Optimizely's Stats Engine, developed with Stanford statisticians, uses sequential testing to let users see results that are always valid any time they peek, without needing to guess a minimum detectable effect in advance (Optimizely). When they re-ran 48,000 historical experiments, fixed-horizon statistics declared a winner or loser in 36% of tests, while the sequential Stats Engine declared winners or losers in only 22%—uncovering 39% fewer conclusive results (Optimizely). That might sound like a downside, but it's actually a feature: those 39% were likely false positives that would have led you to ship a bad change.

Even Netflix uses sequential canary tests to detect regressions across the entire distribution, not just the mean, so a metric like PlayDelay—the time for a title to start once play is pressed—with a heavier tail gets flagged as a severe performance regression while strictly controlling the false-detection probability (arXiv). If sequential testing is good enough for Netflix's streaming quality, it's good enough for your landing page.

What I'd Actually Do

Here's my concrete recommendation: stop using fixed-horizon p-values if you have any tendency to peek. Switch to a sequential method, either using a tool like Optimizely's Stats Engine or implementing Evan Miller's simple procedure yourself. If you must use a fixed sample size, then do the math upfront: calculate the sample size based on your baseline conversion rate and the minimum effect you want to detect, and then don't look at the results until you've reached that number. Period. And don't forget to run A/A tests quarterly to make sure your testing tool isn't lying to you (Optimizely).

The bottom line is that A/B testing is a statistical discipline, not a spectator sport. You can't just watch the game and call a winner whenever you feel like it. Pre-commit to a plan, or use a method that lets you peek without breaking the math. Your future self—and your conversion rate—will thank you.

Sources

• Optimizely - https://www.optimizely.com/optimization-glossary/ab-testing/
• Evan Miller - https://www.evanmiller.org/how-not-to-run-an-ab-test.html
• Optimizely Stats Engine - https://www.optimizely.com/insights/blog/statistics-for-the-internet-age-the-story-behind-optimizelys-new-stats-engine/
• Evan Miller - https://www.evanmiller.org/sequential-ab-testing.html
• arXiv - https://arxiv.org/abs/2205.14762
• Optimizely A/A testing glossary - https://www.optimizely.com/optimization-glossary/aa-testing/