Stop Peeking: How Fixed-Horizon A/B Tests Sabotage Your Results

You've been lied to. The most common advice in A/B testing—"check your results daily and stop as soon as you hit significance"—is exactly how you end up shipping a change that does nothing, or worse, hurts your business. The numbers don't lie: if you peek at an ongoing experiment ten times, what you think is 1% significance is actually just 5% significance (Evan Miller). In a worst-case scenario where you stop the test the moment 5% significance appears, your actual false-positive rate can reach 26.1% (Evan Miller). That's a one-in-four chance of declaring a winner that doesn't exist. So stop peeking. Pre-commit to a sample size, and if you can't resist looking, use a sequential testing method that lets you peek without inflating your error rates. This article is for anyone who's ever felt the itch to check their dashboard after lunch. You'll learn a concrete, practical how-to for running A/B tests that you can actually trust.

Why Your Gut Is Your Worst Enemy

Let's be blunt: your intuition about when to stop an experiment is garbage. The classic approach—decide on a fixed sample size, run the test, then analyze once—is the gold standard, but it's wildly unintuitive. You want to see progress, you want to celebrate wins, and you want to kill losers fast. But every time you peek at the data and think "that looks significant," you're committing a statistical sin. Repeated significance testing always increases the rate of false positives (Evan Miller). The more you look, the more likely you are to see a mirage. You're not being smart; you're being impatient. The fix is simple: pick your sample size before you start, based on the minimum effect you care about, and don't touch the data until you've hit that number.

Set Your Sample Size Like You Mean It

Here's the practical part. You need to know your baseline conversion rate and decide on the minimum detectable effect (MDE)—the smallest improvement that matters to your business. With those, you can calculate the sample size per variation using the rule of thumb: n = 16 times (sigma-squared divided by delta-squared), where sigma-squared is the variance of your metric (for a binomial conversion rate, that's p(1-p)) and delta is your MDE (Evan Miller). Let's make it concrete. Say your current checkout completion rate is 5%, and you'd be thrilled with a 10% relative improvement, meaning an absolute lift of 0.5 percentage points. For a conversion rate, the variance is p(1-p) = 0.05 * 0.95 = 0.0475. Delta is 0.005. So n = 16 * (0.0475 / 0.000025) = 16 * 1900 = 30,400 visitors per variation. That's a lot, but it's honest. If you can only get 10,000 visitors, you won't detect a 0.5% change with 80% power—you'll be wasting your time.

Peeking Is a Disease—Here's the Cure

Now that you have a target sample size, you need the discipline to not look until the data is in. But if you're human, you'll peek. So what do you do? You have two options. First, you can pre-commit to a fixed end date and ignore the data until then. That's the classic Evan Miller advice: peeking is acceptable as long as you restrain yourself from stopping early (Evan Miller). But let's be real—most of us can't resist. That's where sequential testing comes in. Sequential methods like the Sequential Probability Ratio Test (SPRT) let you stop early if the treatment is a clear winner, without inflating your false-positive rate (Optimizely). Even simpler: Evan Miller's sequential procedure says to pick a sample size N, assign subjects randomly 50/50, and stop when the treatment-minus-control success count reaches 2 times the square root of N (declaring a winner) or the total success count reaches N (declaring no winner). In some cases, this can cut your required sample size by 50% or more (Evan Miller). It's a brilliant hack that gives you the speed you crave without the statistical corruption.

What Can Go Wrong: The Sample Ratio Mismatch Trap

Even if you set your sample size perfectly and resist peeking, you can still get garbage results if your test is broken. One of the most insidious issues is Sample Ratio Mismatch (SRM)—when the observed ratio of visitors in your control and variation doesn't match the expected 50/50 split. SRM is like a fever: it's a symptom of a variety of underlying data quality problems (KDD 2019). Ignoring an SRM without knowing its root cause may result in a bad product modification appearing to be good and being shipped to users, or vice versa (KDD 2019). So before you trust any result, check that your traffic split is actually what you think it is. If you see an SRM, stop the test, find the bug, and run it again. Your sample size calculations are worthless if the data you're collecting is garbage.

Statistical Significance Is Not Enough

Even if you do everything right—pre-committed sample size, no peeking, no SRM—you still need to ask: is this effect practical? Statistical significance only tells you the result is unlikely to be due to chance; it says nothing about how large or valuable the effect is (Nielsen Norman Group). For example, a completion-rate difference of 0.2 percentage points (85.0% versus 85.2%) can be statistically significant (p = 0.03) yet too small to justify a design change (Nielsen Norman Group). With a very large sample size, even a microscopic improvement such as a 0.03% drop in form-field errors can produce a p-value below 0.001 (Nielsen Norman Group). So don't be fooled by a tiny p-value. Look at the actual lift and decide if it's worth the effort. Conversely, an effect that is not statistically significant may still warrant action—for example, an 80% drop in task completion observed in a small-sample usability test (Nielsen Norman Group). Use your judgment, not just the p-value.

Case Study: Bing's Real-World Scale

To see these principles in action, look at Bing. On a typical day, over 250 experiments are running simultaneously, and almost every Bing user is exposed to about 15 different experiments (Bing). With that many tests, false positives are rampant, so Bing alerts experimenters to statistical interactions between experiments (Kohavi et al.). They also use pre-experiment data to reduce variance with CUPED, which in three recent experiments cut metric variance by 45%, 52%, and 49%—effectively achieving the same statistical power with half the users or half the experiment duration (Deng et al.). That's the kind of efficiency you should be aiming for. At Microsoft, a 1% improvement to revenue equals about $10 million annually in the US (Kohavi et al.), so even a small effect is worth detecting accurately. But remember, at Bing, less than a third of ideas move the metrics they were designed to improve (Bing). So don't fall in love with your hypothesis—let the data, properly gathered, be the judge.

Sources

• Evan Miller - How Not to Run an A/B Test: https://www.evanmiller.org/how-not-to-run-an-ab-test.html
• Evan Miller - Sample Size Calculator: https://www.evanmiller.org/ab-testing/sample-size.html
• Evan Miller - Simple Sequential A/B Testing: https://www.evanmiller.org/sequential-ab-testing.html
• KDD 2019 - Diagnosing Sample Ratio Mismatch: https://www.kdd.org/kdd2019/accepted-papers/view/diagnosing-sample-ratio-mismatch-in-online-controlled-experiments-a-taxonom
• Nielsen Norman Group - Practical Significance: https://www.nngroup.com/articles/practical-significance/
• Bing - Large Scale Experimentation at Bing: https://blogs.bing.com/search-quality-insights/August-2013/Large-Scale-Experimentation-at-Bing/
• Deng et al. - CUPED (WSDM 2013): https://exp-platform.com/Documents/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf