A/B Test Peeking: The 26.1% False-Positive Trap and How to Escape It

Imagine you're a product manager at a mid-sized SaaS company. You've launched a new onboarding flow, and after just three days, the dashboard shows a 12% lift in activation with 95% significance. Your instinct screams "Ship it!" But then you remember the last time you did that—and the feature you rushed out ended up hurting retention.

Peeking at an A/B test before it's finished is one of the most common and dangerous mistakes in experimentation. The numbers are brutal: checking results daily and stopping as soon as significance appears can inflate your false-positive rate from 5% to as high as 30% (Optimizely). In a worst-case scenario where you stop at 5% significance after every observation, the actual false-positive rate can hit 26.1% (Evan Miller). That's a one-in-four chance of shipping a dud.

This article is for anyone who runs or analyzes A/B tests—marketers, product managers, data analysts. We'll walk through a practical, step-by-step process to keep your tests honest and your decisions sound.

Who This Is For (and Why You Should Care)

If you've ever felt that itch to check your test results before the planned end date, this is for you. We're going to show you how to resist that temptation and, if you absolutely must peek, how to do it without wrecking your conclusions. We'll cover the core concepts of hypothesis testing, power, and sample size, and then give you a concrete plan to follow.

Step 1: Understand the Null Hypothesis and What a P-Value Really Means

Every A/B test starts with a null hypothesis: the assumption that there's no real difference between the control and the variation (Spotify). The p-value tells you the probability of seeing data as extreme as what you observed, assuming the treatment has no effect. A p-value of 0.03 means such a result would happen by chance 3% of the time (Optimizely). But here's the kicker: it does not mean there's a 97% chance your variant is better (Optimizely). That's a common misconception.

When you peek and stop early, you're effectively running multiple hypothesis tests. Each look increases the chance of a false positive. The more you peek, the more your significance levels are off (Evan Miller). So before you even start, you need to decide on your sample size and stick to it.

Step 2: Pre-Commit to a Sample Size and End Date

The simplest defense against peeking is to calculate your required sample size upfront and set a fixed end date. Sample size calculations depend on your baseline conversion rate and the minimum detectable effect (MDE) you care about (Optimizely). A rule-of-thumb formula is n = 16 × (σ² / δ²), where δ is the minimum effect and σ² is the variance (Evan Miller). For a binomial conversion rate, variance is p(1-p).

Let's make this concrete. Suppose your baseline conversion is 10%, and you want to detect a 20% relative lift (so an absolute increase of 2 percentage points). That means your MDE is 0.02. Your variance is approximately 0.1 × 0.9 = 0.09. Plugging in: n = 16 × (0.09 / 0.02²) = 16 × (0.09 / 0.0004) = 16 × 225 = 3,600 visitors per variation (Evan Miller). If you have 1,000 visitors a day, you'd need about 7.2 days for both groups (Optimizely). So you'd pre-commit to running the test for 8 days, no peeking.

This approach is boring but effective. It prevents the inflated false positives that come from peeking (Optimizely). And it's the recommended practice from Evan Miller: decide on a sample size in advance and restrain yourself from stopping early (Evan Miller).

Step 3: Use Sequential Testing for Flexibility

But what if you can't wait? Maybe your product team needs a decision fast, or you're running a canary test for a new feature. Sequential testing methods allow you to stop early if the result is clearly positive or negative, without inflating false positives (Evan Miller). Optimizely's Stats Engine, for example, uses sequential testing and can call a winner or loser up to 2.5 times as fast as fixed-horizon statistics for experiments over 50,000 visitors (Optimizely).

One simple sequential procedure from Evan Miller: choose a sample size N, assign subjects randomly 50/50, and stop when the treatment-minus-control success count reaches 2×√N (declaring a winner) or the total success count reaches N (no winner) (Evan Miller). This can reduce the number of observations needed by 50% or more in some cases (Evan Miller).

Netflix uses similar sequential canary tests to monitor streaming quality metrics, flagging regressions while controlling false-detection probability (arXiv). The key is that you can peek as often as you want, and your inference remains valid (arXiv).

Step 4: Watch Out for Sample Ratio Mismatch (SRM)

Even if you follow the plan, your data might be lying to you. A Sample Ratio Mismatch (SRM) occurs when the observed sample ratio differs from the expected ratio—like a fever, it's a symptom of underlying data quality issues (KDD). Ignoring an SRM can cause a bad feature to look good and get shipped, or vice versa (KDD). So before you analyze results, check that your traffic split is actually 50/50 (or whatever you set). If it's off, investigate the cause before trusting any numbers.

Also, be aware of novelty and primacy effects. Novelty is the initial spike in engagement with a new feature that fades over time, while primacy is the gradual increase as users learn it (arXiv). These can skew your results if you run the test for too short a time. That's another reason to stick to your pre-determined duration.

Step 5: Interpret Results with Confidence Intervals, Not Just P-Values

When the test is over, don't just look at the p-value. Look at the confidence interval. A 95% confidence interval is the range that, if you repeated the experiment many times, would contain the true effect 95% of the time (Optimizely). It gives you a sense of the precision of your estimate. If the interval is wide, you might need more data even if the p-value is significant.

Also, compute the relative lift: (variant rate – control rate) / control rate × 100 (Optimizely). This tells you the practical impact, not just the statistical one.

What Can Go Wrong: The 26.1% Trap

Consider a cautionary tale: a team peeks at their test every day and stops as soon as they see 95% significance. In the worst case, with a significance test after every observation, the actual false-positive rate is 26.1% (Evan Miller). That means over a quarter of their "wins" are actually due to chance. They might ship a feature that does nothing or even hurts performance, all because they were impatient.

Another classic mistake is testing many goals and variations simultaneously. Peeking and testing many goals can increase the chance of incorrectly declaring a winner by over 5x (Optimizely). In fact, when Optimizely re-ran 48,000 historical experiments, fixed-horizon statistics declared a winner in 36% of tests, while sequential testing only found 22% conclusive—39% fewer false positives (Optimizely). So be selective about what you measure.

Step 6: Consider A/A Tests to Validate Your Setup

Before you trust your A/B test results, run an A/A test periodically—testing two identical versions—to make sure your system isn't broken. Optimizely recommends doing this quarterly (Optimizely). If an A/A test reports a statistically significant difference between identical versions, that's a red flag (Optimizely).

Step 7: Use Sequential Testing Tools or Pre-Commit—Your Choice

So what's the single best move? Pre-commit to a sample size and end date, or use a sequential testing tool that allows valid peeking. For most teams, I recommend the latter: sequential testing gives you flexibility without the statistical guilt. Tools like Optimizely's Stats Engine are built for this (Optimizely). If you're using a simple calculator, pre-commit and trust the plan.

Remember, the goal is to make good decisions, not to get a false sense of certainty. By following these steps, you'll avoid the 26.1% trap and ship features that actually work.

Bottom Line

The single best move you can make is to pre-commit to a sample size and end date, or use a sequential testing method that lets you peek safely. Don't let the siren song of early significance wreck your experiment.

Sources

• Optimizely - https://www.optimizely.com/optimization-glossary/ab-testing/
• Evan Miller - https://www.evanmiller.org/how-not-to-run-an-ab-test.html
• Optimizely Stats Engine - https://www.optimizely.com/insights/blog/statistics-for-the-internet-age-the-story-behind-optimizelys-new-stats-engine/
• KDD 2019 - https://www.kdd.org/kdd2019/accepted-papers/view/diagnosing-sample-ratio-mismatch-in-online-controlled-experiments-a-taxonom