How One Peeking Habit Nearly Shipped a Bad Feature: A Case Study

Imagine you're a product manager at a mid-sized SaaS company. You've just launched a redesigned onboarding flow, and you're eager to see if it boosts activation. You check the dashboard after day one: the variation is up 8%. After day two, it's up 12% and the p-value is below 0.05. You're tempted to call it a winner and ship it. But should you?

This is the exact scenario that leads to countless bad decisions. The answer is no—not yet. And the reason lies in the statistics behind A/B testing, specifically the dangers of peeking and the importance of pre-committing to a sample size. In this case study, we'll walk through how a typical team might fall into this trap and how you can avoid it using the facts from the field.

The Setup: A Typical Onboarding Test

Your team decides to run an A/B test comparing the current onboarding (control) against a new version (variation). The primary metric is activation rate—the percentage of new users who complete a key action within 24 hours. You know the baseline activation rate is about 20%. You want to detect a minimum effect of 2 percentage points (i.e., a 10% relative lift).

Using a standard sample size calculator, you find that you need a certain number of visitors per variation. The rule of thumb for sample size is n = 16 * (sigma^2 / delta^2), where sigma^2 is the variance and delta is the minimum effect you wish to detect (Evan Miller). For a binomial conversion rate, variance is p(1-p). With p=0.2, sigma^2 = 0.16. Delta = 0.02. So n = 16 * (0.16 / 0.0004) = 16 * 400 = 6400 visitors per variation. That's 12,800 total visitors.

You have about 1,000 visitors per day, so you estimate you'll need about 13 days to run the test (Optimizely sample size guide). You set a fixed end date two weeks out.

The Temptation: Peeking at the Data

But after just three days, you have about 3,000 visitors in each group. The variation is showing a 5% conversion rate, the control is at 4.5%, and the p-value is 0.04. That's below the conventional 0.05 significance level. Your dashboard even shows a 95% confidence interval that doesn't include zero. You're excited. You want to stop the test and ship.

Here's the problem: you peeked. Checking results daily and stopping as soon as significance appears can inflate the false-positive rate from a nominal 5% to as high as 30% (Optimizely). That means there's up to a 30% chance that you're seeing a difference that isn't real. In fact, if you peek ten times, what you think is 1% significance is actually just 5% significance (Evan Miller). And in a worst-case scenario where you stop as soon as 5% significance appears, the actual false-positive rate can reach 26.1% (Evan Miller).

So that 'statistically significant' result might be pure noise. You haven't reached your pre-committed sample size of 6,400 per variation. You're only at 3,000. The test isn't finished.

The Consequences: A False Positive That Would Have Shipped

If you stop early and ship the variation, you risk a Type I error—a false positive (Spotify Confidence). You'd be concluding the new onboarding is better when it isn't. That could mean wasting development resources and potentially harming user experience for no benefit. Worse, if you had an SRM (Sample Ratio Mismatch) in your data—say, due to a bug in your tracking—you might see a significant difference that's driven by data quality issues, not by your variation (KDD 2019). Ignoring an SRM can lead to shipping a bad modification (KDD 2019).

Let's put numbers on it. Suppose you did peek daily and stopped at day 3. With a nominal 5% significance level, the actual false-positive rate could be as high as 26.1% (Evan Miller). That's a one-in-four chance that your 'winner' is a loser. Would you ship a feature with a 26% chance of being harmful? Probably not.

The Alternative: Pre-commit or Use Sequential Testing

So what should you do instead? Two options are recommended by experts. The first is to pre-commit to a fixed sample size and end date, and not peek until then. This is the classic frequentist approach (Optimizely). The second is to use sequential testing, which allows you to peek at any time without inflating false positives (Optimizely Stats Engine; arXiv - Always Valid Inference).

Sequential methods like the Sequential Probability Ratio Test (SPRT) can reduce required sample sizes and save resources (Optimizely sample size guide). Evan Miller's simple sequential procedure: choose a sample size N, assign subjects randomly 50/50, and stop when the treatment-minus-control success count reaches 2 * sqrt(N) (declaring a winner) or the total success count reaches N (declaring no winner) (Evan Miller). This can reduce observations by 50% or more in some cases (Evan Miller).

Optimizely's Stats Engine, which uses sequential testing, can call a winner up to 2.5 times faster for experiments over 50,000 visitors (Optimizely Stats Engine). And it provides always-valid p-values, so you can peek as often as you like (arXiv - Always Valid Inference).

Comparison: Fixed-Horizon vs. Sequential Testing

The Recommendation: Stop Peeking, or Go Sequential

My recommendation is clear: unless you're using a sequential testing method, do not peek at your results. Pre-commit to a sample size and a fixed end date based on your baseline conversion rate and minimum detectable effect (Optimizely sample size guide). If you find it too hard to resist peeking, switch to a sequential testing tool like Optimizely's Stats Engine or implement Evan Miller's simple sequential procedure (Evan Miller).

Quick tip: If you're using a fixed-horizon test, set your sample size using the rule of thumb n = 16 * (sigma^2 / delta^2) and don't check the results until the test is complete. If you must peek, use a sequential method.

Wrap-Up: Guard Your False-Positive Rate

In our case study, the team that peeked and stopped early would have had a false-positive rate as high as 26% (Evan Miller). By pre-committing to a sample size of 6,400 per variation, they would have waited 13 days and gotten a reliable answer. Or, by using sequential testing, they could have checked daily and stopped as soon as an always-valid p-value crossed the threshold. The takeaway is simple: peeking at your A/B test results without a plan is a recipe for bad decisions. Pre-commit to a sample size, or use sequential testing. Your product—and your users—will thank you.

Sources

• Optimizely - https://www.optimizely.com/optimization-glossary/ab-testing/
• Evan Miller - https://www.evanmiller.org/how-not-to-run-an-ab-test.html
• Optimizely - https://www.optimizely.com/insights/blog/how-to-calculate-sample-size-of-ab-tests/
• Optimizely Stats Engine - https://www.optimizely.com/insights/blog/statistics-for-the-internet-age-the-story-behind-optimizelys-new-stats-engine/
• arXiv - Always Valid Inference - https://arxiv.org/abs/1512.04922