Stop Checking Your Dashboard: What Netflix and 48,000 Tests Teach About Peeking

One Number That Should Terrify You

Checking results daily and stopping as soon as significance appears can inflate the false-positive rate from a nominal 5% to as high as 30% (Optimizely). That means nearly a third of your "wins" could be pure noise. As practitioners, we've all been there: the dashboard glows green, the product manager starts drafting the launch email, and you feel that rush of validation. But that rush is exactly what the statistics are warning against.

The Case for Pre-Commitment

Here's my thesis: the single most important decision in A/B testing isn't the design, the metric, or even the sample size—it's the decision to stop peeking. Pre-commit to a fixed sample size and end date, and let the test run its course. This isn't just theoretical hand-waving; it's backed by hard numbers. When Optimizely re-ran 48,000 historical experiments with their sequential Stats Engine, they found that fixed-horizon statistics declared a winner or loser in 36% of tests, while the sequential method declared conclusive results in only 22%—that's 39% fewer false conclusions (Optimizely Stats Engine). Those extra 14% of tests that "won" under fixed-horizon were likely artifacts of peeking.

What Peeking Really Costs You

Let's put this in perspective. Say you're testing a new checkout button and your baseline conversion rate is 5%. You calculate you need 1,000 visitors per variation to detect a 2% lift with 80% power. But you peek at day two, see a promising 10% lift, and pull the plug. You've just made a decision based on a sample size that's far too small to be reliable. The p-value you're seeing is not what it appears. If you peek ten times, what you think is 1% significance is actually just 5% significance (Evan Miller). That's a five-fold increase in your false-positive rate. It's like reading tea leaves.

The Counter-Argument: Speed vs. Rigor

You might argue: "But we need to move fast!" I get it. The business wants answers yesterday, and waiting weeks for a test feels like an eternity. However, there's a better way. Sequential testing methods like the Sequential Probability Ratio Test (SPRT) can potentially reduce required sample sizes and save resources (Optimizely sample size guide). They allow you to peek at any time without inflating false positives, because the thresholds are adjusted for multiple looks (arXiv - Always Valid Inference). So you can have your cake and eat it too—if you're willing to abandon the illusion of fixed-horizon p-values.

What the Case Studies Show

Netflix uses sequential canary tests to compare entire distributions, not just means. In their PlayDelay metric—the time for a title to start once play is pressed—a heavier treatment tail would trigger a severe performance regression alert, while false-detection probability is strictly controlled (arXiv - Netflix canary testing). This is a real-world case where the cost of a false positive is enormous: shipping a degraded streaming experience to millions. They don't rely on gut checks; they rely on mathematically sound sequential methods.

Optimizely's Stats Engine, developed with Stanford statisticians, began powering results for all customers on January 21, 2015. For experiments over 50,000 visitors, it can call a winner or loser up to 2.5 times as fast as fixed-horizon statistics (Optimizely Stats Engine). That's speed without the statistical sin. When Optimizely re-ran those 48,000 historical experiments, they also found that roughly 20% fewer variations had a false discovery rate below 0.1 than with a false positive rate at the same level—a stark reminder of the multiple-comparisons cost of testing many metrics (Optimizely Stats Engine).

The Practical Fix: Embrace Sequential Testing

So what should you actually do? Stop relying on fixed-horizon p-values. Switch to a sequential testing framework. It's not just a nice-to-have; it's a necessity for any serious experimentation program. The numbers are clear: peeking can triple your false-positive rate, and sequential methods can cut required sample sizes by 50% or more (Evan Miller). If you're still using a fixed-sample approach, at least pre-commit to a sample size and resist the urge to peek. But better yet, adopt a tool that uses sequential analysis.

What I'd Actually Do

Here's my concrete recommendation: if you're running A/B tests, switch to a platform that defaults to sequential testing—like Optimizely's Stats Engine or a custom implementation of Evan Miller's simple sequential procedure. For a quick and dirty test, use this rule: choose a sample size N, assign users 50/50, and stop when the treatment-minus-control success count reaches 2 times the square root of N (declare a winner) or total successes reach N (declare no winner) (Evan Miller). This is simple, transparent, and it respects the data. It might feel counterintuitive to not peek at that tempting dashboard, but the alternative is shipping a feature that's 30% likely to be a dud. Don't let the green glow fool you.

Sources

• Optimizely - https://www.optimizely.com/optimization-glossary/ab-testing/
• Optimizely Stats Engine - https://www.optimizely.com/insights/blog/statistics-for-the-internet-age-the-story-behind-optimizelys-new-stats-engine/
• Evan Miller - How Not to Run an A/B Test - https://www.evanmiller.org/how-not-to-run-an-ab-test.html
• Evan Miller - Simple Sequential A/B Testing - https://www.evanmiller.org/sequential-ab-testing.html
• arXiv - Always Valid Inference - https://arxiv.org/abs/1512.04922
• arXiv - Netflix canary testing - https://arxiv.org/abs/2205.14762