Stop Peeking at Your A/B Test: Why Patience Beats p-Values

Here's a confession that might get me kicked out of the optimization club: I've stopped caring about p-values, at least in the way most people use them. We've all been taught that a p-value under 0.05 means our variation won, and that's the green light to ship it. But that's a dangerous oversimplification. The real enemy isn't the p-value itself—it's the way we peek at our data and stop the test the moment significance appears. That habit can inflate your false-positive rate from a nominal 5% to as high as 30% (Optimizely). If you're making product decisions based on that, you're not optimizing; you're gambling.

In this article, I'm going to bust some myths and answer the questions I hear most from practitioners. We'll talk about what p-values really mean, why peeking is so damaging, and how to run tests you can actually trust. Because at the end of the day, we're not in the business of collecting statistically significant notches on our belts—we're in the business of improving our products.

What does a p-value of 0.03 actually mean?

Ask five marketers, and you'll get five different answers. The most common misconception is that a p-value of 0.03 means there's a 97% chance that our variation is better. That's wrong. A p-value is the probability of observing data as extreme as what you saw, assuming the treatment has no effect. So a p-value of 0.03 means that if the variation truly had no effect, you'd see results this extreme just by chance 3% of the time (Optimizely). That's it. It says nothing about the probability that your variation is better. That's a subtle but crucial difference.

Think of it this way: under the null hypothesis—which assumes no real difference between control and variation (Spotify Confidence)—a p-value of 0.03 is a surprise. It's like rolling dice and getting snake eyes three times in a row. You start to suspect the dice are loaded, but you can't be certain. The p-value is a measure of surprise, not a measure of truth.

Why is peeking at my test results so dangerous?

I get it. The temptation to check results daily is almost irresistible. We're all eager to see if our shiny new button is crushing it. But here's the dirty secret: every time you peek and consider stopping, you're making a decision based on incomplete data. If you stop as soon as you see significance, you're more likely to catch a false positive—a result that looks significant but isn't real. The math is brutal: checking results daily and stopping at the first hint of significance can inflate the false-positive rate from a nominal 5% to as high as 30% (Optimizely). That means nearly one in three of your "wins" could be complete flukes.

The fix is simple: pre-commit to a fixed end date or use sequential testing methods that adjust thresholds for multiple looks (Optimizely). In other words, decide before you start how long you'll run the test, and don't touch it until that date. If you can't resist peeking, use a tool that does sequential testing for you—it's designed to handle multiple looks without inflating false positives.

What's the difference between Type I and Type II errors?

Here's a quick refresher for anyone who's been out of stats class for a while. A Type I error is a false positive—you conclude the variation is better when it's not. A Type II error is a false negative—you miss a real improvement (Spotify Confidence).

In the world of A/B testing, we tend to obsess over Type I errors because they lead us to ship changes that don't actually help. But Type II errors are just as damaging in their own way. If you run a test with too little traffic or stop too early, you might conclude that a variation is a dud when it actually would have been a winner. You've just missed out on a genuine improvement.

The balance between these errors is controlled by your significance level (for Type I) and your statistical power (for Type II). Before you even start, you should set your desired power—commonly 80%—and your minimum detectable effect (MDE) (Optimizely). The MDE is the smallest effect you care about detecting. If you set it too small, you'll need a huge sample size; too large, and you might miss a meaningful change.

How long should I run my A/B test?

There's no one-size-fits-all answer, but here's a rule of thumb: run it long enough to reach the sample size that gives you 80% power to detect your MDE. That means you need to calculate the required sample size before you start, not after. If you don't, you're flying blind.

For example, let's say your control conversion rate is 5% and you want to detect a 10% relative lift—that's a new conversion rate of 5.5%. To have 80% power, you'd need roughly 140,000 visitors per variant. If you're only getting 1,000 visitors a day, that's 140 days. That's a long test, but if you're making a major change, it's worth the wait.

Of course, sometimes you can't wait that long. That's where sequential testing comes in. It allows you to look at your data multiple times without inflating your false-positive rate, because it adjusts the significance thresholds as you go. It's not magic—it still requires a pre-specified maximum sample size—but it's a smarter way to handle the urge to peek.

Is Bayesian testing better than frequentist?

Ah, the eternal debate. Frequentist methods, which give you p-values and confidence intervals, are the standard in most tools. But Bayesian testing is gaining popularity because it outputs something that feels more intuitive: "there's a 92% probability that variant B is better" (Optimizely). That's a direct probability statement, which is exactly what most people think a p-value gives you, but it doesn't.

Bayesian testing also allows continuous monitoring without inflating false positives (Optimizely). You can peek all you want, and the posterior probability will just keep updating. That's a huge advantage for teams that can't resist looking.

But here's the catch: Bayesian methods require you to specify a prior, which introduces subjectivity. And the probability you get is only as good as your prior. If you're not comfortable with that, stick with frequentist—just follow the rules.

For most teams, I recommend Bayesian because it aligns with how we think and it's more forgiving of our human nature. But whichever you choose, the key is to pre-register your analysis plan. Decide your stopping rule, your significance level, and your power before you collect any data. That's the only way to get results you can trust.

What about relative lift and confidence intervals?

Once you've got your results, you'll often see metrics like relative lift and confidence intervals. Relative lift is calculated as (variant conversion rate minus control conversion rate) divided by the control rate, times 100 (Optimizely). So if your control converts at 5% and your variant at 5.5%, the relative lift is (0.055 - 0.05) / 0.05 * 100 = 10%. That's a handy way to communicate the practical impact.

A 95% confidence interval is a range that, if the experiment were repeated many times, would contain the true effect 95% of the time (Optimizely). So if your interval for the lift is [2%, 18%], you can be fairly confident the true lift is somewhere in that range. But remember: a confidence interval doesn't tell you the probability that the true effect lies within it. It's about the long-run frequency, not a single experiment.

Now, let's talk about a common misconception: "If the confidence interval includes zero, the test is inconclusive." That's true, but it doesn't mean the variation is useless. It just means you don't have enough evidence to say it's different from zero. You might be missing a real effect because your test was underpowered. That's a Type II error, and it's just as real as a false positive.

Bottom line

The single best move you can make is to pre-commit to your stopping rule before you launch the test. Decide your sample size, your significance level, and your power based on your MDE, and don't peek until the data is fully collected. If you absolutely must peek, use a sequential testing method or switch to a Bayesian approach that allows continuous monitoring without inflating false positives. But whatever you do, stop treating p-values like a magic wand. They're just a tool, and they're only useful when you use them correctly.

Sources

• Optimizely (A/B testing) - https://www.optimizely.com/optimization-glossary/ab-testing/
• Spotify Confidence - https://confidence.spotify.com/glossary/frequentist-ab-testing
• A/B testing terminology - https://en.wikipedia.org/wiki/A/B_testing