Peeking Is Not the Only Sin: The Hidden Risks of A/B Tests

Everyone talks about peeking. And sure, checking your results daily and stopping as soon as you see significance can inflate your false-positive rate to as high as 30% (Optimizely). But I'm here to tell you that peeking is not the only sin—and it might not even be the most dangerous one. The real threats to your A/B testing integrity are silent, subtle, and far less discussed. If you're not watching for sample ratio mismatches, novelty effects, and the perils of multiple comparisons, you're flying blind.

I've spent years running and reviewing A/B tests, and I've seen more bad decisions come from these hidden pitfalls than from simple peeking. So let's bust some myths and answer the questions you should actually be asking.

Is Peeking Really That Bad?

Yes, but not for the reason you think. Peeking inflates false positives because if you stop as soon as you see significance, you're cherry-picking the moment when random noise looked like a real effect. Evan Miller's worst-case analysis shows that stopping at the first 5% significance can push your actual false-positive rate to 26.1% (Evan Miller - How Not to Run an A/B Test). That's one in four tests declaring a winner that doesn't exist. But here's the thing: the solution isn't just to stop peeking. It's to use methods that allow peeking without penalty.

What Is a Sample Ratio Mismatch and Why Should I Care?

Imagine you set up a 50/50 split between control and variation, but when you check your data, you see 45% in one group and 55% in the other. That's a sample ratio mismatch (SRM). It's like a fever—it's not the disease itself, but a symptom of underlying issues like a bug in your tracking code or a caching problem. Ignoring an SRM can lead you to ship a bad change because the data is skewed (KDD 2019 - Diagnosing Sample Ratio Mismatch). I once saw a test where the variation had a 20% higher conversion rate, but the SRM was so severe that the result was pure artifact. Always check your sample ratios before you trust any outcome.

What Are Novelty and Primacy Effects?

Users don't behave like lab rats. When you introduce a new feature, some users will click it just because it's new—that's the novelty effect. Others will slowly warm up to it—that's the primacy effect (arXiv - Novelty and Primacy (Sadeghi et al.)). These effects can make a new variation look brilliant in week one and terrible in week four, or vice versa. If you run a short test, you might mistakenly kill a good idea or ship a bad one. The fix is to run tests long enough to let these effects stabilize, which often means going beyond the minimum sample size.

Is Bayesian Testing the Cure-All?

Bayesian methods are all the rage, and for good reason: they let you monitor continuously without inflating false positives, and they give you intuitive statements like "there's a 92% probability that Variant B is better" (Optimizely). They can also produce actionable results almost 50% faster (VWO). But they're not magic. You still need to define your prior, and if you're not careful, you can fool yourself. I've seen people use Bayesian methods as an excuse to peek even more, which defeats the purpose. The real advantage is that you can peek without penalty, but you still need to pre-register your analysis plan and be honest about your prior.

Why Is My Sample Size Calculation So Complicated?

Sample size isn't just a number you Google. It depends on your baseline conversion rate, your minimum detectable effect (MDE), your significance level (usually 0.05), and your desired power (often 0.8) (Optimizely sample size guide). Evan Miller gives a handy rule of thumb: n = 16 * (sigma^2 / delta^2), where delta is the effect you want to detect and sigma^2 is the variance (for a binomial, that's p(1-p)) (Evan Miller - How Not to Run an A/B Test). But here's the twist: if you use sequential testing, you can often need fewer visitors. In some cases, the sequential procedure can reduce the number of observations needed by 50% or more (Evan Miller - Simple Sequential A/B Testing). So stop calculating a fixed sample size and then peeking—that's the worst of both worlds.

What's the Best Way to Run an A/B Test?

Stop using fixed-horizon tests and start using sequential methods. Sequential testing allows you to peek at any time and still have valid results (arXiv - Always Valid Inference (Johari et al.)). It's the difference between driving with a blindfold and driving with a GPS. When Optimizely re-ran 48,000 historical experiments, fixed-horizon statistics declared a winner or loser in 36% of tests, while their sequential engine did so in only 22%—that's 39% fewer conclusive results (Optimizely Stats Engine). Why? Because sequential methods are more conservative, but they're also more honest. Netflix uses sequential canary tests to detect regressions in the tail of metrics like PlayDelay, not just the mean (arXiv - Netflix canary testing (Lindon et al.)). That's the future.

Here's a quick tip: before you launch any test, run an A/A test—two identical versions—to calibrate your system. If you see a "significant" difference between identical pages, something is wrong with your setup (Optimizely A/A testing glossary).

Bottom Line

Stop treating A/B testing like a coin flip. Use sequential testing, check for sample ratio mismatches, and let novelty effects fade. If you only take one thing from this, make it this: switch to a sequential testing tool today. Your future self—and your conversion rate—will thank you.

Sources

• Optimizely - https://www.optimizely.com/optimization-glossary/ab-testing/
• Evan Miller - How Not to Run an A/B Test - https://www.evanmiller.org/how-not-to-run-an-ab-test.html
• Optimizely Stats Engine - https://www.optimizely.com/insights/blog/statistics-for-the-internet-age-the-story-behind-optimizelys-new-stats-engine/
• arXiv - Always Valid Inference (Johari et al.) - https://arxiv.org/abs/1512.04922
• KDD 2019 - Diagnosing Sample Ratio Mismatch - https://www.kdd.org/kdd2019/accepted-papers/view/diagnosing-sample-ratio-mismatch-in-online-controlled-experiments-a-taxonom