Why Your A/B Test Results Are Lying: Peeking, Power, and Practical Significance

You've been running an A/B test for three days, and the numbers look great—your variation is up 10% with 95% significance. Can you call it a winner and ship it? No. That's the single most common mistake we see in experimentation. As a practitioner, you need to know the statistical traps that make results lie, and how to avoid them.

Why can't I just peek at my results and stop when it's significant?

Peeking—checking your test daily and stopping as soon as you see a p-value below 0.05—is the surest way to get a false positive. It's like flipping a coin until you get five heads in a row; eventually you will, but it doesn't mean the coin is biased. In A/B testing, if you peek at an ongoing experiment ten times, what you think is 1% significance is actually just 5% significance (Evan Miller - How Not to Run an A/B Test). In the worst case, where you stop as soon as 5% significance appears after every observation, the actual false-positive rate can reach 26.1% (Evan Miller - How Not to Run an A/B Test). That's a 1-in-4 chance of shipping a change that does nothing.

The fix is simple: pre-commit to a fixed sample size and end date before you start. If you can't resist the urge to peek, use sequential testing methods that adjust thresholds for multiple looks (Optimizely Stats Engine).

What does a p-value actually tell me?

Here's the misconception: a p-value of 0.03 does not mean your variation has a 97% chance of being better. It means that, if the treatment truly had no effect, you'd see data as extreme as yours 3% of the time (Optimizely (A/B testing)). It's a statement about the data, not about the probability that your hypothesis is true. So when a stakeholder says “we have a 97% chance this works,” gently correct them.

How do I choose a sample size in advance?

Sample size calculations boil down to four things: baseline conversion rate, minimum detectable effect (MDE), significance level (usually 0.05), and power (usually 0.8) (Optimizely sample size guide). A rule of thumb from Evan Miller is n = 16 × σ² / δ², where σ² is the variance (for a binomial rate, p(1-p)) and δ is the minimum effect you care about (Evan Miller - How Not to Run an A/B Test). For a conversion rate around 5% (σ² ≈ 0.0475) and an MDE of 1 percentage point (δ = 0.01), you'd need about 7,600 visitors per variation. That's a lot, which is why so many tests are underpowered.

Underpowered tests are a quiet killer. If you don't have enough traffic, you'll miss real effects, and you might even see a false negative and wrongly reject a good idea. Remember: power is the probability of correctly rejecting a false null hypothesis (Stat Trek (power of a hypothesis test)). And with a very large sample, even a microscopic improvement can become statistically significant—a 0.03% drop in form-field errors can produce a p-value below 0.001, but does that tiny change matter to your users or your business? (Nielsen Norman Group - Practical Significance).

What's the difference between statistical significance and practical significance?

Statistical significance (p < 0.05) just means the result is unlikely to be due to chance. It says nothing about the size or value of the effect (Nielsen Norman Group - Practical Significance). A completion-rate difference of 0.2 percentage points (85.0% vs. 85.2%) can be statistically significant (p = 0.03) yet be too small to justify a redesign (Nielsen Norman Group - Practical Significance). Conversely, an effect that isn't statistically significant might still be worth acting on—an 80% drop in task completion in a small usability test, even if it's not significant, deserves attention (Nielsen Norman Group - Practical Significance).

So before you launch a test, define what a meaningful effect looks like. That's your MDE. If the confidence interval for your lift is [0.01%, 0.5%], it's statistically significant but practically meaningless.

Why is my A/A test showing a 'significant' difference?

An A/A test—where both groups see the identical page—is a sanity check. If it reports a statistically significant difference (greater than 95% significance) between two identical versions, that's a red flag that your testing implementation is broken (Optimizely A/A testing glossary). Similarly, a Sample Ratio Mismatch (SRM)—where the observed sample ratio differs from the expected ratio—is a symptom of data quality issues (KDD 2019 - Diagnosing Sample Ratio Mismatch). Ignoring an SRM can make a bad change look good and get shipped, or vice versa (KDD 2019 - Diagnosing Sample Ratio Mismatch). Run A/A tests quarterly and always check for SRM before trusting results.

Should I use Bayesian or frequentist methods?

Both have their place, but here's our take: for most teams, sequential testing is the pragmatic middle ground. Bayesian methods give you probabilities like “92% chance B is better” and allow continuous monitoring without inflating false positives (Optimizely (A/B testing)). VWO reports that Bayesian can produce actionable results almost 50% faster (VWO A/B testing guide). But frequentist sequential testing—like Optimizely's Stats Engine—also lets you peek safely and can call a winner up to 2.5× faster for experiments over 50,000 visitors (Optimizely Stats Engine).

Our recommendation: if you're just starting, use a tool that does sequential testing or Bayesian analysis. If you're doing it yourself, pre-commit to a sample size and don't peek. And always remember that a p-value is not a verdict.

What about variance reduction? Can I get answers faster?

Yes—CUPED (Controlled-experiment Using Pre-Experiment Data) can cut your metric variance by about 50% using pre-experiment data, which means you need only half the users or half the time (Deng et al. - CUPED (WSDM 2013)). At Bing, three experiments showed variance reductions of 45%, 52%, and 49% (Deng et al. - CUPED (WSDM 2013)). If your platform supports it, use it. It's one of the few free lunches in experimentation.

Sources

• Optimizely (A/B testing) - https://www.optimizely.com/optimization-glossary/ab-testing/
• Evan Miller - How Not to Run an A/B Test - https://www.evanmiller.org/how-not-to-run-an-ab-test.html
• Optimizely sample size guide - https://www.optimizely.com/insights/blog/how-to-calculate-sample-size-of-ab-tests/
• Nielsen Norman Group - Practical Significance - https://www.nngroup.com/articles/practical-significance/
• Optimizely A/A testing glossary - https://www.optimizely.com/optimization-glossary/aa-testing/
• Deng et al. - CUPED (WSDM 2013) - https://exp-platform.com/Documents/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf