The 96% That Lied to Me: Why I Stopped Trusting My A/B Test Dashboard

Last spring, I watched a dashboard tell me a new button color was a sure win. 96% significance, it said, after just three days. My finger hovered over the 'ship it' button. But something felt off. So I dug into the math, and what I found made me sick: that 96% was probably a lie.

Here's the thing about classic A/B tests: they assume you'll pick a sample size, run the experiment, and not touch it until the end. But we're human. We peek. And every time you peek and consider stopping, you're essentially running a new hypothesis test on the same data. Do that enough times, and your 'significant' result becomes about as reliable as a coin flip.

Optimizely's own research found that if you check your results daily and stop at the first sign of significance, your false-positive rate jumps from 5% to as high as 30% (Optimizely). Evan Miller's numbers are even more alarming: stop at 5% significance after every observation, and your actual false-positive rate hits 26.1% (Evan Miller). That means one in four of your 'winners' is pure noise.

I remember the exact moment this hit home. I was running a test on our pricing page—changing the CTA from 'Start Free Trial' to 'Get Started'. The tool showed a 96% significance level after three days. I almost shipped it. Then I remembered a tweet from Evan Miller: 'If you peek at an ongoing experiment ten times, what you think is 1% significance is actually just 5% significance' (Evan Miller).

So why do we still use fixed-sample tests? Because that's what we were taught. The classic recipe: calculate sample size, run, wait, look. But that recipe has a fatal flaw—it assumes you won't peek. And if you're human, you will.

Peeking: The Slot Machine Effect

Here's the mental trap: you check your test on Monday, see 85% significance. Tuesday, it's 90%. Wednesday, 95%—but you decide to wait one more day. Thursday, it's back to 88%. Now you're frustrated. You just want a win. So you stop on Friday when it hits 94% and tell yourself it's close enough. Sound familiar?

That's not experimentation. That's playing the slots. The numbers look great, but they're built on a shaky foundation. The more you peek, the more likely you are to stumble upon a spurious 'significant' result. It's not a discount; it's a lie.

How bad can it get? Let's do some math. Suppose you're testing a new checkout flow. Your baseline conversion is 3%. You want to detect a 10% relative lift—that's a delta of 0.003. Sigma-squared is 0.03 * 0.97 = 0.0291. Using the rule-of-thumb formula n = 16 * (sigma^2 / delta^2), you get n = 16 * (0.0291 / 0.003^2) = 16 * (0.0291 / 0.000009) = 16 * 3233 = 51,728 visitors per variation (Evan Miller). That's over 100,000 total. At 5,000 visitors a day, that's three weeks of waiting. Three weeks! Meanwhile, your competitor has already shipped three new features.

But wait—what if the truth is obvious after two days? Too bad. You're locked in. And there's a hidden cost: the opportunity cost of running one long test instead of several shorter ones (Optimizely). While you're waiting for your 100,000 visitors, you could have run three smaller tests. But if you try to run those with fixed-sample methods, you'll fall into the same peeking trap.

Sequential Testing: Peek All You Want, No Guilt

Sequential testing flips the script. Instead of forcing you to commit to a fixed sample size, it lets you look at your data as often as you want, with the guarantee that your false-positive rate stays controlled. It's been around for decades, but it's finally making its way into mainstream A/B testing platforms.

How does it work? In Evan Miller's simple sequential procedure, you choose a maximum sample size N, assign visitors randomly 50/50, and stop when the treatment-minus-control success count reaches 2 times the square root of N (declaring a winner) or the total success count reaches N (declaring no winner) (Evan Miller). That's it. No p-value calculations until the end, and no penalty for peeking along the way.

This method can reduce the number of observations needed for a successful experiment by 50% or more, and it works extremely well with low conversion rates (Evan Miller). For our checkout test, with a 3% baseline, we could potentially call a winner in 10 days instead of 20. That's a massive speed boost for a product team that needs to iterate quickly.

Why Your Tool Might Be Lying to You (and How to Fix It)

If you're using a classic frequentist tool that reports p-values and confidence intervals, it's probably not designed for continuous monitoring. Optimizely's own research shows that when they re-ran 48,000 historical experiments, fixed-horizon statistics declared a winner or loser in 36% of tests, while their sequential Stats Engine declared winners or losers in only 22%—39% fewer conclusive results (Optimizely). That's a huge difference. With fixed-horizon, you're more likely to declare a winner, but many of those declarations are false.

So what should you do? First, check if your tool supports sequential testing. Optimizely's Stats Engine, launched in 2015, uses sequential methods and can call a winner or loser up to 2.5 times faster than fixed-horizon statistics for experiments over 50,000 visitors (Optimizely). Similarly, VWO's Bayesian approach lets you monitor continuously without inflating false positives and can provide actionable results almost 50% faster (VWO). If your tool doesn't offer this, consider switching or using a standalone sequential calculator.

Second, if you're stuck with a classic tool, at least pre-register your sample size and resist the urge to peek. But let's be honest—will you? The whole point of this article is that we can't trust ourselves. That's why we need better tools.

A Field Guide to Sequential Adoption (or: How I Learned to Stop Worrying and Love the Dashboard)

Here's a realistic scenario: You're the data scientist at an e-commerce site. Your conversion rate is 3%, and you want to test a new checkout flow. Your old tool says you need 50,000 visitors per variation, which at 5,000 visitors a day is 20 days. That's too long. You switch to a sequential tool, set a maximum of 50,000 visitors, and let it run. After 10 days, the tool signals a winner. You peek at the dashboard daily, but you know the sequential method keeps the false-positive rate at 5%. You ship the new flow, and A/B test after A/B test, you see the same pattern: faster decisions, fewer false alarms.

Of course, sequential testing isn't a silver bullet. You still need to watch out for other issues like Sample Ratio Mismatch (SRM), which can indicate data quality problems (KDD 2019). And you should still run A/A tests periodically to validate your setup (Optimizely). But the core insight is clear: if you're not using sequential methods, you're leaving speed and reliability on the table.

So next time your dashboard flashes a shiny 96%, take a breath. Ask yourself: is my tool designed for peeking? Or am I about to ship a ghost? For me, the answer was to switch. My only regret is not doing it sooner.

Sources

• Optimizely - https://www.optimizely.com/optimization-glossary/ab-testing/
• Evan Miller - https://www.evanmiller.org/how-not-to-run-an-ab-test.html
• Optimizely Stats Engine - https://www.optimizely.com/insights/blog/statistics-for-the-internet-age-the-story-behind-optimizelys-new-stats-engine/
• VWO - https://vwo.com/ab-testing/
• KDD 2019 - https://www.kdd.org/kdd2019/accepted-papers/view/diagnosing-sample-ratio-mismatch-in-online-controlled-experiments-a-taxonom