Your A/B Test Tool Is Probably Lying: Why You Need Sequential Testing

The Myth: Your A/B Test Results Are Trustworthy

There's a common misconception that if your A/B testing tool reports a p-value below 0.05, you have a winner. That's wrong. In fact, if you've been peeking at your results—which most of us do—your tool is likely lying to you. The problem isn't your experiment; it's the statistical method baked into your tool. Fixed-horizon testing, the old-school approach, is fundamentally flawed for the way we actually work. It assumes you'll decide your sample size in advance and never peek. But who does that? We're curious, impatient, and deadlines loom.

The good news? There's a better way: sequential testing. It's not new—the math has been around for decades—but it's finally available in mainstream tools. I'm not here to give you a balanced overview. I'm here to tell you to switch to sequential testing. Now.

The Question: Which Optimization Tool Should You Trust?

You're evaluating A/B testing platforms. You see terms like 'p-value,' 'confidence interval,' and 'sample size calculator.' They all look similar. But under the hood, they're not. The critical question is: does your tool use fixed-horizon statistics or sequential testing? This one choice determines whether your results are valid when you peek, and whether you're wasting time and money.

Many tools still default to fixed-horizon. They'll happily let you check results daily and stop as soon as you see significance. That's a trap. Let me show you why.

The Peeking Problem: How Your Tool Betrays You

Peeking—checking your test every day and stopping as soon as it hits 95% significance—inflates your false-positive rate. How bad? Evan Miller, a respected statistician, calculated that in a worst-case scenario where you stop as soon as 5% significance appears and run a significance test after every observation, your actual false-positive rate can reach 26.1% (Evan Miller). That means over a quarter of your 'winners' are actually placebos. If you peek ten times, what you think is 1% significance is actually just 5% significance (Evan Miller).

Your tool isn't stopping you from doing this. In fact, most tools encourage it by showing you colorful confidence intervals in real time. They're setting you up for false confidence. The more you peek, the more your reported significance levels are off (Evan Miller).

So what's the defense? Pre-committing to a fixed end date or using sequential testing methods that adjust thresholds for multiple looks (Optimizely). The former is hard in practice; the latter is the real solution.

Sequential Testing: The Fix You've Been Waiting For

Sequential testing flips the script. Instead of saying 'you must decide your sample size in advance and never peek,' it says 'peek all you want—your results remain valid.' How? It adjusts the thresholds for significance every time you look, using a method like the Sequential Probability Ratio Test (SPRT) (Optimizely sample size guide).

Evan Miller's simple sequential procedure is a beautiful example. You choose a sample size N, assign subjects randomly 50/50, and stop when the treatment-minus-control success count reaches 2 times the square root of N (declaring a winner) or the total success count reaches N (declaring no winner) (Evan Miller). That's it. No p-values to misinterpret. And it can reduce the number of observations required for a successful experiment by 50% or more, especially with low conversion rates (Evan Miller).

Netflix uses sequential canary tests for their streaming quality. They test the entire distribution, not just the mean, so a metric like PlayDelay—the time for a title to start once play is pressed—with a heavier tail gets flagged as a severe regression while controlling false-detection probability (arXiv - Netflix canary testing). If Netflix trusts it for something as critical as streaming performance, you can trust it for your button color.

Fixed-Horizon vs. Sequential: A Head-to-Head

Optimizely's Stats Engine, for example, uses sequential testing. When they re-ran 48,000 historical experiments, fixed-horizon statistics declared a winner or loser in 36% of tests, while the sequential engine did so in only 22%—uncovering 39% fewer conclusive results (Optimizely Stats Engine). That's not a bug; it's the truth. Many of those 'conclusive' results were false positives.

What About A/A Tests and Sample Ratio Mismatch?

Before you switch, make sure your tool can handle the basics. Run an A/A test—comparing identical versions—to validate your setup. If your tool reports a statistically significant difference between two identical versions, that's a red flag (Optimizely A/A testing glossary). It means your implementation is broken.

Also, check for Sample Ratio Mismatch (SRM). That's when the observed sample ratio differs from the expected ratio. It's a symptom of data quality issues, and ignoring it can make a bad variation look good (KDD 2019). Your tool should flag SRM automatically. If it doesn't, that's another reason to switch.

My Recommendation: Switch to Sequential Testing Today

Here's my advice, plain and simple: choose an A/B testing tool that uses sequential testing by default. Not one that offers it as an add-on, but one that's built on it. Optimizely's Stats Engine is one example. Others are emerging. Don't settle for a tool that lets you peek yourself into a 26.1% false-positive rate.

Quick tip: If you're stuck with a fixed-horizon tool, at least pre-commit to a sample size using a calculator and resist the urge to peek until the end. It's better than nothing, but it's not great.

Remember: The single most important thing to remember is that your A/B test results are only as trustworthy as the statistics engine behind them. Switch to sequential testing, and you'll stop fooling yourself.

Sources

• Evan Miller - How Not to Run an A/B Test - https://www.evanmiller.org/how-not-to-run-an-ab-test.html
• Evan Miller - Simple Sequential A/B Testing - https://www.evanmiller.org/sequential-ab-testing.html
• Optimizely Stats Engine - https://www.optimizely.com/insights/blog/statistics-for-the-internet-age-the-story-behind-optimizelys-new-stats-engine/
• arXiv - Netflix canary testing - https://arxiv.org/abs/2205.14762
• Optimizely A/A testing glossary - https://www.optimizely.com/optimization-glossary/aa-testing/
• KDD 2019 - Diagnosing Sample Ratio Mismatch - https://www.kdd.org/kdd2019/accepted-papers/view/diagnosing-sample-ratio-mismatch-in-online-controlled-experiments-a-taxonom