How to Stop Peeking and Let Your A/B Test Run: A Field Guide

You've got a shiny A/B testing tool, and it's tempting to check your results every day. But that's the fastest way to get a false positive. Peeking at your data and stopping early inflates your error rate from 5% to as high as 30% (Optimizely). That's not a minor issue—it's a recipe for shipping a bad feature. Here's how to do it right, using a concrete scenario.

The Scenario: You're a Product Manager at a SaaS Company

Imagine you're a product manager at a SaaS company. You've just redesigned your pricing page, and you want to see if the new design increases sign-ups. You set up an A/B test in your favorite tool: control vs. variation. The tool asks for a sample size, but you're not sure what to enter. You're tempted to just run it for a week and see what happens.

Stop right there. That's the classic peeking trap. You need to plan the test properly before you launch it. Here's the step-by-step process.

Step 1: Choose Your Sample Size and Duration

Before you launch, you need to decide how long to run the test. This is not a guess—it's a calculation based on your baseline conversion rate and the minimum effect you care about (Optimizely). For example, say your current sign-up rate is 5%. You want to detect a 10% relative improvement (so a 5.5% conversion rate). Using a power analysis with 80% power and 5% significance, you can calculate your required sample size. A rule of thumb is n = 16 * (variance / effect^2), where variance for a conversion rate is p(1-p) (Evan Miller). For p=0.05, variance is 0.0475, and effect is 0.005 (absolute difference). So n = 16 * (0.0475 / 0.000025) = 30,400 per variation. That's a lot of visitors. But you might not have that much traffic. If you get 1,000 visitors per day, you'd need 61 days to reach that sample size. That's a long time, but it's the honest cost of detecting a small effect.

If that's too long, you have two options: accept a larger minimum detectable effect (which means you'll only catch big changes) or use a sequential testing method. Sequential testing can cut your sample size by up to 50% (Evan Miller). That's a game-changer for busy teams.

Step 2: Understand the Peeking Problem

Why is peeking such a big deal? Because every time you check the data and consider stopping, you're doing a significance test. If you do this repeatedly, your false-positive rate skyrockets. Evan Miller shows that if you peek ten times, what you think is 1% significance is actually just 5% significance (Evan Miller). In a worst-case scenario, stopping as soon as you see 5% significance after every observation gives a true false-positive rate of 26.1% (Evan Miller). That's a 1-in-4 chance of shipping a useless change.

Your tool might try to help with this. Some tools use sequential testing, which lets you peek without inflating error rates. But if your tool uses traditional frequentist statistics, you must resist the urge to peek and stop early. The fix is to pre-commit to a fixed sample size or end date, or use a tool that supports sequential methods (Optimizely).

Step 3: Run the Test and Resist Peeking

Now you've got your plan. You'll run the test for 61 days (or use sequential testing to shorten it). During the test, you'll be tempted to check the dashboard. Here's the thing: peeking is okay as long as you don't stop early. You can look at the data for sanity checks, but you must not call a winner until the planned end. If you see a result that looks promising on day 3, ignore it. It's just noise.

If you're using a sequential method, you can stop early if the result is overwhelming. Evan Miller's simple procedure: pick a sample size N, assign users 50/50, and stop when the difference in successes reaches 2 * sqrt(N) (declaring a winner) or total successes reach N (declaring no winner) (Evan Miller). This gives you valid inference at any stopping time.

But if you're using a fixed-horizon test, you must wait. And here's a key point: even if you wait, you might still get a false positive. That's why you need to validate your results with an A/A test. Run two identical versions against each other. If your tool reports a significant difference between them, something is wrong with your implementation (Optimizely). Do this quarterly to keep your setup honest (Optimizely).

Step 4: Check for Sample Ratio Mismatch (SRM)

Another thing to check before you trust results: sample ratio mismatch. If your test is supposed to split traffic 50/50, but you're seeing 48/52, that's a red flag. SRM is a symptom of data quality issues, and ignoring it can lead you to ship a bad feature (KDD 2019). Your tool might have a built-in check for this, but if not, you can do a quick chi-square test. If you see an SRM, investigate the cause—maybe a caching issue or a race condition—before trusting any result.

What I'd Actually Do

Here's my blunt advice: stop relying on your tool's default settings. Take control of the experiment design. First, estimate your baseline conversion rate and decide on a minimum effect you care about. Then, use a sample size calculator to get the required number of visitors per variation. If that number is too large for your traffic, consider using a sequential testing method—many tools support this now, and it can save you weeks. If your tool doesn't support sequential testing, switch to one that does, or set up a manual procedure.

Second, commit to a fixed end date and don't touch the test until then. If you must peek, use a tool that provides always-valid p-values, so your peeking doesn't invalidate the results (arXiv). Third, run an A/A test quarterly to ensure your setup is sound. And finally, always check for sample ratio mismatch before declaring a winner.

This might feel like overkill, but it's the difference between trusting your optimization program and chasing noise. You've got better things to do than ship a feature that doesn't work.

Sources

• Optimizely - A/B testing glossary - https://www.optimizely.com/optimization-glossary/ab-testing/
• Evan Miller - How Not to Run an A/B Test - https://www.evanmiller.org/how-not-to-run-an-ab-test.html
• Optimizely - Stats Engine - https://www.optimizely.com/insights/blog/statistics-for-the-internet-age-the-story-behind-optimizelys-new-stats-engine/
• Optimizely - A/A testing glossary - https://www.optimizely.com/optimization-glossary/aa-testing/
• KDD 2019 - Diagnosing Sample Ratio Mismatch - https://www.kdd.org/kdd2019/accepted-papers/view/diagnosing-sample-ratio-mismatch-in-online-controlled-experiments-a-taxonom
• Evan Miller - Simple Sequential A/B Testing - https://www.evanmiller.org/sequential-ab-testing.html