Stop Peeking: How to Run Valid A/B Test Statistics
A practical walkthrough for analysts and PMs: how to avoid peeking, set sample size, and interpret p-values correctly in A/B tests.
Statistical Analysis
Statistical Analysis
8 articles in this category
Statistical Analysis
Peeking Kills A/B Tests: Why I Choose Sequential Testing Over Fixed-Horizon
Peeking inflates false positives to 30%. I compare fixed-horizon, sequential, and Bayesian methods to show when each works and which I trust.
Statistical Analysis
Stop Peeking: Why Your A/B Test Results Are a Statistical Illusion
Peeking at A/B test results can inflate false positives to 30%. Precommit to a sample size or use sequential testing to make valid decisions.
Statistical Analysis
Peeking Is Killing Your A/B Tests: Stop It Now
Checking results early and stopping at significance is a trap. It inflates false positives dramatically. Pre-commit or use sequential testing. Here's ...
Statistical Analysis
Stop Peeking: Why Your A/B Test Results Are a Lie
Peeking at A/B test results inflates false positives to 30%. Pre-commit to sample size or use sequential testing. Here's the fix.
Statistical Analysis
Why Your A/B Test Results Are Lying: Peeking, Power, and Practical Significance
Stop peeking, pre-commit to sample sizes, and remember: a p-value isn't the truth. Here's how real practitioners avoid the statistical traps that sink...
Statistical Analysis
Peeking vs. Sequential Testing: Why Your A/B Test Lies
Peeking at A/B test results inflates false positives. Fixed-horizon or sequential testing? Here's the straight talk on which to use and when.
Statistical Analysis
Peeking Is Not the Only Sin: The Hidden Risks of A/B Tests
Think peeking is your biggest A/B testing problem? Think again. Sample ratio mismatches, novelty effects, and multiple comparisons can quietly destroy...