A/B Testing Field Notes: A Growth PM's Playbook for Clean Experiments

The Messy Reality of A/B Testing

Six months into my first real C-end product role, I've run more A/B tests than I can count—and messed up plenty of them along the way. The first one I did in an internship was just different push notification copy, which barely counts. But the real ones, the ones that decide whether a feature ships or a strategy scales, those are a different beast.

Most people think A/B testing is just splitting traffic and comparing numbers. It's not. The number of times I've seen an experiment fail because the setup was sloppy, or the metrics were muddy, or nobody knew what to look at after launch—it's embarrassing. So I put together a standard operating procedure that covers the whole lifecycle, from hypothesis to decision. It's not perfect, but it's saved me from a lot of late-night debugging.

Two Ways to Run an Experiment: A/B Test vs. Gray Release

In real product work, you'll mostly use one of two approaches. The first is the classic A/B test: split users randomly into a control group (old version) and a treatment group (new version), run them in parallel, and compare a few key metrics. It's clean, it's controlled, and it gives you a trustworthy read on whether your change actually moved the needle. That's why most growth, recommendation, and search teams default to it.

The second is gray release (or canary release). You roll the new feature out to a tiny slice of users—say, 5%—and watch what happens. If nothing blows up, you increase the percentage gradually. The goal here isn't to measure effect; it's to control risk. I've learned to include your test and engineering buddies in that first 5% so you can catch bugs early, and make sure your operations team is watching for complaints. A gray release can exist independently of a formal A/B test, but you can also run both at once—gray for safety, A/B for measurement.

Two Non-Negotiables in Experiment Design

First: one variable only. If you change the UI, the recommendation algorithm, and the sorting logic all at once, and your conversion rate goes up, you have no idea why. That's not a test; that's a guess with extra steps. Keep the treatment and control identical except for the single thing you're testing.

Second: think about the effective sample size, not just the traffic split. A 20% traffic split sounds solid, but if your new strategy only affects a subset of content—say, 20% of what gets exposed—then the real sample is just 4% of your total users. I've seen this trap more than once. Before you launch, ask: how many users will actually be touched by this change? Then size your traffic accordingly.

The Full A/B Test Workflow

Here's the sequence I follow, religiously now:

• Ask the right question. Start with a business problem: conversion dropped, click-through is low, feature usage is flat. Then form a hypothesis that's testable—"If we change X, then Y will improve."
• Design the experiment. Decide on the unit (user ID or device ID—pick one and stick to it), set up control and treatment groups, and calculate the minimum sample size based on your baseline conversion and expected lift. I usually run these for 7 to 14 days to catch day-of-week effects.
• Launch with a small pilot. Before going big, run a 2–5% traffic test for a few hours just to check that the feature actually works, the tracking events fire correctly, and the metrics compute as expected. It's not about results; it's about sanity.
• Scale up. Once the pilot passes, expand to 10–50% and start looking at data after a day or two. If things look good, step up gradually: 10%, 20%, 50%, then full rollout.

Build a Metric Hierarchy, Not Just One Number

You can't judge an experiment by a single metric. I use four categories:

• Core metrics: The main goal—conversion rate, GMV, retention. This determines go/no-go. A critical pro tip: before you finalize, trace exactly how this metric is computed. For an e-commerce conversion rate, know the exact button click event. If the copy on that button changes, it might break your tracking.
• Input metrics: Did the strategy actually get used? Feature usage rate, coverage, share of impressions. If these are low, your core metric might not move just because nobody saw the change.
• Process metrics: The funnel from exposure to click to conversion. These help you locate where the drop-off happens.
• Guardrail metrics: The things that shouldn't get worse—user negative feedback, time spent, system performance. If these tank, even a positive core metric might not be worth shipping.

How to Analyze the Results

My analysis rule: start with the big picture, then drill down. First, verify the strategy actually affected the intended users. Then look at the overall core metric—did it move? If yes, is the change plausibly caused by this experiment, not some external factor? If it looks real, start slicing.

Slice by user type (new vs. returning, active vs. lapsed, different interest segments) and by content or product attributes (exposure, clicks, conversions for different types). This is where you start to understand why something worked or didn't. For example, a recommendation tweak might boost overall CTR but only because it helped new users—and hurt returning ones. That's a story you need to tell.

Making the Call: Ship or Kill

When the data's in, you have to decide. My checklist:

• Was there random assignment and a control group?
• Is the core metric statistically significant?
• Is the effect size big enough to matter in the real world?
• Are guardrail metrics stable?
• Does the result hold up over time, or is it a short-term spike?
• Did we rule out system biases or weird outliers?
• Is the sample representative of the broader user base?

If you can check all those boxes, you can trust the result. If not, treat it as a hint, not a verdict.

Here's the thing: A/B testing isn't just a "split traffic and see" exercise. It's a full loop—problem definition, experiment design, careful observation, deep analysis, and a clean decision. Building a repeatable process is what turns data into better product decisions. It's not glamorous, but it works.