“The Life of an A/B Test: Everything That Happens Before, During, and After the Experiment”

Let’s follow an A/B test from beginning to end and see what actually happens inside a modern data-driven company.

Step 1: Someone Has an Idea

Every experiment begins with a hypothesis.

Imagine a product manager notices that many users abandon the checkout process before completing a purchase.

The team proposes a simple change:

“What if we reduce the checkout process from three pages to one page?”

Notice that this is not yet an experiment.It is merely an idea.Many ideas sound good. Some even seem obvious.The purpose of experimentation is to determine whether the idea actually creates value.

Step 2: Define the Business Question

A common mistake is jumping directly into implementation. Instead, teams first define the exact question they are trying to answer.

Bad question: “Does the new checkout page look better?”

Better question: “Does the new checkout page increase purchase conversion rate?”

The metric matters because experiments should answer business questions, not design opinions.

Step 3: Define Success Metrics

Before running the experiment, the team decides how success will be measured.

Primary metric:

• Purchase conversion rate

Guardrail metrics:

• Page load time
• Refund rate
• Customer support tickets

Imagine conversion increases by 10%, but refund rates double.

Would you still launch the feature? Probably not.

Guardrail metrics prevent teams from optimizing one metric while unknowingly damaging another.

Step 4: How Many Users Do We Need?

Before running an experiment, we must determine whether we can collect enough data to detect a meaningful effect.

Suppose your current conversion rate is 5%. You believe the new design can improve conversion to 5.5%. That sounds small.

Yet for a large business, a 0.5% absolute increase may represent millions of dollars annually. The challenge is distinguishing a real improvement from random variation.

This is where concepts such as:

• Statistical power
• Significance level
• Minimum Detectable Effect (MDE) become important.

If the sample size is too small:

• Real improvements may go undetected.

If the sample size is excessively large:

• Time and resources are wasted.

A surprising number of experiments fail before they even begin because insufficient traffic is allocated to them.

Step 4: Randomization — The Secret Ingredient

Randomization is the foundation of trustworthy experimentation.

Every eligible user is assigned to either:

• Control Group (A)
• Treatment Group (B)

through a random process.

Why?

Because users are different. Some users are loyal customers. Others are first-time visitors. Some browse on mobile devices. Others use desktops.

If one group contains systematically different users, comparisons become meaningless. Randomization helps balance these differences.

Step 5: Launching the Experiment

Once the experiment launches, teams monitor:

• Data pipelines
• Event logging
• User assignment systems
• Traffic allocation

A perfectly designed experiment can still fail if the instrumentation is broken.

Imagine users are assigned correctly, but purchase events are not recorded for treatment users. The analysis becomes worthless regardless of how sophisticated the statistical methods are. Data quality is as important as statistical rigor.

Statistical Significance Is Not Business Significance

This distinction is critical.

Imagine a company with 100 million users.

An experiment produces:

• Conversion increase: 0.02%
• p-value: 0.00001

The result is statistically significant. But does anyone care about a 0.02% improvement? Maybe not.

Now imagine:

• Conversion increase: 8%
• p-value: 0.07

The result is not statistically significant. Yet the potential business impact may justify further experimentation. Business value matters.