A Practical Guide to A/B/n Testing: When One Challenger Is Not Enough

“Do not put all your eggs in one basket.” — Miguel de Cervantes

In experimentation, that means testing more than one idea at a time.

Introduction: The Limitation Nobody Talks About

You have been running A/B tests. You understand the framework — control group, treatment group, p-value, sample size. And you have had success with it.

But here is a scenario you will inevitably face. Your design team has not one new homepage variant. They have four. Your product manager says it will take months to test them sequentially. Can you test them all at once?

Or this: your data science team wants to find the optimal discount level — 5%, 10%, 15%, or 20%. Testing each against a control one at a time would take a year. What then?

A/B/n testing (also called multi-variant or multi-arm testing) is the answer. It extends the A/B framework to simultaneously test three or more variants against each other — finding the winner faster, with less total exposure, and with full statistical rigor if done correctly.

But A/B/n testing introduces a problem that standard A/B testing does not have: the more comparisons you make, the more likely you are to find a false winner. Understanding and controlling for this is the central challenge — and the central skill — of multi-arm experimentation.

Part 1: What A/B/n Testing Is and When to Use It

An A/B/n test runs n variants simultaneously:

Population
    ↓
Random split into n groups
    ↓           ↓            ↓           ↓
Control (A)  Variant B   Variant C   Variant D
    ↓           ↓            ↓           ↓
         Measure primary metric for all groups
                    ↓
        Find the best-performing variant

The n can be as small as 3 (one control, two challengers) or as large as your traffic and statistical budget allow.

When A/B/n is the right choice:

Use A/B/n when:
  → You have multiple competing hypotheses worth testing simultaneously
  → Sequential testing would take too long given your traffic volume
  → You want to find an optimal level of a continuous variable (pricing, discount, threshold)
  → The cost of holding back a better version is high (opportunity cost)
Use standard A/B when:
  → You have one clear hypothesis
  → Traffic is limited (more variants = more diluted exposure per arm)
  → Your primary metric is rare (needs concentrated power per comparison)

A key trade-off: splitting traffic n ways means each arm gets 1/n of the total. With the same total traffic, each comparison has less power than a two-arm test. You are trading experiment duration against the number of hypotheses tested.

Part 2: The Multiple Comparisons Problem — Why A/B/n Is Harder

This is the core statistical challenge of A/B/n testing and the reason it must be handled differently from running multiple A/B tests in sequence.

The problem:

With α = 0.05 as your significance threshold, each individual test has a 5% chance of a false positive under the null hypothesis. With multiple comparisons, these errors accumulate.

Family-Wise Error Rate (FWER):
The probability of at least one false positive across all comparisons.
FWER = 1 − (1 − α)^k
Where k = number of pairwise comparisons
For n = 5 variants (A, B, C, D, E):
  k = C(5,2) = 5!/(2! × 3!) = 10 pairwise comparisons
  FWER = 1 − (1 − 0.05)^10 = 1 − 0.9510 = 40.1%

Testing five variants with no correction means you have a 40% chance of declaring a false winner — even when none of the variants is actually better than control. That is not a test. That is a lottery.

Part 3: Correction Methods — Controlling the Error Rate

Method 1: Bonferroni Correction

The simplest and most conservative approach. Divide the significance threshold by the number of comparisons:

Adjusted α_bonferroni = α / k
For k = 10 comparisons, α = 0.05:
  Adjusted α = 0.05 / 10 = 0.005
Reject H₀ for comparison i only if p_i < 0.005

Advantage: Simple, guaranteed control of FWER.

Disadvantage: Very conservative — it assumes all tests are independent. When tests share a control group (all compared against A), they are correlated, and Bonferroni over-penalises. This increases the Type II error rate (more false negatives — real effects missed).

Method 2: Holm-Bonferroni (Step-Down)

Less conservative than Bonferroni while still controlling FWER. Rank p-values from smallest to largest, then apply sequentially:

Sort p-values: p(1) ≤ p(2) ≤ ... ≤ p(k)
Reject H₀(i) if p(j) ≤ α / (k − j + 1) for all j ≤ i
Step-by-step:
  Compare p(1) against α/k
  Compare p(2) against α/(k−1)
  Compare p(3) against α/(k−2)
  ... and so on
Stop rejecting at the first non-significant comparison.

Holm is uniformly more powerful than Bonferroni — it should be preferred whenever you are testing multiple hypotheses.

Method 3: Benjamini-Hochberg (FDR Control)

Instead of controlling FWER (probability of any false positive), control the False Discovery Rate (FDR) — the expected proportion of discoveries that are false.

Sort p-values: p(1) ≤ p(2) ≤ ... ≤ p(k)
For each i, compute: q(i) = p(i) × k / i
Reject all H₀(j) where p(j) ≤ q* = α × i* / k
Where i* = largest i such that p(i) ≤ α × i/k

Example with 5 comparisons, α = 0.05:

Sorted p-values: 0.001, 0.021, 0.044, 0.089, 0.142

BH thresholds:  0.05×1/5, 0.05×2/5, 0.05×3/5, 0.05×4/5, 0.05×5/5
              = 0.010,    0.020,    0.030,    0.040,    0.050

Compare each p(i) to its threshold:
  0.001 ≤ 0.010 → Reject (significant)
  0.021 > 0.020 → Accept
  0.044 > 0.030 → Accept
  0.089 > 0.040 → Accept
  0.142 > 0.050 → Accept

Result: only the first variant is declared a winner.

BH is more powerful than Bonferroni and Holm — it finds more true positives — at the cost of allowing a small fraction of false discoveries. It is the right choice when you have many variants and are willing to accept that a small proportion of declared winners might not be real.

Part 4: Sample Size for A/B/n Tests

With n arms, sample size per arm is determined by the desired pairwise power after correction.

Using Bonferroni correction, replace α with the corrected threshold:

n_per_arm = 2 × (z_{α_adj/2} + z_β)² × p(1−p) / δ²
Where α_adj = α / k (Bonferroni adjusted level)
      k     = number of pairwise comparisons = C(n,2) = n(n−1)/2
      δ     = minimum detectable effect per comparison
      p     = baseline conversion rate

Example: 4 variants (A, B, C, D), k = 6 comparisons, α_adj = 0.05/6 = 0.0083
  z_{α_adj/2} = z_{0.00417} ≈ 2.64  (vs 1.96 for standard A/B)
n_per_arm = 2 × (2.64 + 0.84)² × 0.10 × 0.90 / 0.01²
          = 2 × 12.11 × 0.09 / 0.0001
          = 21,803 users per arm
          = 87,212 total users (4 arms)

Compare to the standard A/B test: 14,112 per arm (28,224 total). The multi-arm test needs 3× more total traffic — a cost that is justified when it replaces four sequential experiments.

Part 5: Bayesian Multi-Arm Testing — A More Natural Framework

Bayesian approaches handle multiple comparisons more naturally. Instead of correcting p-values, you compute the probability that each variant is the best across all arms.

For each variant i, model conversions as:
  successes_i ~ Binomial(n_i, p_i)
Place a Beta prior on each conversion rate:
  p_i ~ Beta(α_prior, β_prior)
Posterior after data:
  p_i | data ~ Beta(α_prior + successes_i, β_prior + failures_i)
Probability that variant i is the best:
  P(pᵢ = max across all arms) = estimated by Monte Carlo sampling
Sample K = 10,000 draws from each posterior:
  For each draw, record which variant has the highest sampled value
  P(variant i is best) ≈ (draws where i is max) / K

The output is direct and interpretable: “Variant C has a 73.2% probability of being the best-performing variant across all four tested versions.”

No threshold. No correction. Just probabilities that update as data accumulates.

The Thompson Sampling extension uses these posteriors to dynamically allocate more traffic to better-performing arms mid-experiment — the basis of multi-armed bandit approaches. This reduces the cost of running the experiment itself, since users are not held in poor-performing arms longer than necessary.

Part 6: Analysis and Reporting — What to Present After the Test

Always report:

For each variant vs control:
  Observed conversion rate and 95% CI
  Absolute lift and relative lift
  Corrected p-value (or posterior probability if Bayesian)
  Statistical significance after correction
  Practical significance (is the lift large enough to matter?)

Relative lift and its confidence interval:

Relative lift = (p̂_treatment − p̂_control) / p̂_control × 100%
Example:
  Control: 10.0%  Treatment: 11.5%
  Relative lift = (11.5 − 10.0) / 10.0 × 100% = +15%

What to do when no variant wins:

A null result is a result. If no variant beats control after adequate sample size, the conclusion is: “None of the tested variants produced a detectable improvement.” This is valuable. It prevents shipping changes that do not work, and it informs future hypotheses.

Conclusion: Test More, Decide Faster, Control the Errors

A/B/n testing is not just A/B testing with more rows. It introduces a genuine statistical challenge — the multiple comparisons problem — that must be handled deliberately. Ignore it and your false positive rate spirals out of control. Handle it correctly with Holm-Bonferroni or BH correction, and you gain the ability to test multiple hypotheses simultaneously without sacrificing rigour.

The practical recipe: define one primary metric, pre-specify all variants before launch, calculate sample size using the corrected α, apply Holm-Bonferroni or Benjamini-Hochberg correction at analysis, and report both statistical and practical significance for every comparison.

More variants. Same rigour. Faster answers.

Found this useful? Follow for more applied data science and experimentation guides. How many variants does your next experiment have?