Stop Guessing! The Ultimate Guide to A/B Testing in Python (With Real Code)

Don’t launch features based on gut feelings. Here is how to use Python and Statistics to make data-driven decisions that scale.

Have you ever launched a feature you were absolutely sure would work, only to watch your metrics crash and burn?

We’ve all been there. You are the Data Scientist at a hot new E-commerce startup. The Head of Product walks into your office and says, “I read a blog post that said Green buttons convert better than Blue ones. We need to change the ‘Buy Now’ button immediately.”

If you change it based on that hunch, you are gambling with company revenue. Not on my watch.

In this guide, I won’t just bore you with theory. I will walk you through a real-world scenario and show you, step-by-step, how to run a rigorous A/B test using Python.

We will cover:

1. 🧠 Formulating Hypotheses (Null vs. Alternative)
2. ⚙️ Setting up the Experiment
3. 🐍 Analyzing the results with Python
4. 📊 Making the final decision (Statistical Significance)

Let’s dive in.

1. The Scenario: Blue vs. Green

The Setup:

• Control Group (A): Sees the classic Blue “Buy Now” button.
• Treatment Group (B): Sees the new Green “Buy Now” button.
• Metric: Conversion Rate (Did they buy the item or not?).

We randomly show these buttons to 1,000 users each. At the end of the week, we have our data.

2. The Data

First, let’s simulate some real-world data. In the real world, you would pull this from your SQL database (e.g., Snowflake or BigQuery).

For this tutorial, we will simulate a dataset where Group B actually performs slightly better, but we need to prove it’s not just luck.

import numpy as np
import pandas as pd
from scipy import stats
import statsmodels.stats.api as sms

# Optional: For nice plots
import matplotlib.pyplot as plt
import seaborn as sns

# Set seed for reproducibility
np.random.seed(42)

# Simulation parameters
n_control = 1000
n_treatment = 1000

# Actual conversion rates (unknown to us during the test)
# Control (Blue) has a 12% conversion rate
# Treatment (Green) has a 14% conversion rate
control_converted = np.random.binomial(1, 0.12, n_control)
treatment_converted = np.random.binomial(1, 0.14, n_treatment)

# Create the DataFrame
df = pd.DataFrame({
    'group': ['Control'] * n_control + ['Treatment'] * n_treatment,
    'converted': list(control_converted) + list(treatment_converted)
})

print(df.head())

3. Exploratory Data Analysis (EDA)

Before jumping into complex math, let’s look at the raw numbers. This gives us a “sanity check.”

summary_df = df.groupby('group')['converted'].agg(['count', 'sum', 'mean'])
summary_df.columns = ['Total Samples', 'Conversions', 'Conversion Rate']
print(summary_df)

Output might look like this:

Okay, looking at the means, the Green button (14.2%) looks better than the Blue button (12.0%). But is this difference real? Or did we just get lucky with random traffic?

To answer this, we need Statistics.

4. The Hypothesis Test

This is the core of A/B testing. We need to define two hypotheses:

• Null Hypothesis (H0​): There is no difference between the Blue and Green buttons. (Any difference we see is due to random chance).
• Alternative Hypothesis (H1​): The Green button is different (better) than the Blue button.

We usually set a Significance Level (α) of 0.05. This means we are willing to accept a 5% risk of being wrong (False Positive).

The Math (Z-Test)

Since we are comparing two proportions (Conversion Rates), a Two-sample Z-test is the perfect tool.

Think of the Z-score as a measurement of “how many standard deviations away from the mean” our observation is. If the Z-score is huge, it means the difference is rare.

from statsmodels.stats.proportion import proportions_ztest, proportion_confint

# Get the counts
control_converted_count = summary_df.loc['Control', 'Conversions']
treatment_converted_count = summary_df.loc['Treatment', 'Conversions']

n_obs = [summary_df.loc['Control', 'Total Samples'], 
         summary_df.loc['Treatment', 'Total Samples']]

successes = [control_converted_count, treatment_converted_count]

# Perform the Z-test
z_stat, pval = proportions_ztest(count=successes, nobs=n_obs)

# Calculate Confidence Intervals (lower, upper)
(lower_con, lower_treat), (upper_con, upper_treat) = proportion_confint(successes, nobs=n_obs, alpha=0.05)

print(f"Z-Score: {z_stat:.4f}")
print(f"P-Value: {pval:.4f}")

5. Interpreting the Results

This is the moment of truth. Run the code above.

Scenario A: The P-Value is < 0.05 If your p-value is, say, 0.03, congratulations!

• Interpretation: The probability of seeing this result if the buttons were actually identical is only 3%.
• Action: Since 3% < 5% (our alpha), we Reject the Null Hypothesis.
• Business Result: Roll out the Green button to everyone!

Scenario B: The P-Value is > 0.05 If your p-value is 0.15:

• Interpretation: There is a 15% chance this difference is just random noise.
• Action: We Fail to Reject the Null Hypothesis.
• Business Result: We don’t have enough evidence. The Green button is not statistically better. Keep the Blue button.

Visualizing the Confidence Intervals

Humans are visual creatures. Let’s plot the conversion rates with their error bars (Confidence Intervals). If the error bars overlap significantly, the test is usually not significant.

plt.figure(figsize=(8,5))
x = ['Control (Blue)', 'Treatment (Green)']
y = [summary_df.loc['Control', 'Conversion Rate'], 
     summary_df.loc['Treatment', 'Conversion Rate']]

# Error bars represent the 95% Confidence Interval
yerr = [summary_df.loc['Control', 'Conversion Rate'] - lower_con, 
        summary_df.loc['Treatment', 'Conversion Rate'] - lower_treat]

plt.bar(x, y, yerr=yerr, capsize=5, color=['royalblue', 'forestgreen'])
plt.ylabel('Conversion Rate')
plt.title('Conversion Rate by Group (95% CI)')
plt.show()

Common Pitfalls to Avoid

If you want to impress your interviewer or boss, remember these three traps:

1. Peeking: Don’t check the result every day and stop the test as soon as you see a “winner.” This inflates false positives. Decide on a sample size before you start (use Power Analysis).
2. Sample Size: If you only test 50 users, your test is useless. Use Python’s NormalIndPower().solve_power() to calculate how many users you actually need.
3. Novelty Effect: Sometimes users click the Green button just because it’s new, not because it’s better. Watch the metric over time to ensure it stabilizes.

Conclusion

Data Science isn’t just about building complex neural networks. Often, the most value you can provide comes from fundamental statistics applied correctly.

Next time someone says “I think this color looks better,” you can reply with: “Let’s A/B test it.”

Now, go forth and experiment!

Did you find this helpful?

👏 Clap 50 times to help others find this article.

🔔 Follow me for more Python, Data Science, and Tech tutorials.

💬 Comment below if you have any questions!