Causal Inference with DoWhy (6): Diagnostics with Standardised Mean Difference (SMD)

With libraries like DoWhy, it’s easy to build a causal model, and just as easy to unknowingly build a bad one. (Photo by Vedrana Filipović on Unsplash)

If you’ve followed Parts 1 to 5 of this series, you should already be able to run causal inference with DoWhy in practice. The concepts can feel abstract at first, but the tooling makes it almost too easy to get results.

And that’s exactly the problem.

All code in this post can be found in the accompanying Kaggle notebook.

Why Diagnostics Matter

At first glance, diagnostics might feel like overkill.

You might think:

Do we really need this? This sounds like something from clinical trials or academic research.

That’s why you should consider doing it. Diagnostics is the step where you stop treating causal inference like a tool, and start treating it like an investigation.

It’s the step where you move from “I ran the model” to “I understand why I should trust (or not trust) the result”.

Diagnostics vs Refutation Tests

In the previous article, we focused on refutation tests.

The idea was simple: Try to break your model and see if it still holds.

You might shuffle variables, inject noise, or simulate alternative scenarios. If your conclusions stay consistent, your model is likely robust.

Diagnostics, however, asks a more fundamental question: Was the comparison valid in the first place?

Instead of breaking the model, we inspect:

• The data distributions
• The similarity between treatment and control groups
• Whether key causal assumptions are violated

Overlap Check

When we perform propensity score matching, we estimate the probability that each subject receives treatment. This is called the propensity score.

Some users will naturally have higher probabilities, others lower, depending on their features.

In a perfect Randomized Controlled Trial (RCT):

• Every subject has a 50% chance of receiving treatment
• Propensity score = 0.5 for everyone

In observational data, this is never true. But we try to approximate this randomness.

The Positivity Assumption

The positivity assumption says:

Every type of subject should have a chance of being treated and a chance of not being treated.

Why does this matter?

Imagine this scenario:

• Users above age 65 are never shown a personalized homepage
• Younger users are always shown it

Can we compare these groups fairly? No, because age and treatment are completely entangled. There is no overlap:

• You will never find a 65-year-old in the treatment group
• You will never find a 25-year-old in the control group

So any difference in the outcome (target variable) could be due to:

• Age
• Treatment
• Or both

And you have no way to separate them. That’s why overlap is everything.

Implementation in Python

We first estimate propensity scores using a simple logistic regression:

# Fit a propensity model ourselves (for diagnostics)
X = df[
    [
        "tenure_days", "prior_week_minutes", "is_mobile",
        "age", "traffic_search", "traffic_social", "traffic_direct"
    ]
]
t = df["personalized_homepage"]

X_train, X_test, t_train, t_test = train_test_split(X, t, test_size=0.3, random_state=RANDOM_SEED, stratify=t)

ps_model = LogisticRegression(max_iter=2000)
ps_model.fit(X_train, t_train)
propensity_scores = ps_model.predict_proba(X)[:, 1]
df["propensity"] = propensity_scores

print("\nPropensity summary (treated vs control):")
print(df.groupby("personalized_homepage")["propensity"].describe()[["mean", "std", "min", "max"]])

# Overlap check
plt.figure()
df[df.personalized_homepage == 1]["propensity"].hist(bins=30, alpha=0.6, label="Treated")
df[df.personalized_homepage == 0]["propensity"].hist(bins=30, alpha=0.6, label="Control")
plt.title("Propensity score overlap diagnostic")
plt.xlabel("Estimated P(Treatment=1 | confounders)")
plt.ylabel("Count")
plt.legend()
plt.show()

Output:

Propensity summary (treated vs control):
                           mean       std       min       max
personalized_homepage                                        
0                      0.435297  0.151712  0.182384  0.998946
1                      0.586289  0.192902  0.199197  1.000000

Then we compare distributions:

The treatment group (blue bars) should have higher propensity on average. This makes sense. More importantly, there must be a region where both groups overlap. These are the people in the treatment and control groups who are actually comparable.

If there is little or no overlap, like the example below:

(A Crash Course in Causality: Inferring Causal Effects from Observational Data)

• Treated users all have propensity scores between 0.3–1.0
• Control users all have scores between 0.0–0.8

For a treated user with score 0.9, the closest control might be 0.8. That’s not necessarily “close”.

This is also called extrapolation, a common concept in data science where we estimate values beyond the observed data.

Each propensity score value represents a certain type of user. If a certain score (e.g. 0.9) only has users in the treatment group, we are asking: “What would happen if this type of user (who is always treated) were not treated?”

But the data never shows you that scenario. So your model has to guess.

Standardised Mean Difference (SMD)

We want to know we can effectively match the two groups. We also want to know whether the two groups are actually similar after matching. This is where SMD comes in. We compare the difference of means per covariate between the two groups. For example, are the mean ages similar between two groups after matching?

The formula looks intimidating. But the concept is super simple.

1. We can calculate the SMD per covariate.
2. Numerator → difference in means. E.g. mean age of treatment group — mean age of control group.
3. If the means between two groups are similar, this means the two matched groups are similar and comparable. But even if the means are far apart, that may simply be due to the higher variability within that covariate. We need to account for this, hence the denominator.

• Denominator → average variance between the two groups. σ represents the standard deviation. σ squared represents the variance. The denominator takes the average of the variances, then applies the square root to convert it back to a standard deviation, hence the name Standardised Mean Difference.
• In short, SMD finds “How different are the two groups, relative to their variability?”

1. Before Matching: Expect Imbalance

Before matching, we compute SMD like this:

def smd(x_treated, x_control):
    """Compute standardized mean difference."""
    mean_t = np.mean(x_treated)
    mean_c = np.mean(x_control)
    var_t = np.var(x_treated, ddof=1)
    var_c = np.var(x_control, ddof=1)
    return (mean_t - mean_c) / np.sqrt((var_t + var_c) / 2)

# Pre-treatment SMD (before any adjustment)
covariates = [
    "tenure_days",
    "prior_week_minutes",
    "is_mobile",
    "age",
    "traffic_search",
    "traffic_social",
    "traffic_direct",
]

smd_pre = {}

for col in covariates:
    smd_pre[col] = smd(
        df.loc[df.personalized_homepage == 1, col],
        df.loc[df.personalized_homepage == 0, col],
    )

smd_pre_df = (
    pd.DataFrame.from_dict(smd_pre, orient="index", columns=["SMD_pre"])
    .sort_values("SMD_pre", key=np.abs, ascending=False)
)

print("\nPre-treatment SMD:")
display(smd_pre_df)

Output

At this stage:

• Large SMD values are expected
• This is because treatment assignment was not random. Certain types of users are more likely to receive treatment.

2. Matching Step

We then perform nearest-neighbour matching using propensity scores.

# Step 1: Separate treated and control units
treated = df[df.personalized_homepage == 1].copy()
control = df[df.personalized_homepage == 0].copy()

# Step 2: Nearest-neighbor matching on propensity score
from sklearn.neighbors import NearestNeighbors

nn = NearestNeighbors(n_neighbors=1)
nn.fit(control[["propensity"]])

distances, indices = nn.kneighbors(treated[["propensity"]])

# Step 3: Apply caliper (distance threshold)
caliper = 0.05
mask = distances.flatten() <= caliper

# Keep only well-matched treated units
treated_matched = treated.loc[mask].copy()

# Select matched control units
matched_control = control.iloc[indices.flatten()].copy()
matched_control = matched_control.loc[mask].copy()

# Optional: align indices (helps with diagnostics later)
matched_control.index = treated_matched.index

# Step 4: Construct matched dataset
df_psm = pd.concat([treated_matched, matched_control], axis=0)

For each treated user, the algorithm matches a control user with the closest propensity score. The caliper acts as a threshold to ensure we only accept matches that are close enough. This avoids poor matches that would distort results, like the scenario we discussed in the overlap check.

After Matching: What Good Looks Like

smd_psm = {}

for col in covariates:
    smd_psm[col] = smd(
        df_psm.loc[df_psm.personalized_homepage == 1, col],
        df_psm.loc[df_psm.personalized_homepage == 0, col],
    )

smd_psm_df = (
    pd.DataFrame.from_dict(smd_psm, orient="index", columns=["SMD_post_PSM"])
    .sort_values("SMD_post_PSM", key=np.abs, ascending=False)
)

print("\nPost-PSM SMD:")
display(smd_psm_df)

SMD values are very close to 0 after matching, meaning there is not much difference between the two groups.

Now we recompute SMD. If matching worked:

• Covariates should look very similar, indicated by low SMD values
• The dataset should resemble a pseudo-RCT

Rule of Thumb:

• SMD < 0.1 → Good balance
• 0.1–0.2 → Acceptable, but watch it
• > 0.2 → Problematic imbalance

Final Thought

If there is one thing to take away from this series, it is this: running a model is easy, but trusting it is the hard part.

Causal inference forces us to confront something traditional machine learning often overlooks. Most models are built on correlation. They predict well, but they don’t answer the question we actually care about: what happens if we intervene? As data-driven decisions are fuelled by the rise of generative AI, this shift from correlation to causation is becoming essential.

Causal inference is not just a toolkit, but a way of thinking. And once you start thinking this way, it’s hard to go back. If you’d like to go deeper, I highly recommend The Book of Why: The New Science of Cause and Effect and A Crash Course in Causality: Inferring Causal Effects from Observational Data.

And with that, this series comes to an end. But hopefully, a different way of thinking begins.

[embed]Causal Inference with DoWhy (1): Chains, Forks, and Colliders Correlation can fool us. Learn how chains, forks, and colliders help uncover real causal effects using DoWhy.medium.com

[embed]Causal Inference with DoWhy (2): Linear Regression How linear regression estimates causal effects, why it works, and where it falls short compared with matching methods.medium.com

[embed]Causal Inference with DoWhy (3): Propensity Score Matching In the previous two articles, we talked about the basics of causal inference and the simplest way to control for…medium.com

[embed]Causal Inference with DoWhy (4): Inverse Probability of Treatment Weighting (IPTW) In the previous article, I talked about propensity score matching. The propensity score is the probability that a…medium.com

[embed]Causal Inference with DoWhy (5): Refutation Tests Refutation asks a simple but powerful question:medium.com