Causal Inference with DoWhy (2): Linear Regression

Photo by razi pouri on Unsplash

In the previous article, we went through the basics of causal inference, causal diagrams, and how to identify the covariates we need to control for in order to isolate the causal impact of a treatment on an outcome.

In this article, we will look at how linear regression can be used as a causal adjustment method. We will also briefly discuss why, despite its simplicity, linear regression has important limitations, and why matching methods are often preferred in practice.

Why can randomisation isolate causal effects?

Let’s go back to the example we used last time. We want to quantify the impact of a personalised homepage on the time spent on our product.

One way to do this is by running an A/B test, which is a form of a randomised controlled trial (RCT).

The key reason an A/B test can isolate the causal effect is randomisation. When treatment is assigned randomly, it becomes independent of users’ background characteristics and behaviours. As a result, both the treatment group and the control group are comparable on average.

In our example, this means:

• The proportion of users coming from direct traffic is the same in both groups.
• The proportion of mobile users is the same.
• The distribution of historical time spent is the same.

Because these confounding variables are balanced by design, any difference we observe in time spent can be attributed to the personalised homepage itself, rather than to pre-existing differences between users.

Controlling for covariates without randomisation

In real-world settings, we often cannot randomise. Instead, we observe data where treatment assignment is influenced by user behaviour or business rules.

From the previous article, we identified three covariates that need to be controlled for:

• traffic_direct
• is_mobile
• prior_week_minutes

One of the simplest ways to control for these variables is to use linear regression.

Linear regression as backdoor adjustment

Below is the DoWhy code used to estimate the causal effect using linear regression. All code in this post can be found in the accompanying Kaggle notebook.

# Linear regression adjustment (backdoor)
estimate_lr = model.estimate_effect(
    identified_estimand,
    method_name="backdoor.linear_regression",
    test_significance=True
)
print("[Estimate] Backdoor linear regression:")
print(estimate_lr)
print("ATE estimate:", round(estimate_lr.value, 3))

Output:

[Estimate] Backdoor linear regression:
*** Causal Estimate ***

## Identified estimand
Estimand type: EstimandType.NONPARAMETRIC_ATE

### Estimand : 1
Estimand name: backdoor
Estimand expression:
           d                                                                   ↪
────────────────────────(E[weekly_minutes|prior_week_minutes,is_mobile,traffic ↪
d[personalized_homepage]                                                       ↪

↪          
↪ _direct])
↪          
Estimand assumption 1, Unconfoundedness: If U→{personalized_homepage} and U→weekly_minutes then P(weekly_minutes|personalized_homepage,prior_week_minutes,is_mobile,traffic_direct,U) = P(weekly_minutes|personalized_homepage,prior_week_minutes,is_mobile,traffic_direct)

## Realized estimand
b: weekly_minutes~personalized_homepage+prior_week_minutes+is_mobile+traffic_direct
Target units: ate

## Estimate
Mean value: 7.561454335936393
p-value: [0.]

ATE estimate: 7.561

The output shows an estimated average treatment effect (ATE) of about 7.56 minutes.

The key line in the output specifies the regression formula:

weekly_minutes ~ personalized_homepage + prior_week_minutes + is_mobile + traffic_direct

In equation form, the model is:

weekly_minutes = β₀

• β₁ · personalized_homepage
• β₂ · is_mobile
• β₃ · traffic_direct
• β₄ · prior_week_minutes
• ε

Here, the β’s are the coefficients (in statistics terms) or parameters (in machine learning terms).

How to interpret the statistical test

test_significance=True makes DoWhy compute a p-value for the treatment effect coefficient, to test if 𝛽1 = 0. In this case, the p-value is extremely small, so we reject the null hypothesis that β₁ = 0.

However, significance does not equal causal validity. It does not mean that the causal relationship can be established by this statistical test. Causal validity comes from the DAG assumptions.

How to interpret the treatment coefficient

The most important coefficient is β₁, which corresponds to the treatment variable personalized_homepage.

Its interpretation is:

Holding all other variables constant, changing personalized_homepage by one unit leads to a β₁ change in weekly_minutes.

In plain English, if β₁ is 0.5, then enabling the personalised homepage increases weekly time spent by 0.5 minutes on average, after accounting for user type (in terms of their main referrer to our product), device, and historical behaviour.

This is exactly what we want from a causal estimate.

Why linear regression is often not enough

Despite its simplicity and popularity, linear regression has several important limitations when used for causal inference.

1. Strong functional form assumptions

Linear regression assumes a linear relationship between each variable and the outcome. In real life, user behaviour is rarely linear. For example, prior usage may have diminishing returns, or treatment effects may vary across users.

2. Extrapolation risk

Regression estimates effects even in regions of the data where treated and untreated users barely overlap. This means it may rely on extrapolation rather than actual comparisons between similar users.

3. One-size-fits-all treatment effect

A single coefficient assumes the treatment effect is the same for everyone. In practice, the effect of a personalised homepage may differ dramatically between new users and power users.

4. Less intuitive “apples-to-apples” comparison Regression adjusts differences mathematically, but it does not explicitly ensure that treated users are being compared with similar untreated users. This can make the causal story harder to reason about and debug.

Because of these limitations, linear regression is often best viewed as a baseline causal method, rather than the final answer.

What’s next

Matching methods aim to more closely mimic a randomised experiment by explicitly comparing treated users with similar untreated users, making the comparison more “apples to apples”.

In the next article, we will explore matching in detail and see how it improves upon linear regression in causal inference.