Causal Analysis using Instrumental Variables to Reduce Employee Attrition

Does Salary Reduce Employee Attrition? A Causal Analysis Using Instrumental Variables in Python

Employee attrition is one of the most important challenges organizations face. Losing experienced employees increases recruitment costs, disrupts teams, and reduces productivity. A natural question for HR analytics is:

Does paying employees more reduce attrition?

At first glance, the answer seems obvious. Higher salaries should make employees less likely to leave. But analyzing this question with data is not straightforward. Simply running a regression of attrition on salary can produce biased results. In this article, we use an econometric technique called Instrumental Variables (IV) to estimate the causal effect of salary on employee attrition.

Nurse attrition in the U.S. healthcare system has reached historically high levels, making it a critical priority — particularly for hospitals. The dataset that we will explore includes employee- and company-level features suitable for supervised learning, unsupervised learning, and exploratory analytics. It contains an attrition indicator (whether an employee left), which can be used as the target variable in predictive models.

The data is synthetic and derived from the IBM Watson attrition dataset, with employee roles and departments adapted to reflect the healthcare context. Additionally, some outcome labels were modified to improve machine learning model performance.

The Problem with Simple Regression

Suppose we estimate the following model:

The attrition of the employee i as a function of their salary

where:

• Attrition = whether the employee left the company
• Salary = employee's monthly income

The coefficient β would appear to measure the effect of salary on attrition. However, there is a major issue: endogeneity. Salary is not randomly assigned. Many unobserved factors affect both salary and attrition, including:

• Employee ability
• Motivation
• Leadership potential
• Performance

These factors may also influence an employee’s decision to leave. This creates a correlation between salary and the error term, which biases the regression estimate.

Instrumental Variables: A Solution

To address this issue, we use Instrumental Variables (IV). The idea is to find a variable that:

1. Affects salary
2. Does not directly affect attrition

Such a variable is called an instrument. If we find a valid instrument, we can isolate the variation in salary that is independent of unobserved factors.

The Instrument: Job Level

In this analysis we use JobLevel as an instrument for salary. The intuition is simple:

• Employees with higher job levels typically earn higher salaries.
• Job level strongly predicts salary.

Formally:

This relationship represents the first stage of the instrumental variables approach. However, for the instrument to be valid, we must assume that job level affects attrition only through salary, once other variables are controlled for.

Dataset

The dataset that we are using is the **Kaggle Employee Attrition for Healthcare**. The dataset contains employee-level HR information, including:

• Age
• Monthly income
• Job level
• Years at the company
• Job satisfaction
• Work-life balance
• Overtime status
• Attrition indicator

Model Specification

We estimate the following causal model. Second stage:

Attrition as a function of other regressors

Because salary may be endogenous, we instrument it using JobLevel.

Python Implementation

Loading the data:

import pandas as pd
df = pd.read_csv('watson_healthcare_modified.csv')
df["Attrition"] = df["Attrition"].map({"Yes": 1, "No": 0})
df.head(3)

The model is estimated using the linearmodels package.

from linearmodels.iv import IV2SLS
formula = """
Attrition ~ 1 + Age + YearsAtCompany + WorkLifeBalance + JobSatisfaction + OverTime
+ [MonthlyIncome ~ JobLevel]
"""
model = IV2SLS.from_formula(formula, data=df_model).fit()
model.summary

First Stage Results

Before interpreting the IV results, we must verify that the instrument is strong. The first-stage regression produced the following diagnostics:

R-squared = 0.2067

Partial R-squared = 0.2039

Partial F-statistic = 261.52

P-value = 0.0000

A commonly used rule of thumb states that the first-stage F-statistic should be greater than 10 to avoid weak instrument problems. Our value of 261.52 is far above this threshold. This indicates that JobLevel is an extremely strong predictor of salary.

Correlation Between Job Level and Salary

Let us check the correlation of our variables:

Notice that EmployeeCount and StandardHours are constants

The correlation between the instrument and salary is corr(JobLevel, MonthlyIncome) = 0.9516. This confirms that job level almost completely determines salary in the dataset. From an econometric perspective, this strongly satisfies the relevance condition for an instrumental variable.

The causal interpretation is:

JobLevel → Salary → Attrition

We use job level only to extract the component of salary determined by the firm’s compensation structure, rather than by individual employee characteristics. This allows us to estimate the causal effect of salary on attrition.

Important Limitation

Instrumental variable models rely on a key assumption known as the exclusion restriction. The instrument must affect the outcome only through the endogenous variable. In our case this means:

JobLevel → Attrition (only through Salary)

In reality, job level may also affect attrition through other mechanisms:

• career opportunities
• job responsibility
• stress levels
• promotion expectations

Because of this, the instrument may not be perfectly exogenous.

This limitation should always be acknowledged when interpreting IV results.

Key Takeaways

This analysis illustrates several important concepts in causal inference:

1. Correlation does not imply causation: A simple regression of attrition on salary may be biased.
2. Endogeneity can arise from unobserved factors: Employee ability or motivation may influence both salary and attrition.
3. Instrumental variables can isolate causal variation: By using job level as an instrument, we focus on salary variation driven by organizational structure.
4. Instrument strength matters: Our first-stage F-statistic of 261.52 indicates an extremely strong instrument.

Final Thoughts

Instrumental variables are widely used in economics and increasingly in data science when randomized experiments are not available.

They provide a powerful framework for answering questions such as:

• Does education increase wages?
• Do taxes affect consumption?
• Does salary reduce employee attrition?

However, the validity of IV results always depends on the credibility of the instrument. Careful reasoning about the data-generating process is just as important as the statistical model.