From Coursework to Causal Inference: Why “Just Accuracy” Stopped Being Enough

As an undergraduate student specializing in Artificial Intelligence and Machine Learning, I’ve been passionate about building intelligent systems that not only perform well, but also make sense in real-world settings. When getting started with machine learning, there was a basic, simple goal: increase the “accuracy” of the model. The higher the validation score, the better the model, and that often was the end of the story.

Over the past year, that stopped feeling satisfying, moreover it stopped feeling enough. The more real‑world style problems were explored, the more uncomfortable it became to treat models like ‘black boxes’-especially as small changes in data or environment made them behave unpredictably. That discomfort motivated me to explore into something that could help me understand models. That exploration made me crash into a concept called “Causal Inference” and eventually led me to building a causal analysis framework using libraries like DoWhy and EconML.

Why “just accuracy” started to feel wrong

Most standard ML setups reward one thing: performance on a unseen test set.

This is powerful, but it hides a big assumption: data stationarity. When data shifts, new user groups arrive, or policies change, a model that learned only surface‑level patterns can fail silently.

At some point, questions started to change from “How do I push this metric higher?” to:

• “What patterns is the model actually using?”
• “Would this still hold if certain conditions changed?”
• “Is this a real cause, or just a coincidence in the data?”

The heart of causal inference lies in asking what changes when we intervene, moving beyond simply asking what tends to happen together.

Correlation vs causation (the intuitive way)

A simple way to see the difference is to think about everyday examples.

• If people carrying umbrellas tend to wear raincoats, that is a correlation (they both happen during rain).
• If wearing a raincoat keeps you dry, that is a causal effect (the raincoat actively changes the outcome).

A predictive model trained on weather data could easily learn the "umbrella -> raincoat" pattern. But if everyone suddenly started staying inside whenever it rained, those correlations may weaken or reverse. The model was never reasoning about why people stayed dry—it was only exploiting patterns in the data.

Causal ML tries to go a level deeper, asking “If we change X, how does Y respond?” and making explicit what assumptions are needed to answer that question.

Fig 1.1: A simple flowchart to explain the difference between Traditional ML and Causal ML

A small yet significant adjustment of mindset

As an undergraduate this transition appeared gradually. Instead of thinking “I have a dataset; which model gives the highest score?”, the question became “What is the real question here? Am I trying to predict outcomes, or trying to understand the impact of some action or policy?”

This sounds trivial, but it completely changes how a project is designed.

• You begin by identifying the action you want to study, the result you care about, and other factors that influence both.
• You become more careful about where the data come from and the biases baked into it.
• You accept that sometimes, with the data you have, a question is simply unanswerable, no matter how fancy the model is.

To make this mindset practical, a dedicated project was developed: a reproducible Causal Analysis Framework, CausalBench, which wraps common causal tools in a consistent, experiment‑friendly pipeline.

You can find the project here: **CausalBench**

Building CausalBench

The CausalBench project grew out of a simple need — a place to systematically run causal experiments instead of scattered notebooks. The project includes the ability to:

• Load and validate diverse datasets: Access benchmark examples like IHDP, Twins, and Lalonde, among many others.
• Run various estimation methods: Utilize standard techniques such as IPW, propensity score matching, doubly robust estimators, and modern ML-based approaches like DML and DRLearner.
• Perform causal discovery: Employ algorithms (PC, FCI, NOTEARS) to automatically learn causal relationships and graph structures from the data.
• Use an interactive dashboard: Upload your own data via a user-friendly Streamlit interface and run full causal analyses without writing any code.

Under the hood, the project fixes random seeds, logs runs, runs tests, and generates summary reports so that experiments are reproducible and easier to extend. Those details are exactly what make a project feel research‑grade rather than just a quick demo.

How DoWhy changed how problems were framed

DoWhy is an open-source Python library for causal inference that provides a four-step process to help users ask and answer “what if” and “why” questions using data. The package DoWhy is itself referred to as a framework for causal reasoning or an end-to-end library for causal inference.

What makes DoWhy special is not just its algorithms, but the structured thinking it imposes:

1. Modeling: Define the problem visually using a causal graph to map which variables might influence others.
2. Identification: Based on the graph, determine if the effect you care about can actually be measured using your observed data.
3. Estimation: Calculate the effect using one or more statistical or ML-based estimators.
4. Refutation: Test the validity of your estimate through sensitivity analyses and alternative assumptions.

By embedding DoWhy into the framework, these rigorous steps became natural. Before jumping into writing code for an estimator, users were required to stop and address critical questions:

• “Exactly what treatment and outcome am I defining?”
• “Which confounding variables should I include, and what important factors might be missing?”
• “If I find an effect, how can I test how reliable or fragile it is?”

That crucial fourth step — actively trying to break your own result — is where the mindset truly shifts toward robust, research-oriented practice.

You can find more official resources via the DoWhy documentation or it’s **GitHub repository**.

Where EconML fits in

While DoWhy provides the structure for the causal workflow and robustness checks, the framework incorporates EconML to bring in powerful machine learning models. EconML is used to estimate how the effects of an action can vary across different individuals.

Wrappers within the framework for methods like Double Machine Learning (DML) and Doubly Robust Learner (DRLearner) make it easy to ask practical questions such as:

• “On average, does this job training program actually increase earnings?”
• “Which specific subgroup experiences the strongest or weakest positive outcome?”

These are known as heterogeneous treatment effects: the vital concept that a single treatment or policy does not help all individuals universally or equally.

The ability to explore these nuanced questions within the same consistent pipeline, using real benchmark datasets, helps move causal ML from an abstract theory into a practical tool relevant to real-world policy decisions.

Useful links include the EconML GitHub repository and the official **documentation**.

How this shapes what comes next

After spending time focusing on Causal ML and building a dedicated framework around it, I find it difficult to view machine learning projects purely as “prediction problems.”

In many domains, such as cybersecurity, smart campus management, and AI for education, the most interesting and impactful questions revolve around interventions, not just forecasting:

• “If we change this security policy, what happens to the rate of network attacks or staff alert fatigue?”
• “If we adjust an energy-saving rule, which specific buildings actually benefit, and which do not?”
• “If we recommend a certain study plan, which students improve their outcomes the most?”

Tools like DoWhy and EconML support the rigorous causal thinking required to answer these questions in a structured way, while reproducible pipelines ensure that our results remain honest, transparent, and extensible.

This post serves as a starting point, explaining why simply optimizing for “prediction accuracy” no longer feels sufficient in these contexts. Future posts will dive deeper into specific experiments, benchmarks, and applications, linking these causal ideas with concrete systems and code found within the main CausalBench repository.