Day 5: Automating ML Workflows with Makefiles

Welcome to Day 5 of the 100 Days of MLOps challenge!

Over the past four days, we have isolated our environments, configured our workspaces, pinned our dependencies, and structured our project directories. However, executing all these steps manually every time you clone a repo or start a new pipeline is tedious and highly prone to human error.

In MLOps, automation is everything. Today, we bring order to the chaos by orchestrating our common tasks using a classic, battle-tested tool: The Makefile.

Let’s look at how to fix a broken Makefile for the xFusionCorp Industries data science team.

The Challenge

The ML team has a draft Makefile located at /root/code/fraud-detection/Makefile, but running make all throws errors and fails to complete. Our goal is to bring this file in line with the team's standard by defining exactly six targets.

Our Objectives:

1. **setup**: Create a virtual environment at mlops-venv/ and install dependencies.
2. **data**: Run the data processing script.
3. **train**: Run the model training script.
4. **test**: Execute the test suite using pytest.
5. **clean**: Remove all __pycache__ directories, clear .pytest_cache, and empty the models/ directory.
6. **all**: Run setup, data, train, and test sequentially.

Crucially, all targets must be declared as .PHONY to prevent file name conflicts, and all recipes must be indented with a real tab character.

Step-by-Step Solution

Step 1: Observe the Failure

First, navigate to the project directory and run the draft to see what goes wrong.

cd /root/code/fraud-detection/
make all

You will likely see errors about missing targets, missing separators (a classic tab indentation issue), or commands failing.

Step 2: Edit the Makefile

Open the Makefile in your preferred terminal text editor.

vi Makefile

Step 3: Write the Corrected Makefile

Here is the complete, corrected Makefile.

⚠️ CRITICAL WARNING: Copy-pasting from web pages often converts tabs to spaces. You must ensure that the indentation before every command (like python3 -m venv...) is a single TAB character, not spaces. Make will reject any recipe indented with spaces.

.PHONY: setup data train test clean all

setup:
    python3 -m venv mlops-venv/
    mlops-venv/bin/pip install -r requirements.txt

data:
    python src/data/process_data.py

train:
    python src/models/train.py

test:
    pytest tests/

clean:
    find . -type d -name "__pycache__" -exec rm -rf {} +
    rm -rf .pytest_cache
    rm -rf models/*

all: setup data train test

Step 4: Understanding the Code

Let’s break down why this specific configuration is the standard for MLOps:

• **.PHONY:** By default, make looks for a file matching the target name. If you happen to have a folder named data (which we created in Day 4!) and run make data, make will see the folder, assume the target is "up to date," and do nothing. Declaring .PHONY: data tells make that this is a command, not a file.
• The setup Target: Notice that we explicitly use the pip binary inside the newly created mlops-venv/ directory (mlops-venv/bin/pip). Because make runs every line in a separate subshell, simply running source mlops-venv/bin/activate on one line won't keep the environment active for the next line.
• The clean Target: ML projects generate a lot of clutter. The find command recursively hunts down every __pycache__ directory and deletes it safely, while we use rm -rf to clear out old test caches and stale model artifacts.
• The all Target: This acts as our master pipeline. By setting its dependencies to setup data train test, we define a strict execution order.

Step 5: Test the Automation

Save the file and run the master command:

make all

You should see your terminal light up as it sequentially builds the environment, installs packages, processes the data, trains the model, and runs the tests — all without any further human intervention.

Why This Matters

A well-structured Makefile is the connective tissue between a data scientist's laptop and a CI/CD pipeline. When your code is pushed to a remote repository, the CI runner doesn't need to know the complex series of python commands required to build your project; it just needs to run make test.

By standardizing our commands, we have significantly reduced the onboarding time for new engineers and paved the way for automated deployments.

See you on Day 6 as we continue to level up our MLOps game!

Tags: #MLOps #Makefiles #Automation #DevOps #Python #DataScience