Before You Build a Model, Ask These Questions Or Watch Your ML Project Die in Production

Before You Build a Model, Ask These Questions Or Watch Your ML Project Die in Production

Photo by Mahdis Mousavi on Unsplash

Most ML projects don’t fail because of bad algorithms. They fail because nobody asked the right questions before writing a single line of code.

I’ve watched teams spend six months training a model, only to discover the business didn’t have a decision problem, they had a process problem. The model was irrelevant. Here’s how to avoid that.

The Real Definition of an ML Ready Problem

Start here: find where humans make repetitive, data informed decisions. That’s it. That’s the whole signal.

Not “where is the data?” Not “what model should we use?” Those come later. First, find the human in the loop who is making the same judgment call over and over and ask whether historical data from those decisions exists.

If yes, you have raw material. If no, stop. You don’t have an ML problem yet.

Run the ML Suitability Test First:

Before any Jira ticket gets created, before any data scientist gets looped in, run this fast filter:

1.Does a pattern exist? If there’s no underlying pattern in the data, no model will find one. Garbage in, garbage in forever.

2. Does historical data exist? ML is supervised learning in disguise. Without examples of past decisions and their outcomes, you’re building on air.

3. Is the logic too hard or impossible to program explicitly? If a rules engine can do it, use one. ML shines when explicit logic breaks down , fraud detection, demand forecasting, NLP.

4. Is being wrong acceptable?

This is the question nobody asks until it’s too late. If a wrong prediction causes a patient to miss a cancer diagnosis, that’s not a model problem , that’s a risk problem that needs a different solution.

Fail any one of these? Don’t launch ML.

Decompose the Business Problem Like a Product Manager

Once you’ve passed the suitability test, go deeper with the Business Problem Decomposition Canvas

What decision is being made badly or slowly? What would the perfect decision look like? What information does a human need to make it correctly? Can we capture that information as data?

Work through all four before touching a dataset. This is your product spec for the ML system.

Don’t Launch ML First. Start with heuristics. Not models.

Here’s the four step sequence they follow:

1. Choose heuristics build simple rules that approximate the decision
2. Don’t launch ML yet ship the heuristic baseline, measure it in production
3. Go to ML only when heuristics plateau when the ceiling is clear, the ceiling is your baseline
4. Launch your first model simple don’t start with a transformer when logistic regression hasn’t been tried

This is the discipline that separates teams that ship from teams that research. A heuristic in production beats a model in a notebook.

Check data readiness before committing:

You’ve got the problem scoped. Now audit your data. Check for:

Velocity — Is it available in Realtime, or is there lag?

Volume — Do you have enough examples to learn from?

Veracity — Is it trustworthy, or dirty and inconsistent?

Variety — Does it cover edge cases, or just the happy path?

Labels — Do you have ground truth? Who created it? Is it biased?

Leakage — Does any feature encode the answer you’re trying to predict?

Drift — Does the data distribution shift over time?

Data leakage and drift are the silent killers. A model that looks perfect in evaluation and fails in production has usually hit one of these two.

The Mindset Shift That Changes Everything

Stop asking “Can we build a model for this?”

Start asking: “Is there a repetitive human decision here, backed by data, where being occasionally wrong is acceptable?”

If yes -run the suitability test. Decompose the business problem. Start with heuristics. Audit your data. Then, and only then, train your first model.

This is how ML projects survive production. Everything else is just expensive experimentation.