The Model Is Only Two Lines. Data Science Is Everything Before That.

Anyone can call .fit() and .predict(). The hard part is making the data worth modeling in the first place.

Photo by James Harrison on Unsplash

Every machine learning tutorial follows roughly the same rhythm.

You load a dataset. You split it into training and test sets. You pick a model, pass in your features, and then the machine learning happens:

model.fit(X_train, y_train)
model.predict(X_test)

The notebook runs. An accuracy score appears. It feels like something real just happened.

In a tutorial, it did. In real work, those two lines are almost never the hard part. They are the end of a long chain of decisions that the tutorial quietly made for you before the first cell ran.

The hard part is everything that had to be true before those two lines could mean anything.

Tutorials Start After the Hardest Work Is Done

I am not criticizing tutorials here. They have their place. They help illustrate how modeling would look once all the pieces are in order.

It’s precisely this where the beginner lacks experience. Once the dataset for the tutorial lands in your hands, it is already cleaned up. You have your target column, and it already makes sense. You have your features already in order. Your rows represent things exactly as they should be represented. You have the problem formulated into something learnable by a model.

None of this is present in an actual project.

You are shown the model through the tutorial. But this is not where it starts.

The First Hard Part Is the Question

But before doing anything with the data, there is a fundamental question that must first be addressed: What is the problem you are trying to solve?

At first glance, this seems rather obvious. Nothing could be further from the truth.

Is the goal to predict customer churn, or to find out which customers are still valuable enough to be saved? Is it to predict future demand, or to provide an ordering recommendation to inventory managers with a strict time frame? Is the task fraud detection or fraud review support?

There is a lot at stake here because the nature of the problem determines all the following factors: the target definition, metric, allowable error rate, and ultimately what “success” looks like for the problem. There are no models that can salvage poorly specified questions. A model can successfully do something but fail at solving your problem, which is always quite a discouraging realization to arrive at after weeks of effort.

A model will simply learn the problem that you presented to it. If that problem is flawed, then so is the learning.

The Target Is Not Just a Column

During tutorials, the target label column is pre-set. You use it without thinking.

When it comes to the real-world, selecting your target variable can easily become the most critical step in the entire project — yet no machine-learning knowledge is involved whatsoever.

Is churn defined by any customer who hasn’t made a purchase for the last thirty days? Does a successful recommendation mean just a click on an ad or a complete purchase as well? How does fraud look: all transactions flagged or only those that led to financial losses?

Different definitions lead to different datasets, different models, and different outcomes. Minor differences can completely invert the concept of signals vs. noise.

Your target variable is not something you just select from your list of columns. This is a business metric converted into data; thus, if selected incorrectly, will make your model consistent, learnable, and totally pointless at the same time.

Cleaning Data Is a Judgment Call

It is generally understood by novices that data cleaning entails filling missing values, removing duplications, and encoding categorical features. These are all true statements, but these aren’t the tricky parts.

The tricky parts include determining how to interpret the data being processed.

Should the missing value be considered missing due to non-occurrence or just a missing data point? This can make a big difference in how to address the missingness issue. Is the outlier a result of a simple typo, or is it the most critical piece of information that the machine learning model should learn from? Is the duplication just a duplication, or are there really two similar entries that both need to be kept?

These issues are not automatically resolvable. They will always require interpretation of the data source and its business context. A person who does not tailor their data preparation pipeline according to every data science task is not performing data science. He is simply completing the task at hand without reading the instructions carefully.

Features Are Where Domain Knowledge Shows Up

The model learns from the features you feed it. In other words, your engineering — not merely your collecting — determines its capabilities.

With customer-behavior challenges, for example, recency, frequency, and monetary metrics hold more relevance than simple transactional rows. With demand-forecasting tasks, lag features and moving averages trump daily counts. When recommending products to users, a user’s purchases in the last thirty days might be much more indicative of future actions than a history of purchases throughout their life cycle.

None of that is learned by building a model. It is understood based on how well one understands the domain enough to realize what features will provide a useful learning opportunity. Two data scientists working off of the exact same raw data would create completely different feature sets, build entirely different models, and generate meaningful variations of results not because of algorithmic choices, but because of differing levels of domain knowledge.

The algorithmic choice was the least unique to each individual.

Leakage Makes Bad Models Look Brilliant

Data leakage refers to instances when the model unintentionally gains access to some kind of data that will not be available once deployed. Understanding data leakage is probably the single most essential concept to know in the realm of data science, since it does not make your models look bad.

Instead, it makes them look overly great.

Leveraging future data for predictions. Using a feature that is highly correlated with the target. Utilizing features that are created with regard to events taking place after the target variable has been established. Randomly dividing time series data instead of splitting by dates. Making the same client belong to training and test datasets while improving the appearance of the model’s generalization abilities.

Validation scores in all those cases are likely to be high. The algorithm performs well on unseen data. However, when deployed, it does not work well enough, and the reason behind this is that the notebook was assessing something nonexistent.

Learning how to avoid data leakage before considering the models performing well is among the skills that distinguish experienced practitioners from novices.

The Two Lines Are the Output of All the Decisions Before Them

Having gone through problem framing, target definition, data cleaning, feature engineering, and leakage detection, the modeling stage indeed becomes quite short.

.fit() and .predict() do exist. They are no lie.

But they do not turn into sorcery. They become a result of everything that was decided before. The model takes on all the characteristics of your data as you pass them to it. If there are good and informative features, the model learns something. And if the target was well-defined, it optimizes something meaningful. Clean data and fair splits give you performance scores that tell you something about it.

That is why these two lines look so simple. Because somewhere along the way, somebody else — either in a tutorial, in their experience, or by example from a senior practitioner — put in all the effort required to make these two lines happen.

What to Practice Before You Obsess Over Algorithms

While still relatively new to data science, what one should focus on developing is not algorithms, but rather ways of approaching the data prior to modeling.

One should practice asking oneself what each row in the dataset really means. One should also practice considering whether the target even makes sense in a business context prior to running any modeling efforts. Additionally, one should practice starting from a simple baseline rather than jumping straight into more complicated approaches. Furthermore, one should consider leakage whenever creating features and choose metrics based on the objective of the problem and the nature of the solution, not ease of computation.

The point here is not avoiding models, but understanding the idea that there is an extensive process leading up to calling the .fit() method.

Machine learning can look easy when all the difficult decisions have been made by someone else. The moment that decision-making becomes one’s own responsibility, however, those two lines will become the least of one’s worries.

The model is only two lines because the real data science happened before the model ever saw the data.

The contents of external submissions are not necessarily reflective of the opinions or work of Maven Analytics or any of its team members.

We believe in fostering lifelong learning and our intent is to provide a platform for the data community to share their work and seek feedback from the Maven Analytics data fam.

*Submit your own writing here if you’d like to become a contributor.*

Happy learning!

-Team Maven