Power of the Computing Power, Big Fish vs Small Fish

Power of the Computing Power, Big Fish vs Small Fish

I always wondered and thought about how come the winners in Kaggle competitions are always the same people?

There are a few questions that come into mind:

• Do they have a very specific knowledge?
• Do they train it very differently?
• Do they have a different type of model choices that we cannot comprehend?

The answer could be yes to these questions in certain cases, but the point I am going to set as my frame is not the various situations.

The computing power! Yes, usually they are the people who are having access to huge amount of computing power or they are already having that computing power on their local machines.

The solution is always similar. A blend of 10 different models that you don’t even know if they existed before or maybe, derivatives of them.

They can also come up with the most unique hyperparameter sets, because of the AutoML pipelines can be optimizing over hundreds of different tries.

As we can see, how those people with advanced hardware can have the advantages of competition over you. There are still aspects of competition that you can focus on to compensate.

How does the big fish behave?

1. Examine the Data
2. Analyze the Data
3. Preprocess the data
4. (Most important) Create a pipeline or initialize an AutoML pipeline
5. Find the best models and hyperparameters

image from unsplash.com (source)

How to make a difference? Make it make sense.

Feature Engineering (MOST IMPORTANT)

A model or blend could be needed to be trained or optimized for 400 different times to find the best relations. But applying feature engineering on a two different features could speed up the process a lot or could make you find a relation that grid search could never find.

A good feature engineering practice often requires domain knowledge, because a hardware cannot invent its very own context.

Data Cleaning

Tossing your data into AutoML pipelines are much easier, I see. But being able to clean your data and fix the flaws takes a real human effort. A careful analysis and understanding of a data could carry you above.

Finding a leak or flaw can help you to train your data on a proper dataset instead of a mess, because of we want to avoid the ‘Garbage in, garbage out’ trap to be happening.

Custom Loss Functions

Those lazy AutoML pipelines that I mentioned before usually use the same or similar loss functions i.e. LogLoss, RMSE, or MAE etc.

But different problems often require different solutions, writing a specific loss function according to a specific domain is one of them.

Blend of a diligently Picked Models

There are some of the models that history proven that they perform better on specific solutions, AutoML pipelines usually brute force into finding optimal or best performing models then blend them into a custom ensemble. Instead of having an unlimited amount of the computing power, you can actually pick the certain models that they have proven that they performed better on a certain cases.

Conclusion

Having a large amount of a computing power provides you undeniable advantages and can help you win the competition, but there are also many aspects that we could take into an account. Especially on the competition with rewards even a difference of +0.01 can take hours of computing power on thousands of dollars in investments. There are still ways to close the gaps and having a chance to compete. As a small fish, I always wondered about the ways how. This article shows you about my understandings about the dynamics. There are also some variations like tabular data where computing power matters lesser than other types of machine learning fields i.e. CV(Computer Vision), NLP(Natural Language Processing) etc.

Having a great convenience also comes with the downsides like not being able to pay enough attention and getting used to tossing your whole data into an AutoML(Automated Machine Learning) pipeline but it doesn’t apply to all of the successful winners, having a raw power itself won’t help you win at all. They combine years of experience, computing power and domain knowledge altogether. hopefully I will be writing about this subject even more in the future and come up with the advanced ideas.

Thank you for reading my article!

-Okan.