Elegance: The way to stay relevant in the era of AI as a Data Scientist

The AI boom has really hit the ground and people have been feeling left out thinking AI has taken over their regular tasks. While the feeling is real, not everybody has been on the same boat. Some of them are overwhelmed by the capabilities of AI and building a lot many things utilising the same. They are the ones outpacing everybody in the AI-era and thus making an impact. Productivity has gone through the roof and every single discussion mentions AI some uncountable times. Every mail that we see has some AI elements into it. But what’s missing is originality and first principle thinking. This keeps me thinking about the trade-off between elegance and blind execution.

What has been the USP of a ML scientist?

Is it that ML scientists can build a model but others can’t? I doubt. ML scientists understand uncertainty, causality, optimization, statistical reasoning etc. But knowledge of the ML model is democratized and it’s no longer in the hands of only ML scientists. Given a problem, anybody can build an XGBoost model and get a reasonable baseline outcome today. Of course we can debate about the quality of the outcome but a basic model build doesn’t need any knowledge of ML honestly. But was that ever the USP of a core data scientist?

Participating in a decent number of kaggle competitions at least got me an exposure that it was never about a choice of model. It was always about elegance. How elegantly you look at the data, how elegantly you look at the patterns and innovate through feature engg or model architecture.

Elegance is strongly coupled with the foundation of maths , numbers and probability. And ability to think and see through. In fact the later trait is highly under-rated. There are a lot of people who are very good in maths but still underwhelm in solution designing. But somebody who is able to see through and able to think unconventional will have the upper edge. They mostly figure out the maths once they know what to focus on. In the ML world, a simple and elegant solution generally outperforms any baseline no matter how complex the model architecture is.

So what is elegance ? It’s the ability to abstract, thinking from first principles, de-noise a problem statement and identify hidden structure, reducing complexity.

While these are all theories, there are ample real-world examples that I have seen in my decade years of experience working with brilliant talented professionals and researchers. We were working on a problem statement to predict the sales prediction of a new product which is yet to launch. Now this is a classic problem of the new product diffusion model. There are enough theories around the same but somebody in the team borrowed the concept of ant’s pheromone behavior and used a simulation framework to forecast. The forecast accuracy went up significantly and eventually business adopted the framework. Now this was very unconventional back then and there was hardly any literature available around the same. But the ability to take risks, triangulate theories and eventually implement them made the difference. I have seen kaggle winning solutions being a simple histogram based approach. In the recent past, my team was working on an idea to understand how we can drive premiumness for a given SKU. There were a lot of suggestions to use ML

Model based approach around the same. While they had their own merit, we proposed a simple percentile based approach to check feature level rarity in premium vs non premium product lines and take a ratio to calculate the lift.While the approach involved very little traditional machine learning, the insights started to make sense once we applied the framework. While I can go on about these, the same is applicable in the research domain as well. A lot of innovation in model development comes from thinking deep about them and understanding where they lack.

This brings to my last and final point on this aspect. What is knowledge ? Some of us spent 3–4 hours a day listening to research papers / podcasts / other modes of content. In most cases, a video posted on YT will have millions of views, hence millions of users should have the same knowledge. But knowledge is what you understand from the video and also how you superimpose other knowledge from similar / dissimilar video and form POV around the topic.

Knowledge is abundant, but the ability to connect unrelated ideas is scarce. This is what separates exceptional thinkers from average practitioners in their field.

While AI has the general knowledge and its ability to execute is supreme, what remains scarce is to connect distant ideas, uncover hidden structures in messy systems. Perhaps that was always the true USP of a great ML scientist. AI can be the hand but elegance of thoughts must remain the brain.