Towards making Machine Learning work in the real world

One of the biggest challenges in Machine Learning today is in moving from a research model, often considered a prototype and useful to test out hypotheses and ideas, to a reliable, maintainable and reproducible model that works in a production environment and delivers real value to an organization.

A frequently cited paper is Hidden Technical Debt in Machine learning systems published by Google in 2015. The paper provides a visual representation of the complexity surrounding a prediction model in a production environment and gives the reader an intuition of the amount of code that might be required in writing these. One of its key points is that in order to develop an effective and robust system, we need to consider many aspects beyond the actual modeling code. It is not uncommon to spend significantly more time and effort in developing the systems that surround the model than in the model itself.

Image above was taken from: Hidden Technical Debt in Machine learning systems by Google Inc.

Former OpenAI researcher Josh Tobin shared some interesting insights about the challenge of real-world Machine Learning on a recent episode of the Gradient Dissent podcast. He thinks the methodology for ML development is still immature, and that there is a big opportunity in coming up with the “agile” of Machine Learning. While at OpenAI, he gathered insights from highly successful ML practitioners and taught it as a course at UC Berkeley that is now available online.

Among the questions teams need to think about when planning ML projects are:

• How do we build reproducible pipelines and prevent us from losing the ability to train a good model?
• What tests do we need to write for an ML system?
• How do we deploy and monitor systems in production? Do we have the right tools to debug and troubleshoot ML systems?
• How do we choose between different projects? How do we assess feasibility and impact for each project?
• What’s the best way to communicate expectations to leadership and get buy-in?
• How do we manage an ML team composed of different disciplines, including engineers and researchers? How do we create a collaborative environment between these?

Some of these questions are discussed in the Full Stack Deep Learning online course, which I found can be a very helpful framework for thinking about these problems and having a starting point. Josh also shared some insights and useful recommendations in the podcast episode. These are based on his experience as an ML researcher, as well as knowledge gathered from experts:

Start small and gradually add complexity

What makes debugging Machine Learning systems hard is that flaws in the system often mean the model doesn’t improve as much or as fast as it could. Unlike traditional software though, there are no obvious exceptions or errors being thrown. This makes it harder to validate that the system is working at its full potential. A strategy that can save a lot of time in the long run is to start with a small system we know what to expect from, validate it works properly, and only then start adding layers of complexity.

One specific implementation of this idea is when training a new model. It is often a good idea to test the Machine Learning code by overfitting a very small subset of the data (e.g.: a single batch of data). Any reasonable model architecture should be able to drive the loss down to zero and memorize the data. That way, we ensure there are no obvious bugs in the Machine Learning code before adding complexity to the architecture or training on a larger dataset, when it becomes much harder to pinpoint the issue. This approach is also suggested by Jeremy Howard in his amazing 2019 Deep Learning for Coders course.

Pictures above were taken from the 2019 Deep Learning for Coders course.

Focus on the key ideas from fewer papers

The industry is moving at a very fast pace and thousands of papers are published every month. Rather than attempting to keep up with numerous new papers, it is more useful to deeply understand the main ideas from the most important papers. Josh has seen that the most successful people have sometimes spent weeks understanding and even implementing these key concepts.

Spend enough time getting practical, hands-on experience

The field of Machine Learning can be intimidating at first, given the amount of theory behind many of the state-of-the-art algorithms and the pace of change in the industry. Many people get discouraged when faced with the task of understanding complex mathematical equations in research papers. Others spend too much time learning theory before getting practical experience on their own projects and writing code.

Both Jeremy Howard and Josh Tobin stress the importance of coding and hands-on experience as this helps practitioners understand the inputs and outputs of a Machine Learning system and how it can be used in the real world. The coding experience serves not only as useful context for people to understand where a prediction algorithm fits in the broader picture of a Machine Learning system, but also helps practitioners develop an intuition of how an algorithm works, which ultimately provides a different view of the same mathematical notation that at first seemed daunting.

Conclusion

It is an exciting time in Machine Learning as academics and industry practitioners figure out how to reliably move from a research model to a production-grade system that works in the real world. People from leading companies and research labs are compiling these guidelines based on what has worked and what hasn’t, and sometimes driving conversations to help advance the speed of iteration in the industry as a whole. This presents an opportunity for those of us in the field to drive discussions in their own organizations. It also helps us adopt the right mindset around Machine Learning projects and set the right expectations and goals when we plan projects within our teams.

What are other resources of best practices for making ML work in production? What have been the most influential papers or courses in your ML journey? Where do you agree or disagree with the ideas presented in this article?

I’d love to hear your thoughts, please do comment with suggestions and give feedback on this post.

To Learn More

[1] D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Francois Crespo, and Dan Dennison. 2015. Hidden technical debt in Machine learning systems. In Proceedings of the 28th International Conference on Neural Information Processing Systems — Volume 2 (NIPS’15). MIT Press, Cambridge, MA, USA, 2503–2511.

[2] 2019 Practical Deep Learning for Coders, Jeremy Howard (Fast AI).

[3] Full Stack Deep Learning, Josh Tobin et al.

[4] Slow down and change one thing at a time — Advancing AI research with Josh Tobin. Gradient Dissent podcast.

[5] M. Zinkevich. Rules of Machine Learning: Best Practices for ML Engineering.