Challenges in deploying machine learning systems

There are various issues after deploying the machine learning systems into production. In this blog, you will read some of the challenges that occur.

Let’s start this with a Vision Inspection example, let you have build and deploy the vision inspection system into production, the task of this vision inspection system is that this system detect the scratch on the smartphone when a user provides the image of the mobile phone. While training the system you have used neat and cleaned images of the smartphone, but when the user uses this system in some factory then it might be due to some lighting condition the system does not find the neat and cleaned images and the accuracy decreases. This is the big issue in machine learning in production, this is called concept drift and data drift.

Let us understand concept drift and data drift.

Concept drift: The mapping X to y changes over time. Let’s understand this by a real-world example, let’s say you built a machine learning system for predicting house prices, after some time this system will not be reliable because the price of the house may change (increase).

Data drift: The distribution of the input features X changes over time and this would be problematic. Hence the performance of the system degrades over time.

Now let’s discuss Software engineering-related issues.

Software engineering issues: To handle these issues there are some questions that we need to answer them.

Realtime or Batch: Configure whether the system is real-time prediction or batch prediction. For example, voice recognition system, a user speaks something and he/she gets the answer within half a second. And for batch prediction, there is a system that takes patent records and runs them overnight batch process.

Cloud Vs. Edge/Browser: Does the prediction server run on the cloud or browser?

Compute Resources (CPU/ GPU/ memory): Identify how many compute resources you have and give them to the system to work.

Latency and throughput: How much time you allocate to get back the answer. And throughput refers to how many queries per second the system can handle for given compute resources.

Logging: You should keep track of some metrics for analysis purposes to identify the system’s performance.

Security and privacy: Secure your system with userID password and API tokens. Also concerns some other security and privacy-related issues.

Deploying a system requires two broad sets of tasks.

1. Writing software to enable you to deploy the system in production.
2. What needs to be done to monitor the system performance and maintenance when the concept drift or data drift occurs?

These are some key points that we need to keep in mind while deploying the system into production.