Types of Machine Learning Approaches (Part 2/2) #ML03

In the previous article, we talked about different machine learning types and introduced three fundamental ways they are categorized:

1. Supervision during training: Based on the level of human involvement and the type of data used to train the model, we can classify them as Supervised (fully labeled data), Unsupervised (no labeled data), Semi-supervised (a small amount of labeled data), Self-supervised (the model generates its own labels from the unlabeled data), and so on.
2. Incremental Learning Capability: Based on whether the model can learn from new data without having to retrain from scratch, we have: Online Learning (learns incrementally) and Batch Learning (learns from the entire dataset at once).
3. Learning Methodology: Based on how the model makes predictions, we can categorize them as Instance-based (learns by memorizing training instances and then generalizing to new instances using a similarity measure) and Model-based (builds a model from the training data and then uses this model to make predictions).

In the previous post, we covered the first part, exploring how models differ based on the supervision given during training.

In this article, let’s dive into the other two crucial criteria: Incremental Learning Capability and Learning Methodology. Understanding these aspects is vital for choosing the most appropriate machine learning system for a given problem and dataset.

Incremental Learning Capability

The ability for a machine learning system to learn incrementally is a characteristic you might not immediately consider as essential, but it becomes incredibly relevant in real-world scenarios, especially when dealing with continuous streams of new data. If you’ve ever worked with data from an organization that experiences a regular inflow of information, you’ve likely felt the need for this capability.

Let me give a clearer example. Imagine you’ve trained a model to predict customer behavior using data collected over the past six months of 2024. Customer behavior is dynamic and changes rapidly due to new trends, technologies (like AI shifts impacting behavior), and market conditions. To keep your model accurate and relevant, it’s crucial to update it with this new customer behavior data as it becomes available in 2025. Now, what do you do with the data collected in the new months? Are you going to retrain the model from scratch every time with the entire combined dataset from 2024 and 2025? This is where the concepts of Batch Learning and Online Learning become critically important.

01. Batch Learning

Also known as Offline Learning, this is the traditional approach where the model is trained using all the available data at once. Once trained, the model is deployed into production and operates without undergoing any further learning or updates.

This method can be computationally intensive and time-consuming, especially when dealing with very large datasets, as the model needs to process the entire dataset during each training cycle. Incremental training is simply not an option here.

However, as we touched upon before, the utility and accuracy of a model trained using the batch method tend to degrade over time because the real world is constantly evolving. People’s preferences change, businesses adapt, new products emerge, and external factors like economic shifts or technological advancements (like the impact of AI) influence patterns. This phenomenon, where the relationship between the input data and the target variable changes over time, is known as Model Drift.

The speed of this model drift can vary significantly depending on the use case. If a model is trained to classify static product images, the drift might be relatively slow (unless product designs drastically change or many new products are added). However, for tasks like predicting stock market movements or analyzing trending social media topics, model drift can be very fast. A model trained on financial and political news from 2024 might quickly become outdated in 2025 due to significant geopolitical events or policy changes. Think about how rapidly markets reacted and changed after major political events — these cases accelerate model drift.

So, in scenarios where incremental learning is not possible or Batch Learning is the chosen approach, maintaining model performance requires periodically retraining a new version of the model from scratch using the entire accumulated dataset (both old and new data). The newly trained model then replaces the old one in the production environment.

While this retraining process can be automated, it still requires significant computational resources and time, potentially causing delays and monetary losses for a company if the model needs frequent updates. This is precisely where Online Learning offers a compelling alternative.

02. Online Learning

Source: Check References below

Online Learning provides the capability to train models incrementally, meaning the model can learn from new data points or small batches of data as they arrive, without needing to process the entire historical dataset again. This directly addresses the disadvantages of time and cost associated with retraining in the batch learning approach.

This method is particularly valuable for companies and systems that experience a rapid and continuous inflow of new data and operate in environments where the speed of model drift is high. By continuously updating the model, Online Learning allows it to stay relevant and accurate, quickly adapting to changing patterns and trends.

Furthermore, Online Learning is often a preferred option when there are limitations on computing resources or time. For example, if you have a massive dataset that cannot fit into the memory of a single machine, Online Learning allows you to process and learn from the data in smaller chunks.

Source: Check References below

A crucial factor to consider in Online Learning is the learning rate. The learning rate determines how much the model adjusts its parameters in response to each new data instance or batch. It controls the pace at which the model learns from new information and, consequently, how quickly it might “forget” past data. A learning rate that is too high can make the model overly sensitive to recent data, causing it to quickly forget previously learned patterns and potentially perform poorly on older data. Conversely, a learning rate that is too low might prevent the model from adapting quickly enough to new changes, making it slow to react to model drift. For instance, in a fraud detection system or a recommendation engine, it’s important for the model to learn new fraudulent patterns or user preferences quickly, but historical data still holds value for identifying long-term trends or established behavior. Striking a balance with the learning rate is essential to ensure the model effectively learns from both new and old data.

Learning Methodology

The ultimate purpose of most Machine Learning models is to make predictions — whether a customer will make a purchase, diagnose a specific disease, estimate a house price, and so on. How these models arrive at these predictions depends fundamentally on how they learn and generalize from the training data. While it’s true that models learn from data, the method they use to extrapolate from that learned data to make predictions on new data falls into two main categories: Instance-based Learning and Model-based Learning.

01. Instance-based Learning

Source: Check References below

Instance-based learning, sometimes referred to as “lazy learning”, is a type of machine learning where the model does not explicitly create a generalized model or rule from the training data upfront. Instead, it stores or simply memorizes instances of the training data. When it needs to make a prediction on a new data point, it compares this new instance to the stored training instances using a similarity measure and bases its prediction on the most similar stored examples.

This method is particularly useful when the underlying patterns in the data are complex and difficult to summarize with a simple, fixed mathematical model. Rather than deriving a function that maps inputs to outputs, instance-based learning algorithms defer the processing until a prediction is requested.

A well-known example of an instance-based learning algorithm is k-Nearest Neighbors (k-NN). For a new data point, k-NN finds the ‘k’ training instances that are closest to it (based on a distance metric, the similarity measure) and then makes a prediction based on those neighbors (e.g., for classification, it assigns the majority class among the ‘k’ neighbors; for regression, it might take the average of their values).

For example, let’s say a hospital has a database of past patient records, including symptoms, test results, and final diagnoses. When a new patient arrives with a set of symptoms, an instance-based system using k-NN would compare the new patient’s symptoms to those of historical patients using a similarity score. If the 5 closest historical patients all had a particular diagnosis, the system would likely suggest that diagnosis for the new patient. Another example is a recommendation system that suggests movies to a user based on the watch history of other users who are deemed “similar” to them (i.e., have watched many of the same movies in the past).

02. Model-based Learning

Source: Check References below

Model-based learning is a machine learning approach where the system builds an explicit, often mathematical or logical, model of the training data during the training phase. This model captures the underlying patterns, relationships, or rules present in the data. Once this model is trained and its parameters are optimized, it is used to make predictions on new data points. Unlike instance-based learning, the original training data is often no longer needed for making predictions once the model is built (though it might be kept for retraining purposes).

Model-based learning involves selecting a specific model type (e.g., linear regression, decision tree, neural network), training this model on the data to learn the parameters that best fit the data, and optimizing these parameters to minimize prediction errors.

A common example of model-based learning is Linear Regression, where the system learns the coefficients of a linear equation (y=mx+c) that best describes the relationship between the input features (x) and the output (y). To predict a new value, you simply plug the new input features into the learned equation. Other examples include Decision Trees, which build a tree-like structure of rules to classify or predict values; Support Vector Machines (SVMs), which find a hyperplane to separate data points; and Neural Networks, which build complex layered structures to learn intricate patterns. In all these cases, a specific model structure is defined and then fitted to the data.

Let’s consider an example in predictive maintenance. Model-based learning helps predict when industrial machines are likely to fail, allowing companies to perform maintenance proactively before breakdowns occur. We would first collect data from sensors on machines measuring factors like temperature, vibration, pressure, and usage hours. We then train a model (e.g., a regression model or a classification model) based on this data to identify patterns associated with impending failures. For instance, a decision tree model might learn rules like “IF vibration is high AND temperature is rising THEN predict high risk of failure.” Once trained, this model can analyze real-time sensor data from operational machines and detect early signs of wear or malfunction based on the learned patterns, triggering a maintenance alert.

Final Thoughts

We’ve explored two more key ways to categorize machine learning models in this article— based on their incremental learning capability (Batch vs. Online) and their learning methodology (Instance-based vs. Model-based).

Each category has its own strengths and weaknesses, making them better suited for different types of problems and deployment scenarios. Choosing the right system is a critical step in building effective machine learning solutions.

Let’s connect with another informative article soon. See you then!

Reference:

• Aurélien Géron (2022). Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow. O’Reilly Media.