Auto Labelling vs Manual Labelling: What ML Practitioners Actually Need to Know

A practical look at how training data is actually produced in modern ML pipelines and why the debate is missing the point.

Photo: labelleer.com

Nobody will tell you this when you start working on real ML projects:

The biggest chunk of the time you spend on the project will not be around the model itself, rather it will be spent arguing about labels.

Your focus won’t be on architecture choices or hyperparameters. Labels are the star of the show.

Photo: kdnuggets.com

Is this uncommon case considered a positive or a negative? Should the ambiguity of this sentence lead it to be neutral or slightly negative? A question is certain; Why did a disagreement happen between Daniel and Sarah on 15% of the same batch of data?

Once reality is faced, comparing auto labelling and manual labelling does become a matter of practicality rather than a theoretical debate.

To further our analysis we will take a deeper look into the two methods, discuss both the strengths and weaknesses for each path and figure out the reason behind the fact that advanced pipelines are often combining them. If you know the basics already, you should stick around as details will further unveil key elements to determine the decision.

Manual Labelling’s Stronghand (and Where it Falls Apart)

It wouldn’t take much time to figure out what is Manual Labeling: a human that reviews a data point and after an adequate amount of consideration decides how to label it, it is as simple as it sounds. And for the longest time this was the only option available. Even in today’s age, it is still a standard in many situations.

The reasoning behind this is simply context, the human race has a gift for spotting patterns and things that might be hard to describe. The tone in a text. An ambiguity in medical images. Even sarcasm in a customer review. In similar cases, where labels must showcase that kind of evaluation, this is most needed when you’re in a pre-mature phase of a project, where that labels are still undefined, that’s when manual annotation performs the best.

Manual labeling is also key when building a strong ground truth. A small and carefully labeled dataset that has been reviewed by domain experts will give you a strong baseline to compare everything else to. It is crucial to determine whether your auto labeling is working or not.

This is the main reason that explains perfectly how sometimes human review is non-negotiable. For example, we can use situations in areas that require precision, where a bad label isn’t a drop in accuracy anymore, it is now a serious matter with effects.

Still, keep in mind that manual labeling also has its limitations and you will realize this limitation, as you work with many examples and will clearly notice that the total number of examples you created does not meet your expectation. In case you need 200,000 labeled examples and it takes about 30 seconds to label an individual example manually, you will have to complete approximately 1,700 hours of manual labor. This doesn’t include all the other interactions such as training, managing quality, and resolving disputes. In other words, achieving consistency in performance with humans over multiple days and weeks is challenging.

It is also important to remember as human beings we are all subject to the effects of fatigue, so repetitive labeling of examples for multiple hours could lead to an accumulation of small but potentially significant errors in the labeled examples. Many of the errors in this situation are consistent; therefore, the accumulation of errors is increased and introduces increased risk.

Auto-labeling: Assures Speed, but does it Escape Failure?

Unlike its rival, Auto-labelling in definition is a broad term that covers a bunch of different techniques. So it can only be fair that we clarify what you are choosing between.

Rule-based labeling:

It assigns labels with the help of heuristics that are already predefined. Let’s say a document contains the word “invoice” and it includes a total amount, it will be labeled as an invoice. Super fast yet only relies on limited patterns that you can describe. Anything you don’t mention, it doesn’t include.

Model-assisted labeling:

A model is trained on a small labeled set and it is used to predict labels for newly introduced data. This method uses reviewers to correct the predictions instead of doing the labeling from scratch. This is much faster than what we consider as a pure manual review. However, if you are not careful enough, you are carrying your model’s biases to your training data, this problem can build up quietly across all training.

In practice, many teams rely on a *data labeling platform *to manage this workflow, handle reviews, and keep annotation consistent at scale.

LLM-based labeling:

At present, the most popular option with the greatest momentum is LLMs. The most well-known examples of LLMs such as GPT-4 or Claude have the capability to receive prompts and perform classification and extract entities at large scales. In general, the results generated from LLMs are extremely good, and may be compared to human annotations through crowdsourcing. However, “good” does not imply “reliable.” Believe it or not, LLMs have a propensity for “hallucination.” They demonstrate increased sensitivity to the phrasing of prompts, and they may contain biases that coincidentally fit your needs or frequently do not.

From the evidence gathered from various automatic labeling techniques, we can conclude that they can all reliably produce massive amounts of labeled data; however, they all contain errors. While error rates will differ between techniques, rule-based automatic labels will almost always use incorrect assumptions, model-based automatic labels will be biased, and LLM-based automatic labels will have inherited biases from their initial training data irrespective of whether they apply to your application.

Despite the potential errors indicated by these techniques, the impact of those errors cannot be overstated, as they demonstrate that automatic labeling should not stand alone but should always be accompanied by an independent validation method.

Real-World Pipelines are Hybrid, Why is that and How does it look like?

Working on a real ML project would mean that you will not be exclusively doing either manual or auto labeling. You will be applying both and usually it will be in a sequential manner, where humans would take the hard segments.

An example of a typical flow will be something like this:

1. Start with a small, high-quality, human-labeled set. Taking 500 to 2,000 examples validated by people that have a full grasp on the task. This is your ground truth. This is where label schema debugging occurs. This is the part of the process where edge cases appear, and humans are needed for handling such cases before any automation begins.

2. Bootstrap a model on that set and start generating predictions. Even the average model speeds up the process of annotation. Reviewers correcting 80% of the accurate predictions would still be faster than reviewers labeling from scratch.

3. Use active learning to make the most of human effort. Instead of randomly picking new examples for review, active learning puts the spotlight on the examples where the model is least confident about. These are the examples where human judgment can actually change the model. Low-confidence predictions are marked for review; high-confidence predictions are auto-labeled and occasionally reviewed by humans.

4. Run periodic audits on the auto-labeled data. A random selected sample of examples that were auto labeled and reviewed by humans every few iterations detects wrong systemic labeling before it becomes a bigger problem. How often this review is conducted depends on how quickly your data distribution changes, as this indicates a shift in data labeling patterns.

Photo: superb-ai.com

Contrary to what you might think of at first, the goal of this structure isn’t to get rid of human annotators. It’s to make sure human effort is properly allocated such as cases of ambiguity, rare types and shifts of distributions. Certainly not cases that any good enough rule or model can get right.

Another important factor that is made clearer by this structure is: If the quality of your initial manual labels is better, the quality of your auto labeling is better. If your ground truth is noisy, your model’s predictions will definitely be also noisy, and your next round of training data will be those noisy predictions. The decline is gradual yet real.

Where most Teams go Wrong

It is not the selection of one strategy over another that leads to the error. But it is the assumption that the selection is a one-time event that happens only at the beginning of the project. As ML teams continue to work on their respective projects, the data distribution changes. As its amount increases, the weaknesses of the model are destined to increase. In the sixth month of the project, the categories that were well-defined in the first month are now edge cases. As the data distribution continues to evolve, the auto-labeling strategy that worked well initially may not work as efficiently now. It is generally more fruitful to treat the labeling as an ongoing process rather than a one-time data pipeline. As you should know the best auto-labeling tool was not developed by teams that created the most trustworthy labeled datasets, but by teams that created the most obvious feedback loop between the model and the labels.

Conclusion

Manual labeling gives context and quality. Auto-labeling gives speed and scale. Yet they are co-dependent. Both need someone looking for potential errors. By bringing these two ideas side by side, the debate between manual and auto-labeling is not really the main comparison. The real question is what ratio should you use to involve the two methods two each and every level of work or data. You are encouraged to answer that question honestly. Make a process out of that, and revisit it often. That is where the real work is.