The Bigger Picture: Leveraging Vision-Language Models for Computer Vision

Written By: Hilit Unger

You’ve probably heard the saying, “A picture is worth a thousand words.” For us data scientists, it’s our daily reality. Vision-Language Models (VLMs) have great potential to expedite the development of tools like visual classification engines.

At DoubleVerify, our classification engine, Universal Content Intelligence (UCI), utilizes advanced machine learning to analyze all content types, including video and images, helping advertisers protect their brands and optimize their campaigns. In this post, I’ll share insights from my team’s work on UCI.

Training a Computer Vision Model

Let’s start by discussing what it takes to train a computer vision model.

In data science, the process of training a successful computer vision model is as follows:

Figure 1

Curating a high-quality dataset to serve as a training set for a computer vision (CV) model is an art. Objects in images can appear in various shapes, colors, sizes or textures. Scientists must include many examples in the training set to teach a machine-learning model to generalize and detect all these variations. It’s challenging to predict how the model will respond to corner cases. Even in a “big data” era, curating data that is both suitable for the problem and has a permissive license that allows its use in production can be a complex task.

This has become an even bigger challenge in social networks. Uploaded videos are often captured with mobile phones in challenging conditions, such as dark, blurry, and noisy environments. These are not the old-world, carefully selected, sharp, clear images. The amount of data uploaded to platforms like Facebook, Instagram, or TikTok is incredible. This vast amount of data also contains unusual corner cases we should understand and address.

Another unique challenge for social network data is data drift. Data changes rapidly due to new filters, TikTok challenges, or other trends, so yesterday’s training set may not accurately represent today’s videos.

The best-kept secret of data scientists: If your dataset is representative and your labels are clean, training a high-quality classifier is a piece of cake!

What Are VLMs?

A vision-language model (VLM) is a generative machine-learning model that processes two data modalities: images and text prompts. A prompt is an input or query provided to the model to generate a response. VLMs can also output these modalities.

Figure 2

A VLM can be used for various tasks. Some examples are illustrated in Figure 2, including object detection, semantic segmentation, visual question answering, image-text retrieval, and other applications. Since these models were trained on massive amounts of data, they can be utilized for zero-shot tasks, leveraging a pre-trained model without requiring retraining or fine-tuning on data specific to the task at hand.

An essential attribute of such models is the embedding space, also known as the “latent space.”

Intuitively, every image or sentence is encoded into a vector of numbers that represents it semantically and captures its holistic features. Depending on our downstream task, it may be important to encode a teddy bear and a child in the image, but it might be less critical if the teddy bear is positioned on the left or right side of the image or what the child’s hair color is.

The key feature of this latent space is that it preserves distances: similar semantic images will be grouped closer together in the latent space. For example, images of a child holding a teddy bear, regardless of small changes in details, will have a similar numeric vector. VLMs were trained to ensure that images and their textual descriptions are similar, i.e., the numerical representation of the textual description and its corresponding image will usually be close.

We can mathematically define a similarity metric based on this vector - if two images are similar, their similarity score will be close to 1. Figure 3 visualizes an embedding space in 2D: a soup image will have a high similarity score and a short geometrical distance to other soup images, while pizzas or cakes are farther away.

Figure 3

With all the advantages of VLMs, there is one significant drawback - VLM models are compute-heavy. These models’ tunable parameters range from 100M to 70B or more. At DoubleVerify, we classify massive amounts of content daily, so using VLMs as production classifiers would result in a slow and expensive system.

Massive amounts of classified content necessitate the use of more efficient models in our production pipeline. But, this doesn’t mean we didn’t use VLMs in our development cycles!

Using VLMs to Accelerate the Development of a CV Model

Although, due to the reasons I already shared, we chose not to use VLMs for production, we did find ways to enhance our development cycles by using VLMs for data curation or labeling.

Zero-shot Dataset Curation

Collect large amounts of images, process them with a VLM to get their embedding or vector representation, and save these in a “vector database.”

We can then query this database by entering textual descriptions, called prompts, or find images that are visually similar to a reference image. We can curate images from this database by computing the similarity metric to each stored image and taking the highest scores. This method enables us to select the appropriate images for our training set and curate a reasonably sized, representative set. We used this method to curate targeted data for trends where the model underperformed. Figure 4 illustrates two possible methods for curating new images from the vector database.

Figure 4

Zero-Shot Auto-Labeling

We can use a VLM to automatically assign labels for large training or test datasets by optimizing text prompts. This significantly reduces the time, cost, and effort needed compared to human labeling, freeing data scientists from considering dataset sizes. Shortening labeling cycles reduces the full model development cycle and enables us to utilize advanced practices, such as active learning. When we minimized the VLM’s response variability, we found that it was more consistent than the consensus between a few human labelers.

Another reason to use a VLM is that we have found that, in some Visual Question Answering (VQA) tasks, answering questions such as “Does the beverage depicted in the image contain alcohol?” depends on the labeler’s familiarity with alcohol brands. Human labelers would struggle to recognize local beer brands sold in other countries. VLMs, having been trained on massive amounts of data, show better success with this task.

Figure 5

Classification with text similarity:

In cases where collecting a good training dataset is challenging, VLMs can also serve as production classifiers. Building a classifier can be as simple as giving a set of “include” and possibly “exclude” textual descriptions with a similarity threshold for each sentence, using the fact that in VLMs, the embedding vector of a text description sentence is similar to the embedding of its corresponding image. This method enables faster classifier design than traditional methods, eliminating the need for training data collection. As always, it is essential to ensure that the metrics, such as recall and precision, are sufficiently high.

Insights from Using VLMs at DoubleVerify

At DoubleVerify, we classify visual content at scales of hundreds of millions of items per day. Our classification system utilizes VLMs to automatically predict labels for images in offline tasks, including data curation and performance evaluation.

Our automatic labeling prompts consist of two parts: a label name and detailed instructions explaining which concepts should trigger that label. For example, for the label “Underage Usage of Alcohol,” the instructions would include the depiction of a child under the age of 13 touching a vessel (a bottle, can, etc.) containing alcohol.

As part of the prompt optimization process, our prompt specialists found that using meaningful label names, such as “Underage Usage of Alcohol,” may carry implicit meaning or introduce additional context that can interfere with the detailed instructions in the prompt, sometimes resulting in incorrect or non-compliant outputs. In the example of “Underage Usage of Alcohol,” the model assumed the age limit for underage and ignored the detailed instructions.

The prompt specialist found that using shortened neutral codes instead of the implicit label name (e.g., using “UUA” instead of “Underage Usage of Alcohol”) ensures that the model adheres strictly to the instructions part and mitigates against the risk of the model applying its interpretation.

Using this small trick, we were able to improve the automatic labeling precision for this label by 3% and increase the recall of “no avoidance” images, which do not contain underage children consuming alcohol, by 5%.

Turning Knowledge into Action

VLMs are powerful tools that can automate various tasks and significantly expedite the development of visual classifiers by automating data curation and labeling processes, which traditionally require substantial manual effort.

I hope this post will encourage you to identify the pain points in your development process and use VLMs to shorten your development cycles. Remember to keep an eye on details to ensure the best outcomes!