What exactly is CLIP (Contrastive Language-Image Pretraining)?

How teaching an image encoder and a text encoder to share one space lets a model classify any picture it has never been trained on, using nothing but text prompts.

CLIP is a model that learns one shared space where a picture and its caption land at nearly the same point, which lets it recognize images it was never trained to recognize.

We are going to build the heart of CLIP by hand. We will take three images, a cat, a dog, and a car, turn each one and its caption into a vector, and watch a single 3x3 grid of numbers do all the work.

By the end, that grid will tell us not only how CLIP learns, but how it classifies a brand new image using nothing but text. One operation does both. That is the elegance we are after.

The problem with closed-set vision

To appreciate CLIP, we first have to feel the limits of the way vision used to work. A ResNet trained on ImageNet knows exactly 1,000 categories and nothing else. To teach it one new class, you must collect labeled images and retrain it.

The deeper issue is the kind of supervision. The model is told “this is class 207” through a one-hot label, a vector that is all zeros except for a single 1 in slot 207. That label carries no meaning. The network never learns what “golden retriever” actually refers to; it only learns which slot to light up.

As shown in figure 1, this is the gap between a closed label and open language. On the left is the one-hot vector, a single highlighted slot inside a fixed set of 1,000 classes. On the right is the caption “a photo of a golden retriever”, plain language that any reader understands.

Figure 1. A closed-set one-hot label (a single highlighted slot in 1,000 fixed classes) versus open-vocabulary language (the caption “a photo of a golden retriever”). The label is a closed set; the caption carries real, open-ended meaning.

As illustrated in figure 1, the label is a slot and the caption is a sentence. Labeling at this scale is also the bottleneck: ImageNet’s 1.28 million images were hand-labeled by crowds, and hand-labeling does not scale to the open visual world.

Here is the pivot. The internet already contains hundreds of millions of images paired with text, in alt-text and captions. That text is free, rich, open-vocabulary supervision. CLIP replaces fixed labels with language supervision at scale, learning from about 400 million image-text pairs.

The numbers tell the story by themselves. ImageNet offers 1,000 fixed classes and 1.28 million hand-labeled images; CLIP learns from roughly 400,000,000 web image-text pairs and an open vocabulary. Because the supervision is language, the set of classes at test time is open. Any text prompt can be a label, and that is what unlocks zero-shot.

So the plan is to supervise with language. But an image and a sentence are two completely different kinds of object. How do we even compare them?

The two-tower idea

To compare an image and a sentence, we first need to put them in the same room. CLIP uses two encoders for this. An image encoder (a ResNet or a Vision Transformer such as ViT-B/32) turns a picture into a vector, and a text encoder (a Transformer with a 77-token limit) turns a caption into a vector.

Each encoder ends in a linear projection to a common dimension, 512 for ViT-B/32. After that projection, an image vector and a text vector live in the same space and can be compared directly. As shown in figure 2, the two towers meet in one shared embedding space.

Figure 2. The two-tower architecture. An image encoder (ViT-B/32) and a text encoder (a 77-token Transformer) each end in a linear projection to a 512-dim shared embedding space, where image and text vectors can be compared directly.

As illustrated in figure 2, the cat image flows up the left tower and the caption “a photo of a cat” flows up the right tower, and both arrive in the same central box.

Let’s introduce the running artifact we will trace through the entire article. We have three pairs: a cat, a dog, and a car, each with its caption. So our batch size is N = 3. Real CLIP uses 512-dim vectors, but to keep every number hand-checkable we will use tiny 3-dim illustrative vectors:

• cat: image (0.9, 0.1, 0.1); caption “a photo of a cat” (0.85, 0.15, 0.10)
• dog: image (0.1, 0.9, 0.1); caption “a photo of a dog” (0.15, 0.85, 0.12)
• car: image (0.1, 0.1, 0.9); caption “a photo of a car” (0.12, 0.10, 0.88)

Notice that each caption vector is deliberately close to its matching image vector. That is what a trained CLIP produces, and it is what we want our shared space to look like. As shown in figure 3, the matching pairs cluster together.

Figure 3. The shared embedding space. Each image vector (one shape) lands close to its matching caption vector (another shape): the cat pair clusters together, as do the dog pair and the car pair. Matching meaning lands at nearly the same point.

As illustrated in figure 3, the cat image and the cat caption are neighbors, the dog pair are neighbors, and the car pair are neighbors. One space holds both modalities, and meaning lines up across them.

We can now place images and text in one space. But “near each other” needs a number. We need a way to measure how aligned two vectors are.

Measuring alignment

We want a single number that answers “how much do these two vectors point the same way?” What carries meaning here is direction, not length, so we want to compare direction and ignore magnitude. That measure is cosine similarity.

The dot product is a natural starting point, because it already rewards vectors that point the same way. The problem is that it is contaminated by length: a longer vector inflates the score regardless of direction. So we L2-normalize each vector, dividing it by its own length to put it on the unit circle.

After normalization, both vectors have length 1, and their dot product equals the cosine of the angle between them. That is a clean number in the range [-1, 1], where 1 means identical direction and 0 means orthogonal. As shown in figure 4, normalization is what turns a raw dot product into cosine similarity.

Figure 4. L2 normalization projects a raw vector onto the unit circle. Once two vectors have length 1, their dot product equals cos(theta), the cosine similarity, a clean number in [-1, 1].

As illustrated in figure 4, the raw cat image vector (0.9, 0.1, 0.1) sits off the unit circle, and normalizing it slides it onto the circle. Take the normalized cat image and the normalized cat caption, and their dot product is about 0.90. Take the cat image against the dog caption, and the dot product drops to about 0.12.

So a matching pair scores about 0.90, and a mismatched pair scores about 0.10. Every pair of image and text now reduces to one number in [-1, 1].

We have N images and N captions. Comparing every image to every caption gives us not one number but a whole grid. That grid is where CLIP lives.

One grid that says it all

For a batch of N pairs, we compute all N x N cosine similarities. The N correct, matching pairs sit on the diagonal, and the N(N-1) incorrect pairs sit off the diagonal. The entire idea of CLIP is one picture: we want this matrix to look like the identity, bright on the diagonal and dark everywhere else.

For our three pairs that is a 3x3 matrix, with rows for the images and columns for the texts. As shown in figure 5, the diagonal is high and everything else is low.

Figure 5. The 3x3 cosine similarity matrix (rows = images, columns = texts). Diagonal cells (matching pairs) are about 0.90; off-diagonal cells (mismatched pairs) are about 0.10. A trained CLIP produces a near-identity matrix.

As illustrated in figure 5, read the matrix row by row. The cat-image row is high under the cat caption (0.90) and low under the dog and car captions (0.12 and 0.10). The dog row is high under the dog caption, and the car row is high under the car caption.

Mechanically, this whole grid is one matrix multiply. Stack the three normalized image vectors into a (3, 3) matrix, stack the three normalized text vectors and transpose them into a (3, 3) matrix, multiply, and you get the (3, 3) similarity matrix. The diagonal sits at about 0.90 and the off-diagonal at about 0.10, a near-identity matrix.

That is the entire CLIP objective in one sentence: make this matrix the identity. That bright diagonal is what a trained CLIP produces. Before training, the grid looks very different, and training is the story of how one becomes the other.

Pulling the diagonal up

Early in training the encoders are random, so every cell of the matrix is roughly equal. The grid is near-uniform, and the model genuinely cannot tell the matching caption from the others. Training reshapes the grid by pulling the diagonal up and pushing everything off-diagonal down.

As shown in figure 6, this is the visual spine of the whole article: a near-uniform matrix on the left becoming a near-identity matrix on the right.

Figure 6. What training does. The untrained matrix is near-uniform (every cell about 0.33, no dominant diagonal). The contrastive objective pulls the diagonal up and pushes the off-diagonal down, producing the trained near-identity matrix.

As illustrated in figure 6, the untrained matrix gives roughly equal weight to every caption, so after softmax each cell is about 0.33, pure chance. The trained matrix is the near-identity grid from the previous section. The arrow between them is the entire learning process: pull diagonal up, push off-diagonal down.

How do we turn that goal into a loss? Treat each row as a tiny classification problem: “which of the 3 captions matches this image?”, with the correct answer sitting on the diagonal. That is exactly what cross-entropy scores.

So we apply cross-entropy across each row, which is the image-to-text direction, and across each column, which is the text-to-image direction, and we average the two. As shown in figure 7, this symmetric objective is the contrastive InfoNCE loss.

Figure 7. The symmetric contrastive (InfoNCE) loss. Each row is an image-to-text classification (cross-entropy across the row) and each column is a text-to-image classification (cross-entropy across the column), with the diagonal as the correct label. The loss is the average of both directions.

As illustrated in figure 7, the diagonal cell of each row is the correct label, the horizontal bracket reads the row as image-to-text, and the vertical bracket reads the column as text-to-image. The loss is simply the average of the two directions.

The numbers make the before-and-after vivid. In the untrained, near-uniform matrix, the correct-class probability is about 33% and the per-term loss is about -log(1/3) = 1.10. In the trained matrix, the correct-class probability is about 99.99% and the loss is essentially 0.

That before-and-after transformation of the same grid is the spine of the whole article. We said the trained matrix gives 99.99% confidence, but a raw 0.90 versus 0.10 gap is not that dramatic. One small scalar does the sharpening. Let’s follow the cat row through it.

Turning 0.90 into 99.99%

A raw cosine gap of 0.90 versus 0.10 is small, and a plain softmax of those values would be lukewarm, nowhere near confident. CLIP fixes this by multiplying every similarity by a learned scalar called logit_scale, which acts as an inverse temperature. It stretches the gap so the softmax becomes decisive.

The crucial detail is that this scalar is learned, not a knob you set by hand. The network decides how confident to be. It is initialized so the temperature tau = 0.07, which means logit_scale = 1/0.07, about 14.29, and it is clamped so it never exceeds 100.

Let’s follow the cat row through it with exact numbers. As shown in figure 8, we start from the raw cosines, scale them, and softmax.

Figure 8. The temperature and softmax, traced on the cat row. Raw cosines (0.90, 0.12, 0.10) are multiplied by the learned logit_scale of 14.29 to give logits (12.86, 1.71, 1.43), and softmax turns those into probabilities (0.99997, ~0.00001, ~0.00001).

As illustrated in figure 8, here is the chain step by step:

1. Raw cosines: cat-cat 0.90, cat-dog 0.12, cat-car 0.10.
2. Times logit_scale 14.29: 12.86, 1.71, 1.43.
3. Exponentiate: exp(12.86) is about 385,000; exp(1.71) is about 5.53; exp(1.43) is about 4.18. The sum is about 385,010.
4. Divide to softmax: p(cat) = 385,000 / 385,010 = 0.99997; p(dog) = 5.53 / 385,010 = 0.0000144; p(car) = 4.18 / 385,010 = 0.0000109.

So the cat image is matched to its caption with about 99.99% probability. A 0.90 versus 0.10 cosine gap became a 12.86 versus roughly 1.5 logit gap, which became near-certainty. The cross-entropy for the cat row is -log(0.99997), about 0.00003, essentially zero.

The temperature is the dial that converts “slightly more similar” into “confidently correct.” We have now trained an image-text matcher. Here is the part that feels like magic: the very same row operation, with no retraining, is also how CLIP classifies a brand new image.

Classifying without training

At test time we do not have a fixed classifier head waiting for us. We build one on the fly from text. We take each candidate class name, wrap it in the prompt template “a photo of a {class}”, and encode each prompt into a text vector.

Then we encode the test image into one image vector and compute one row of the same similarity matrix: the image against every class prompt. Softmax that row, pick the highest, and we have a prediction. No gradients, no labeled examples, no fine-tuning.

As shown in figure 9, this is exactly the cat-row computation from the previous section, reused at inference.

Figure 9. Zero-shot classification reuses one row of the training matrix. One test image is encoded to a vector; the prompts “a photo of a cat / dog / car” are encoded to three vectors; their cosines are scaled and softmaxed, predicting cat. There are no gradient updates.

As illustrated in figure 9, the single row of cosines (0.90, 0.12, 0.10) against the three “a photo of a {class}” prompts, scaled and softmaxed, picks cat. The training matrix and the inference computation are the same operation; that is the elegance.

The prompt template is not a throwaway detail. Bare class words like “cat” underperform “a photo of a cat”, which better matches the distribution of web captions the text encoder learned from. The template alone lifts ImageNet zero-shot accuracy by about 1.3 points.

Because the classifier is synthesized from text, the set of classes is open. We can classify categories CLIP was never explicitly trained on, just by writing them as prompts. Our toy matrix is clean by construction, though. Does this recipe actually hold at the scale of the real internet?

The numbers that made CLIP famous

The toy matrix proves the mechanism; the real numbers prove it matters. Trained on about 400 million pairs, zero-shot CLIP matched a fully supervised ImageNet model while using zero ImageNet labels, and the contrastive recipe got there far more efficiently than the alternatives.

As shown in figure 10, the accuracy jump is the headline. The team’s early bag-of-words proof of concept reached only 11.5% zero-shot on ImageNet. Full contrastive CLIP reached 76.2%, matching a supervised ResNet-50 trained on 1.28 million labeled images, with zero ImageNet labels.

Figure 10. Zero-shot ImageNet top-1 accuracy. The early bag-of-words proof of concept reached 11.5%; full contrastive CLIP reached 76.2%, matching a supervised ResNet-50 trained on 1.28M labeled images while using zero ImageNet labels.

As illustrated in figure 10, CLIP closed the entire gap to a supervised baseline without ever seeing an ImageNet label. The robustness story is just as striking: on ImageNet variants like ImageNet-R and ImageNet-Sketch, zero-shot CLIP closes up to 75% of the accuracy gap that sinks standard supervised models.

The efficiency story explains how they got there. As shown in figure 11, the gain came in two multiplicative steps. Switching the objective from predicting the exact caption to predicting a bag of words gave a 3x efficiency gain; switching from prediction to the contrastive objective gave a further 4x.

Figure 11. The training-efficiency stack. Moving from predicting the exact caption to a bag-of-words objective gave a 3x gain; moving to the contrastive objective gave a further 4x, roughly 12x more efficient at the same zero-shot accuracy.

As illustrated in figure 11, the two steps compound to roughly 12x, about an order of magnitude more efficient at reaching the same zero-shot accuracy. The simple grid we built scales to the connective tissue of modern multimodal AI.

CLIP does not generate. A common misconception is that CLIP makes images. It does not. CLIP only scores how well an image and a text match. Generative systems like DALL-E 2 and Stable Diffusion use CLIP, but CLIP itself only embeds and compares.

Summary

CLIP is one of those ideas that looks inevitable once you have seen the grid. Here is what to take away.

• One shared space: CLIP trains an image encoder and a text encoder together so a picture and its caption land at nearly the same point in one 512-dim space.
• Cosine in a grid: L2-normalizing both vectors turns the dot product into cosine similarity, and comparing every image to every caption fills a (3, 3) similarity matrix.
• Training is a shape: the contrastive, symmetric InfoNCE loss pulls the diagonal up and pushes the off-diagonal down, turning a near-uniform grid into a near-identity one.
• Temperature sharpens: a single learned scalar (logit_scale about 14.29) stretches a 0.90 versus 0.10 cosine gap into a 99.99% confident softmax.
• One operation, two jobs: zero-shot classification reuses one row of that exact matrix at test time, so CLIP recognizes classes it was never explicitly trained on, reaching 76.2% on ImageNet with zero ImageNet labels.

We just built a model that can match an image to text. The next leap is a model that can talk about an image. In the next piece we wire a CLIP-style vision encoder into a language model to build a nano-VLM from scratch. Spoiler: the same shared-space trick is what lets the language model see.

If you found this helpful, subscribe to follow the multimodal series and get the nano-VLM build the moment it drops. You can go deeper with our hands-on VLM and multimodal bootcamp at vizuara.ai, and we will share the full code in our community, including the runnable 3x3 similarity-matrix demo and the zero-shot “a photo of a {class}” classifier so you can reproduce the payoff in a few minutes.