Semantic Font Search at Cricut

The Problem

Cricut has thousands of high quality fonts. If you know the exact name or the right tag, you’re golden. But what if you just want “something playful and rounded for a kid’s birthday card” or “a classy serif for formal invitations”? Traditional search can’t help much, it’s locked into exact keywords.

Meanwhile, CLIP (Contrastive Language-Image Pre-Training) is a powerful ML tool which matches images to text. However, it was trained on general internet photos, not fonts. Out of the box, it has no idea what “serif” or “script” really means in typography (well, as far as ML models can ‘understand’ anything).

So we taught CLIP to speak font.

A Quick CLIP Primer

Before we dive into the font stuff, let’s talk about how CLIP works. (For the deep technical dive, check out OpenAI’s documentation.)

CLIP is trained on millions of image-caption pairs scraped from the internet. The magic happens through two separate encoders:

• Image encoder: Takes a picture and spits out a vector of numbers (an “embedding”).
• Text encoder: Takes a caption and also spits out a vector.

At first, these vectors are meaningless to each other — they point in completely different directions. But through training, CLIP learns to push matching pairs (image + its caption) closer together and pull mismatched pairs apart. Eventually, the image of a hedgehog and the word “hedgehog” end up near each other in this shared embedding space. Other text like “porcupine” will be close to hedgehog, and “school bus” will be totally separate. In the middle, you get something like a horse (which is both an animal and a mode of transportation).

The picture of hedgehog is physically close to the word ‘Hedgehog’

Once trained, you can:

• Encode any image and find the closest text descriptions.
• Encode any text and find the closest matching images.

That’s the power we wanted to harness for fonts — but first, we had to teach CLIP what fonts actually are.

What We Did (The Big Picture)

To make this work, we needed three things:

1. Images that capture how each font actually looks
2. Captions that describe fonts the way humans search for them
3. Training that teaches CLIP to connect the two

Let’s break down how we tackled each of them:

Step 1: Images

CLIP wasn’t trained on a single image of a dog; in fact, it was probably trained on thousands of different images of dogs of all shapes and sizes to get a more general understanding. We decided to provide 50 images per font — 25 real words and 25 words with random letters. This spread gives the model examples of uppercase, lowercase, punctuation, and kerning, so it learns the font’s ‘personality’ rather than memorizing one specific word.

Real Word

Random Letters of Random Length (between 5–15 characters)

After rendering the text, we stacked them into a grid, padded and resized to 224×224 (CLIP’s required image size). We also applied random transforms during training (slight rotations, translations, and scaling) so the model would recognize fonts even when they’re not perfectly aligned or sized.

Wrapped, cropped version of the random letters

Step 2: Writing Captions

We needed captions that sound like real search queries, not just font names. So we used OpenAI’s GPT-4o to analyze fonts and score them on attributes that people actually search for:

• Stylistic (8 attributes): weight, serif, script, geometric, rounded, decorative, modern, playful
• Thematic (10 custom attributes): wedding, horror, corporate, handwriting, presentation slides, etc.

Note: something like GPT-4o is powerful for analyzing images and understanding nuance, but is computationally (and financially) expensive. Because of these reasons, we use it once during training to fill out a more extensive dataset and let the embeddings do the repetitive work.

GPT-4o rated each font on a 0-100 scale for every attribute. A score of 0 means ‘not at all,’ 50 is neutral, and 100 is ‘extremely.’ For example, a bold blackletter font might score 95 on ‘decorative’ but only 10 on ‘modern.’

Then we built caption templates like:

• ‘A font with {attributes} characteristics’
• ‘This is a {attributes} font’

For each training image, we randomly picked 1–3 attributes to fill in, but here’s the trick: we favored attributes far from neutral (scores near 0 or 100) because those are the most visually distinctive features. We also swapped in synonyms (‘bold’ ‚ ‘heavy,’ ‘thin’ ‚ ‘light’) to prevent the model from just memorizing exact phrases.

Step 3: Training the Model

We started with a pretrained CLIP model and froze most of its weights — think of it like taking a trained chef and only teaching them one new cuisine instead of culinary school from scratch. We finetuned only the last few layers of the image encoder (the part that ‘sees’ fonts), while the text encoder stayed frozen so it wouldn’t forget how to understand English.

How we trained it:

The model learns by comparing pairs:

Positive pairs (image + matching caption): We told the model, ‘These should be close together!’ and gave extra emphasis when attributes were especially distinctive (like ‘very serif’ or ‘extremely decorative’).

Negative pairs (image + wrong caption): We told the model, ‘Push these apart!’ This was especially important for opposite styles like ‘serif’ vs ‘sans serif’ or ‘playful’ vs ‘formal.’

We also used hard negatives (deliberately tricky examples with opposite attributes) to teach the model what fonts aren’t. For instance, pairing a minimalist sans serif with the caption ‘very decorative and ornate.’

We tracked Top-k accuracy (does the right font show up in the top results?) and Negative alignment (are opposite-style captions ranked low?).

Experiments: What Worked (and What Didn’t)

We ran about 12 rounds of experiments. Here’s the journey:

Rounds 1–3 (early wins, then collapse):

Simple captions with font names and tags worked great — over 90% accuracy! However, that meant the model was just memorizing names, not learning what fonts look like. When we removed font names and went pure attribute-based, the model either collapsed (bad embeddings) or gave every font nearly identical embeddings (also bad).

Rounds 4–7 (trying everything):

We experimented with different similarity losses, more negatives, more synonyms. Progress was slow. Some configurations destabilized when we trained both encoders together.

Rounds 8–12 (the breakthroughs):

Three big changes unlocked success:

1. We asked the LLM to score attributes 0–100, then used those scores to weight our caption generation. A font with serif=95 would almost always get ‘serif’ in its caption, while a font with serif=50 (neutral) would rarely mention it. This helped the model focus on distinctive features.
2. Using hard negatives to aid with contrastive loss (“very decorative” vs “minimal and clean”)
3. Leave the text encoder alone, only train the image encoder half of the model

We also added the custom theme attributes (like “wedding” and “scary”) from another LLM pass. This generalized the model across more use cases.

The result? Stable training and embeddings that actually worked for unseen queries.

Results: Does It Actually Work?

Yes! Here are some real examples:

Query: “serif”

Clean, classic serif fonts ranked at the top.

Query: “halloween”

Spooky, dramatic fonts perfect for horror themes.

Query: “wedding invite”

Elegant scripts and formal serifs — exactly what you’d want.

Query: “modern sans serif”

Contemporary, minimal fonts that feel fresh and current.

It also handles compound queries like “bold playful sans for kids’ party” without breaking a sweat.

Conclusion

Teaching CLIP to understand fonts turned out to be a surprisingly effective way to bridge the gap between how people think about typography and how computers search for it. By combining rendered font samples with attribute-rich captions and some clever training tricks, we built a system that actually “gets” what you mean when you search for “bold playful sans” or “elegant wedding script.”