Silent Evolution of Zero-Shot Encoders. From UniNER to GLINER 2

generated by nanobanana2

Intro

Large language models demonstrate remarkable generalization, not only for text generation tasks such as QA assistants, but including classification, named entity recognition and other NLP tasks. On the one hand we are in the start of agentic era, where applications with their tools are managed by Large Language Models, helping us to solve complex tasks. On the other hand — complex tasks with complex solutions and resource-hungry LLM algorithms, require data center — level hardware or vendor lock in with external API dependencies. Both are expensive and there are some tasks like text classification, information extraction (e.g. named entities recognition), where we don’t need biased and hallucinated algorithm and better way — a focused, output-level deterministic model trained for the task with zero shot opportunities.

In this article we will explore evolution that you possibly missed out, relying on LLM’s in those simple tasks. This evolution of Multi Task encoders — called Gliner. Starting with open-domain zero shot NER, latest architecture could handle lots of tasks, focusing Information Extraction and text Classification. Killer feature is that backbones of Gliner mostly are encoder-based models like bert, deberta and modernbert. You don’t need more to solve multitasks, using high latency LLMs and thinking about infrastructure in terms of throughput and quality tradeoffs, no KV-cache, no token generation time, just single forward pass.

Gliner provides simple, but beautiful engineering solution, including all tradeoffs between speed, quality and generalization.

Now everyone can just write:

from gliner2 import GLiNER2
extractor = GLiNER2.from_pretrained("your-model-name")

and resolve 1, 2 or more tasks via single forward

schema = (extractor.create_schema()
    .entities({
        "person": "Names of people mentioned",
        "date": "Dates and time references"
    })
    .structure("appointment")
        .field("patient", dtype="str")
        .field("doctor", dtype="str")
        .field("date")
        .field("time")
        .field("type", dtype="str", choices=["checkup", "followup", "consultation"])
)

text = """
Dr. Sarah Johnson confirmed the appointment with John Smith for 
March 15th at 2:30 PM. This will be a follow-up consultation 
regarding his previous visit on February 1st.
"""
results = extractor.extract(text, schema)

So let’s explore and understand history behind the code above, and let me guide you through this.

Universal NER

In 2023, before GLiNER release, Zhou et al. demonstrated Universal NER — Open Domain NER distilled from ChatGPT and built on top of training other LLMs: Vicuna 7B and 13B, surpassing chatgpt in most cases. I don’t plan to focus on this model in terms of training or datasets collection, my target here is — collect core insights.

Core insights

Knowledge distillation

UniNER uses chatgpt to annotate given passages (texts) with following prompt:

Later the same prompt will be used in GLINER paper as reference to this work and accepting the efficiency.

UniNER actually uses hard label distillation, the same approach resurfaces in DeepSeek’s work. (I think you’ve heard about this story, so plz no holy war in comments XD)

Ultimately UniNER shows following results:

So:

Insight #1: Using Sota LLM (like chatgpt) as annotator and learning on hard labels is enough not only to approximate the teacher, but outperform it.

Single vs Multi label

UniNER evaluated 3 scenarios with 3 different models:

• type — single entity only per forward pass (e.g. “PERSON”)
• all in one — couple of entities in single forward pass (e.g. “PERSON”, “LOCATION”, “ORGANIZATION”)
• definition — only definitions generated for entities by chatgpt

As you see single label per single forward pass model outperforms others in most cases. So keep in mind this tradeoff:

Insight #2: More labels and generalization ≈ lower accuracy

Still Autoregressive

While authors of UniNER created and shared a couple of models, those still not just respond with found entities in given text. They generate it in inference, so we’re still stuck not only with forward pass, but with decoding pipeline of text generation. So we need to deal with KV-cache and other text generation issues. So my insight is not insight it’s more questionable:

Question 1 Do we need text generation for open domain task with custom entities?

And that’s exactly what the next paper answers.

GLiNER

*GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer. by Urchade Zaratiana et. all*

Intro

This paper just began with following issues:

1. Traditional NER models are limited to a predefined set of entity types. New entities require new annotations.
2. Large Language Models (starting from paper “Large Language Models are Few-shot learners”) achieve high generalization capability with comparable accuracy. Meanwhile they consist of Billions of parameters and inference or api costs are high.

Gliner provides small bidirectional Language Model (bert-like), where entity types matching with text spans in latent space. So it’s not just token classification bert, like from transformers library. It’s Span Categorizer, which works not with tokens, it works with spans (the text sequences — parts of passage).

How it works

So as input model needs:

1. Input sentence (or passage)
2. Prompted entity types as list of labels Both are passed through BERT model and from BERT output we could get embeddings space representaion.
3. Those representations are separated (trained to be separated) inside the Network:

entities: with FFN layer

original sentence is split to spans with window of up to 12 tokens inside the span

4. Finally GliNER matches span representations with given entities:

span (i,j) means part of sentence, e.g.: span (0,2) = "John Smith"

And as you could see we could use these phi as probablity, and here it is:

from gliner import GLiNER

model = GLiNER.from_pretrained("urchade/gliner_multi-v2.1")

text = """

Cristiano Ronaldo dos Santos Aveiro (Portuguese pronunciation: [kɾiʃˈtjɐnu ʁɔˈnaldu]; born 5 February 1985) ...
"""

labels = ["person", "award", "date", "competitions", "teams"]
entities = model.predict_entities(text, labels)

print(entities)
# {'start': 1,
# 'end': 36,
# 'text': 'Cristiano Ronaldo dos Santos Aveiro',
# 'label': 'person',
# 'score': 0.9164333343505859},
# {'start': 92,
# 'end': 107,
# 'text': '5 February 1985',
# 'label': 'date',
# 'score': 0.9581286311149597},

'score': 0.9164333343505859},

Here is the phi from formula above.

Core insights

Knowledge Distillation from ChatGPT

as in UniNER:

Not only works in terms of getting annotations.

It just works still in terms to get great results with only 459M model with deberta-v3 backbone:

Insight #3: Insight #1 about hard labels distillation, still works with encoder architecture

Backbones

Original GLINER paper evaluated different existing backbones:

As we see Deberta backbone is the best at the moment of paper release. And empirically still the best one for all models on gliner-like architecture

deberta-v3 backbone still used by GLINER 2 and GliClass (we'll see them soon) as one of the most powerful encoders now

Matching spans, not generating text

As we know from How it works part — gliner just makes the span matching between given text and list of labels. So here’s the solution for an issue of LLMs: Gliner just made text generation task not needed for open-domain NER, outperforming previous SOTA models with lower parameters.

Insight #4: 459M GLiNER > UniNER 7B > ChatGPT (~100B+) outperforming SOTA, no autoregressive decoder required.

GLiNER Ecosystem

Exploring the zoo

GLiNER architecture turned out to be more than just a NER solution. The core idea — matching text spans with label embeddings in latent space — was simple enough to adapt to other NLP tasks. Community picked it up fast.

GliREL applied the same approach to relation extraction. GLinker extended it to entity linking. GliClass adapted span matching for text classification matching full text with labels. Each of these works essentially asked the same question: “what if we replace the task-specific head but keep the encoder backbone and zero-shot matching idea?”

It worked. Every time.

But this created a new problem: a zoo of single-task models, each requiring its own inference pipeline, its own fine-tuning recipe, its own deployment setup. You needed GLiNER for NER, GliClass for classification, GliREL for relations. Three models, three forward passes, three things to maintain.

This is exactly the problem GLiNER 2 was built to solve.

GLiNER 2

*GLiNER2: An Efficient Multi-Task Information Extraction System with Schema-Driven Interface, by Urchade Zaratiana et. all*

GLiNER 2 is not a new architecture. It’s a unification.

Zaratiana et al. took everything that emerged from the GLiNER ecosystem — NER, classification, relation extraction, entity linking — and merged it into a single framework with a single forward pass. The zoo of models becomes one model with a schema-driven interface, which you already saw at the beginning of this article.

Meanwhile deberta-v3 backbone is still here, given that ModernBERT is already released and seems to be faster. One of possible thing, why autors still stay with deberta — Disentangled Attention, provided by Pengcheng He et. all Which gives more comparable results on span and text matching task, especially in NER tasks.

What changed

Multitask in one forward pass. Instead of running separate models for each task, GLiNER 2 handles them simultaneously. You define a schema, pass it with your text, get structured output back.

Extended context. Authors expanded the context length, allowing more text and more labels per inference — important when you’re doing multitask extraction on longer documents.

Label descriptions. Like the definition variant in UniNER, GLiNER 2 supports rich label descriptions in the forward pass, improving accuracy on ambiguous or domain-specific entity types.

Tradeoffs

Unifying tasks into one model introduces its own tradeoffs

Beating GliClass and deberta-v3, but not outperforming now GPT-4o, unlike original GLiNER.

In zero shot NER quality decreased:

Do you remember one of UniNER insights?

Insight #2: More labels and generalization ≈ lower accuracy

So we moved back to this now, but for GLiNER 2:

Insight #5: Multitask unification costs single-task accuracy. GLiNER 2 NER quality is lower than GLiNER 1, classification is strong but doesn’t beat GPT-4o.

Performance speed in comparison to deberta-v3 and gpt-4o is significantly better and degrades slower, but it degrades, yeah.

GLiNER 2 is focused on CPU deployment, but actually as we know for high-throughput performance you’ll actually need the Nvidia GPUs.

We started with a 175B ChatGPT, ended with a 205M encoder doing the same job not better, but comparable as engineering tradeoff. That’s the evolution you possibly missed.

Key Takeaways

Insight #1: Using a SOTA LLM as an annotator and training on its hard labels is enough not only to approximate the teacher, but outperform it.

Insight #2: More labels and generalization ≈ lower accuracy.

Insight #3: Hard label distillation from LLMs transfers effectively even to encoder-only architectures — no autoregressive decoder required. (Hi, Insight 1)

Insight #4: 459M GLiNER > UniNER 7B > ChatGPT (~175B) — outperforming SOTA, no autoregressive decoder required.

Insight #5: Multitask unification costs single-task accuracy. GLiNER 2 NER quality is lower than GLiNER 1, classification is strong but doesn’t beat GPT-4o. (Hi, Insight 2)