Fine-Tuning Beats Architecture: How a Simple DeBERTa Model Solved PII Detection

In part one, I described building PIIBench — a standardized benchmark for PII detection across ten heterogeneous datasets. The result was uncomfortable: every published system achieved span-level F1 below 0.14 on the benchmark. The best tool available, Microsoft Presidio, was still missing most PII types entirely.

The benchmark quantified the problem. Now I needed a solution.

The requirement is specific:

• the model has to run on-premise (no cloud inference),
• be deterministic (same input, same output every time),
• cover the full range of PII types in real enterprise text,
• and be practical to deploy without a dedicated GPU cluster per team.

That rules out LLMs. Encoder-only token classifiers are the right tool.

The question was whether fine-tuning on a properly constructed multi-source dataset could actually close the gap — and by how much.

Why DeBERTa

DeBERTa (Decoding-Enhanced BERT with Disentangled Attention) is an encoder-only transformer from Microsoft. The v3 variant uses an improved pre-training approach that gives it stronger representation quality than standard BERT, particularly on token-level tasks. For BIO token classification — which is exactly what PII detection is — it’s one of the strongest available backbones that remains practical to deploy.

The task is straightforward: given a sequence of tokens, classify each one as either outside (O) or the beginning/inside of a specific PII entity type (B-PERSON, I-PERSON, B-EMAIL, I-EMAIL, etc.). The model sees context on both sides of each token, which is critical for resolving ambiguous cases. Is “John” a person name or part of “Johns Hopkins”? Is “Apple” an organization or the fruit? Context resolves this in a way that regex never can.

Three Approaches, Increasing in Complexity

Rather than just fine-tuning and calling it done, I ran a controlled experiment across three model variants. The motivation was genuine — I wanted to know whether architectural complexity actually helps, or whether the data is doing all the work.

Model A: Direct Fine-Tuned DeBERTa

The simplest approach. Take microsoft/deberta-v3-base, attach a linear classification head that maps each token's representation to one of 165 BIO labels (82 entity types × B/I prefix + O), train on the full multi-source PIIBench training split with weighted cross-entropy.

One detail here is critical. The training data is 83.3% outside tokens — non-PII words. If you train with uniform cross-entropy, the model learns to predict O for everything and gets acceptable loss while achieving zero precision and recall on actual entities. The fix is simple: weight O tokens at 0.1 and entity tokens at 1.0. Every reported model uses this weighting.

Model B: Source-Conditioned Hierarchical (SC+H)

This adds two components. First, a learned source token prepended to each input sequence — [SRC=ai4privacy], [SRC=finer_139], etc. — so the encoder is aware of which dataset the example came from. At inference on new text, [SRC=general] is used. Second, a coarse-to-fine hierarchical classifier: a first head predicts one of ten coarse entity groups (PERSON_GROUP, CREDENTIAL, FINANCIAL_ID, NETWORK, etc.), and its output probabilities are concatenated with the DeBERTa hidden states before the fine-grained BIO head.

The training objective combines both: L = L_fine + 0.3 × L_coarse.

Model C: SC+H with Curriculum Learning

The most complex variant. Takes the SC+H architecture and trains it across three sequential phases, each one epoch, ordered by source family: General NER → Synthetic PII → Financial PII. The hypothesis is that structured progression from simple to complex entity types would build better representations.

The Data Issue I Had to Fix First

Before training, I found parsing errors in the original PIIBench preparation. The NVIDIA Nemotron-PII dataset stores entity annotations as XML-tagged text — <PERSON>John</PERSON> — and the original pipeline hadn't parsed this correctly, producing incorrect BIO labels for those records. Nemotron was also under-represented in the original split.

I rebuilt the pipeline with corrected Nemotron parsing, rebalanced Nemotron to approximately 10% of the corpus through stratified sampling, and re-split everything. The resulting training set is 799,948 records with a 100,002-record held-out test split.

Because the split membership changed, I also re-ran all eight published comparator systems on a new 5,000-record stratified subset (test_5k) for a clean apples-to-apples comparison with the newly trained models.

Results: The Simple Model Wins

DeBerta model fintuned on piibench dataset vs rest of the market

Direct fine-tuned DeBERTa reaches F1 0.6476 on test_5k — an absolute gain of +0.475 over the best published comparator. That's a 3.76x relative improvement.

To confirm this on a larger scale, I ran both Direct DeBERTa and SC+H against the full 100,002-record held-out split:

Top 2 model results

Direct fine-tuning wins by 0.056 F1 across the full distribution. The simplest approach is the best.

Why Architecture Didn’t Help

The SC+H result deserves explanation. The source conditioning and hierarchical decoding weren’t useless — on 28 of 82 entity types, SC+H actually outperforms direct fine-tuning. Types like HTTP_COOKIE (+0.394 in SC+H's favor) benefit from knowing which source domain the text came from. These are contextually ambiguous types where provenance matters.

But those localized gains are swamped by the losses on high-frequency entity types. FINANCIAL_ENTITY (58,821 test instances), IP_ADDRESS (6,178), USERNAME (8,287), ACCOUNT_NUMBER (2,686), PHONE (1,535), SSN (1,261) — direct fine-tuning wins on all of them by significant margins. The +0.475 gap over published systems was mostly closed by the data, not the architecture.

The performance difference between Direct and SC+H (0.056 F1) is an order of magnitude smaller than the gap between direct fine-tuning and any published system (0.475 F1). The data preparation and loss weighting are doing the heavy lifting.

What the Curriculum Learning Failure Teaches

The curriculum experiment produced the most instructive result. Training progressed: General NER (F1 0.131 on fast-validation) → Synthetic PII (F1 0.430) → Financial PII (F1 0.305). F1 improved substantially through phase 2, then dropped after phase 3.

This is catastrophic forgetting. The financial-domain phase 3 overwrote the broader PII representations built in phase 2. Ironically, despite ending on financial domain training, SC+H+Curriculum performs worse than direct fine-tuning on financial identifier types like FINANCIAL_ENTITY and ACCOUNT_NUMBER.

The naive intuition — “train on easy stuff first, then harder stuff” — fails in this setting because the “harder” domain actively interferes with the “easier” domain’s weights. Mixed training on the full distribution from the start beats sequential specialization.

Entity-Level Highlights

To give a sense of what the model actually learned, here are some entity-type F1 scores for direct fine-tuned DeBERTa on the full test:

• PHONE: 0.9954
• VEHICLE: 0.9787
• MEDICAL_RECORD: 0.9658
• SSN: 0.9517
• JOB: 0.9249
• CREDIT_CARD: 0.9115
• ACCOUNT_NUMBER: 0.9095
• PERSON (first name): 0.9326
• CRYPTO_ADDRESS: 0.8641
• EMAIL: 0.6373
• IP_ADDRESS: 0.5528
• USERNAME: 0.5285
• FINANCIAL_ENTITY: 0.3229

The lower scores on USERNAME, EMAIL, IP_ADDRESS, and FINANCIAL_ENTITY reflect genuine difficulty: these types appear across wildly different source formats, and FINANCIAL_ENTITY covers 139 XBRL tag types that are structurally very different from conversational text.

What This Means Operationally

The practical takeaway is about priorities. If you’re building a PII detection system:

1. Get your data right. Multi-source, heterogeneous, correctly parsed. The data preparation quality contributes more to performance than any architectural choice.
2. Handle class imbalance explicitly. 83% outside tokens will kill your model if you use uniform cross-entropy.
3. Use mixed training, not sequential curriculum, when sources are diverse.
4. Don’t add architectural complexity before verifying it helps on your specific failure modes.

Both models are public on HuggingFace:

• Direct fine-tuned (recommended): https://huggingface.co/Pritesh-2711/piibench-deberta-base
• SC+H variant: https://huggingface.co/Pritesh-2711/piibench-deberta-sch

Paper: https://arxiv.org/abs/2605.25816 Code: https://github.com/pritesh-2711/pii-bench

Having the models on HuggingFace is necessary but not sufficient. A model sitting on a model hub doesn’t protect any user data. The next problem was making these models usable — particularly for the GenAI integration pattern where you want to scrub PII before the LLM and restore it after.

That’s part three.