anonypii: A PyPI package for securing the user data in LLM applications

In part one, I built PIIBench and showed that every existing PII detection system falls apart under a realistic cross-domain evaluation.

In part two, I fine-tuned DeBERTa on the full PIIBench training set and reached F1 0.6455 on the held-out test — a 3.76x improvement over the best published alternative in the market.

Both models ended up on HuggingFace. And that’s where a lot of research stops.

A model on HuggingFace is not a compliance control. An enterprise team that wants to protect customer data before it hits an LLM needs something they can pip install and call from production code. They need it to be deterministic, predictable, and to fit into their existing inference pattern without requiring them to rebuild their application flow.

So the third piece of this work was building anonypii.

The Production Pattern

The integration problem for GenAI applications is specific. The flow looks like this:

1. User sends a message containing PII
2. Application receives it and needs to forward it to an LLM (OpenAI, Claude, whatever)
3. PII cannot leave the perimeter
4. LLM responds
5. Response goes back to the user

The naive approach is to redact before sending and never recover. Replace “John Smith’s account #4892-XXXX” with “[PERSON]’s account #[ACCOUNT_NUMBER]” and send that to the LLM. The model responds about “[PERSON]’s account” and you return that to the user.

This works for compliance but breaks user experience. If the user asked “What’s wrong with my account?” and gets back “There is a hold on [PERSON]’s account #[ACCOUNT_NUMBER]”, they’re confused. The answer is correct but it looks broken.

The right pattern is:

1. Detect PII entities in the input
2. Replace each entity with a placeholder: {{PERSON_1}}, {{ACCOUNT_NUMBER_1}}, etc.
3. Store the mapping (the “vault”)
4. Send the redacted text to the LLM
5. Receive the LLM response, which contains the same placeholders
6. Replace the placeholders back with the original values
7. Return a response to the user that reads normally

This is the vault approach. The PII never leaves the system, the LLM sees sanitized text, and the user gets a coherent response.

What anonypii Does

anonypii is a Python package built on top of the PIIBench-trained DeBERTa models. It handles three use cases:

Detection — identify PII spans and their types in a piece of text. Returns entity labels with positions.

Masking — replace PII spans with type-labeled markers. Input: raw text. Output: redacted text.

Vault — the full mask-and-unmask cycle. Input: raw text. Output: redacted text plus a vault object. After the LLM call, pass the LLM output and the vault to the unmask function and get back the restored text.

Installation:

pip install anonypii[all]

The package pulls the model from HuggingFace on first use. Subsequent calls use the cached weights. No external API calls, no data leaving your environment.

Why Encoder-Only Models Make This Practical

The vault pattern has a requirement the naive masking approach doesn’t: the placeholders inserted before the LLM call need to survive the LLM’s response intact. An LLM might paraphrase, rephrase, or restructure the content. But if the placeholder format is consistent and the model is instructed to preserve them, it works reliably.

This is only tractable because the detection step is deterministic. An encoder model running token classification will tag the same input identically on every call. You can build a system around that. A decoder-only generative model doing PII detection would produce different entity spans on repeated identical inputs — the vault mapping would be unreliable.

The determinism of the encoder is not a minor implementation detail. It’s what makes the vault approach viable in production.

The End-to-End Picture

To close the loop on where this work started: the original problem was that GenAI applications needed a way to scrub PII before sending data to cloud LLMs, and the available tools were either too brittle (regex), too expensive and non-deterministic (on-prem LLMs), or too narrow in their coverage (siloed encoder models).

The work went through three stages:

First, I established a common evaluation ground. PIIBench consolidated ten heterogeneous datasets into a single benchmark and revealed that the best available systems peak at F1 0.14 in realistic cross-domain evaluation. The research community now has a shared testbed instead of competing claims made on incompatible private splits.

Second, I showed that a DeBERTa model fine-tuned on the full PIIBench training corpus reaches F1 0.6455 on the held-out test — closing most of the gap between what was available and what production systems actually need. The key finding was that data quality and diversity matter more than architectural complexity.

Third, anonypii packages those models into a form that engineering teams can actually use: detect, mask, and the vault pattern for seamless LLM integration.

Links

PIIBench dataset: https://huggingface.co/datasets/Pritesh-2711/pii-bench

PIIBench paper (benchmark): https://arxiv.org/abs/2604.15776

DeBERTa fine-tuning paper: https://arxiv.org/abs/2605.25816

Models on HuggingFace:

• https://huggingface.co/Pritesh-2711/piibench-deberta-base (direct fine-tune, recommended)
• https://huggingface.co/Pritesh-2711/piibench-deberta-sch (SC+H variant)

anonypii:

• PyPI: https://pypi.org/project/anonypii/
• GitHub: https://github.com/pritesh-2711/anonypii

pii-bench GitHub (code, benchmark): https://github.com/pritesh-2711/pii-bench