The 100% That Leaked: Building a PHI Firewall for Clinical AI Agents

How an adversarial detector that scored perfectly in tests still leaked patient data in production, and what a real evaluation team broke next.

TL;DR

I built ClinicSentry, an open source compliance layer that sits between an AI agent and the outside world in regulated clinical settings. Along the way a PHI detector that scored 100 percent recall in isolation leaked almost everything on the real scan path, because its matches were computed in normalized text and never mapped back to the original. Fixing it with offset preserving normalization took end to end adversarial recall from 42.9 percent to 100 percent across eight evasion families, with no extra model calls and sub millisecond overhead. The transferable lesson: benchmark the production entry point, not the component.

The library was also shaped by putting it in front of a team evaluating it for a real healthcare chatbot. They came back with a specific, useful list of what broke in their setup, and the library now handles all of it: an allowlist so a public support number does not get redacted next to a patient’s private one, detectors for device serial numbers and other identifiers a naive firewall misses, a decorator to register your own detector without forking the code, and more.

pip install clinicsentry. Repo at GitHub.

A disclaimer up front, because it matters in this domain. ClinicSentry is not a medical device, is not FDA cleared, and does not make any system compliant on its own. Its controls align with regulation. Alignment is not certification.

PHI is scrubbed before the model sees it; every action is tier-checked and every event is signed into the audit chain, producing a report an auditor can read.

Why a guardrail library for clinical agents

LLM agents now summarize clinical notes, triage patients, query records through tool calls, and hand intermediate results between specialized sub agents. Every one of those steps is a place where protected health information can leak, where an uncertain output can be acted on without a human, and where a regulated organization later has to prove what happened.

The existing tooling does not fit this shape. Conversational guardrail frameworks like NeMo Guardrails and Guardrails AI shape what a model says, often by spending one or more extra LLM calls per check. De-identification tools like Microsoft Presidio map a string to a redacted string. Neither tracks where a piece of PHI travels across a multi agent pipeline, attaches it to a tamper evident audit trail, or maps controls to specific clauses of HIPAA, the FDA Total Product Lifecycle framework, IEC 62304, and the EU AI Act.

So in most projects this logic gets re-implemented as a thin wrapper that regexes for Social Security Numbers and logs to a file. That is fragile in three ways. Naive detectors are trivially evaded. Nothing tracks where detected PHI goes. And a log that can be silently truncated is no evidence at all under 21 CFR Part 11.

ClinicSentry is a single, lightweight dependency that addresses those gaps. It composes four independent controls behind one facade.

The four controls

Here is the failure I did not expect, and the reason I wrote this post.

A recurring hazard in security tooling is that a component passes its own unit tests but is not wired into the path that matters. I hit exactly that.

I wrote an adversarial normalizer. It strips zero width characters, maps Unicode homoglyphs to ASCII, applies NFKC normalization, and decodes percent escapes. Tested directly, it scored 100 percent recall on the evasion corpus.

Then I submitted PHI to the firewall’s public scan method, the exact call a library user makes. It passed through unredacted.

Two things were wrong. First, the firewall was only running the raw regex detector on that path. Second, and more subtle, even when the normalizer ran, its hits indexed into normalized text. A match at position 12 in the cleaned string does not correspond to position 12 in the original, which still contains the invisible characters. There was no valid span to redact, so nothing was removed. The detector was correct and the data still leaked.

The fix has two parts. Normalization now produces an explicit offset map. For each character of the normalized string, it records the half open span of the original it came from. A hit in normalized space maps back to the union of contributing original spans, so redaction removes the entire obfuscated region including the stripped invisibles. And the firewall now runs the normalizing detector on the production path, with a plain ASCII fast path that skips the work when the input is already clean.

The result, measured on the public scan method rather than the component:

The lesson generalizes well beyond this library. The isolated 100 percent was misleading. Only by evaluating the entry point that users actually invoke did the leak surface. I added a firewall path comparator to the benchmark harness so that any future detector change is measured where it ships.

What it costs

The library is meant to add negligible overhead and zero extra model calls. Over synthetic workloads, a firewall scan finishes with a p95 between 0.04 and 0.09 ms, and an audit append near 0.015 ms. On a roughly 3 KB clinical note a scan finishes in about 0.7 ms at p95. The dominant cost in any real deployment is still the LLM call, which ClinicSentry does not duplicate.

On an end to end synthetic workflow with 10 PHI tokens, an unprotected pipeline leaked 9 of them downstream. The protected pipeline leaked 1, an 88.9 percent relative reduction, at 2.4 ms added per note, and it produced a verifiable audit trail at no extra model call.

Source: ClinicSentry benchmark harness, synthetic workloads. The dominant cost in any real deployment is still the LLM call, which ClinicSentry does not duplicate.

What a real evaluation found next

The offset bug was the failure I found by testing my own entry point. The next round came from someone else testing theirs.

A team evaluating ClinicSentry as the compliance layer for a healthcare RAG chatbot ran it against real support transcripts and sent back a specific list. Two findings stood out because they were both obvious in hindsight and easy to miss while building.

First, the firewall redacted every phone number shape it saw, including the manufacturer’s own toll free support line printed in the bot’s own replies. A public hotline is not PHI under HIPAA, it is not tied to an individual, but the detector had no way to know the difference between that and a patient’s personal number. Second, there was no supported way to teach the library a proprietary identifier format, like a device serial number in a specific company’s format, without reaching into internal, undocumented classes.

The library handles both. An allowlist lets you mark specific values as known-safe per PHI type, so a support number stays readable while a patient’s number next to it still gets redacted:

guard = ClinicSentry(policy={"phi_firewall": {
    "allowlist": {"PHONE": ["1-800-555-0147"]}
}})

scan = guard.firewall.scan(
    "Call support at 1-800-555-0147 or reach me at 415-555-1234"
)
print(scan.redacted)
# -> "Call support at 1-800-555-0147 or reach me at [REDACTED:PHONE]"

And a decorator turns any span-matching function into a first class detector, no fork required:

from clinicsentry import register_detector
import re

@register_detector(label="DEVICE_SERIAL")
def device_serial(text: str):
    for m in re.finditer(r"\b(?:SN|serial)[\s:#-]*([A-Z0-9]{6,})\b", text, re.I):
        yield m.start(), m.end(), 0.95

guard = ClinicSentry(detectors=[device_serial])

The library covers a gap the 18-category HIPAA Safe Harbor list calls out directly and naive firewalls miss: device identifiers and serial numbers are detected out of the box and documented in a published coverage matrix, alongside fax numbers, account numbers, and health plan member IDs. It ships locale packs with real checksum validation for non-US identifiers, like an NHS number check that runs the actual mod-11 algorithm instead of a shape-only regex. It supports reversible tokenization for pipelines that need to de-identify before a model call and restore the real value after. And the default install carries no native dependency, keeping HMAC-key signing behind an optional extra so the base package stays pure Python for serverless deployments.

None of that is a pivot. It is the same “test the entry point someone actually uses” habit from the offset bug, run by someone else’s actual usage instead of my own test suite.

A concrete example: a medication support bot

Consider a health system building a patient support chatbot for people on warfarin, an oral anticoagulant whose dose is adjusted from regular blood tests. This is illustrative, not a description of any real deployment. The bot should help with medication and monitoring questions. It must never give dosing advice, because that is interventional and life critical.

You encode that as a tier, not as a prompt instruction.

from clinicsentry import ClinicSentry, ClinicalRiskTier

guard = ClinicSentry(framework="anticoagulation-support-bot")

# Scrub PHI before the model sees anything
scan = guard.firewall.scan(
    {"message": "I am Jane Doe, MRN 12345678, my INR came back at 3.8"},
    origin_agent="patient_intake",
)
safe_message = scan.redacted["message"]   # -> "I am Jane Doe, [REDACTED:MRN], my INR came back at 3.8"
# note: catching the MRN needs no extra setup. Catching "Jane Doe" needs the
# optional NER extra (pip install 'clinicsentry[phi]') -- deterministic regex
# alone does not recover free-text names, and I would rather say that here
# than have you discover it in production.

# Answering monitoring questions is low risk
@guard.register_action(
    tier=ClinicalRiskTier.INFORMATIONAL,
    description="Answer medication and monitoring questions",
    required_fields={"message"},
)
def answer_support(payload): ...

# Changing a dose is interventional and always blocked
@guard.register_action(
    tier=ClinicalRiskTier.INTERVENTIONAL,
    description="Suggest a warfarin dose change",
    required_fields={"patient_profile", "proposed_change"},
)
def suggest_dose_change(payload): ...

decision = guard.evaluate_action(
    "suggest_dose_change",
    output_text="Increase your warfarin to 7.5 mg daily.",
)
print(decision.action)   # -> "block"

The dosing suggestion never reaches the patient. It is blocked, logged, signed into the audit chain, and routed to a human reviewer. At the end of the session a single command produces the compliance report that an auditor or a regulatory team can read.

clinicsentry report ./audit.sqlite --session <session-id> --format summary

Limitations I want to be honest about

This is research grade and I would rather state the gaps than oversell.

All the numbers above come from synthetic data. Real clinical text shifts the distribution, especially for free text names. The default detector is deterministic and does not recover person names or bare medical record numbers. Names need the optional neural recognizer, whose false positive rate on real clinical text I have not yet characterized on a public benchmark. The end to end leakage number comes from a small workflow and is indicative, not a population estimate. The audit chain protects against an honest operator who might truncate or edit a log, not against a malicious root user who holds the signing key in process, which is why production keys should live in an HSM or KMS.

Try it

pip install clinicsentry
clinicsentry scan "Patient Jane Doe, MRN 12345678, SSN 123-45-6789"
clinicsentry demo --audit-path /tmp/demo.sqlite
clinicsentry policy-schema          # the full config surface, generated not hand written

The library, the benchmark harness, and the configuration that produced every number here are open source under Apache 2.0. If you are building AI that touches patient data, I would genuinely like to hear which of the four controls maps to a problem you already have, or what your evaluation breaks that mine did not. That is how the library got better. Issues and stars at GitHub.

[embed]GitHub - aakash1411/clinicsentry Framework agnostic compliance middleware for clinical AI Agents.github.com