Navigating Compliance in the era of AI — The thin line between deterministic and probabilistic

generated by ChatGPT

Background

The burden on compliance teams is increasing, yet the budget and people capacity in the market to meet that demand and not following. New regulations, standards and other matters are appearing at a faster pace than before, and evidencing meeting the obligations in them is becoming increasingly detailed, time consuming and required spanning expertise and roles across the entire organisation, also making this a growing orchestration challenge.

This is creating a risk profile that increases for organisations over time, hence the push towards a combination of Automation, Generative and Agentic AI has been strong over the last few years.

Early efforts

Our first effort Agentic effort showed great initial promise, the AI system could read regulatory documents, identify obligations, search through internal policies, and suggest which business team should own them and what the coverage mapping should be. In a demo, it felt almost magical. A long regulatory document would come in, and within seconds the system could explain what changed, which department might be impacted, and what action needed to be taken. This initial promise encouraged us to continue but there were challenges ahead…

Enterprise Integration

When we connected the agents to actual enterprise workflows, the AI system was no longer just generating an answer on the screen. It was calling tools, searching internal knowledge bases, fetching policy documents, parsing regulatory text, preparing structured obligation records, and saving data into the compliance system. That is where the difference between “looks intelligent” and “works reliably” became very clear and the need for an enriched reliable experience including extensive use of meta-data.

As a worked example — the AI could correctly understand that a document contained a reporting obligation. But when it came to saving that obligation, the system needed much more than a good summary. It needed the exact regulation ID, regulatory reference, jurisdiction, applicable entity, effective date, reporting frequency, control owner, evidence requirement, and approval status etc. Missing even one of these metadata fields could create problems later during audit or regulatory review.

generated by ChatGPT

We realised that tool calls cannot be treated as a basic extension of the LLM. In our project, the AI was supporting use cases like: data extraction, classification, translation, regulatory search, and saving structured compliance data. The model could understand the regulation, but the outputs were rarely ready to enter the system of record. An extracted obligation still needed a valid regulatory reference. A classification still had to match approved categories. A translated requirement still had to preserve the regulatory meaning. A mapped obligation still had to be checked against the right taxonomy. And before anything was saved, it had to pass schema validation, duplicate checks, date normalization, and traceability rules. That was the key shift — AI helped us interpret the content, but deterministic controls decided what was reliable enough to store, reuse, and defend during audit.

The Working Formula

We now use AI for interpretation, but deterministic logic for execution. When the agent searched internal documents, we do not allow it to rely only on broad semantic search. We added deterministic metadata filters such as jurisdiction, entity type, business unit, regulation category and effective date. A search for “data retention requirement” could return many similar results, but the system had to make sure it was searching in the right country, for the right legal entity, and against the latest approved policy.

Parsing was also key. The AI could summarize a regulatory paragraph very well, but compliance teams needed exact references. So, we used deterministic parsers to break documents into sections, clauses, sub-clauses, annexures, tables, and footnotes. Clause numbers like 4.2(a) or references like Annexure B — Item 7 were captured through rule-based parsing before the AI extracts the content. This helps us avoid a common issue where the model understood the meaning but lost the exact source reference.

The same point applies to saving data. We never allow the model to directly write free-form output into the compliance system of record. Every obligation had to pass through a strict schema; dates had to be normalized. Frequencies had to map to approved values like monthly, quarterly, annual, or event-based. Owners had to be validated against master data. Regulation IDs and clause references had to follow expected patterns.

Impact

This changed the product completely. Before the deterministic layer, the AI felt smart but risky. After adding parsers, schemas, validations, workflow checks, and audit logs, it became something compliance teams could actually trust. The AI is still doing the heavy lifting of reading and interpreting complex regulatory language, but the system around it ensured that only validated, structured, and traceable information entered the workflow.

This is the thin line between probabilistic and deterministic systems in the era of LLMs. AI is excellent when the input is complex, lengthy, and full of context. It can understand regulatory language, summarise obligations, suggest mappings, and help teams move faster. But when the system has to search official records, call tools, save data, assign owners, trigger tasks, or create an audit trail, probability alone is not enough.

generated by ChatGPT

In enterprise AI systems, the LLM should not be the final authority. It should be part of a controlled system — Let the model interpret, let deterministic logic verify, let workflows govern and let humans approve where the risk is high.

Because in compliance, the final question is not just, “Did the AI understand the regulation?”. The real question is, “Can we prove what was extracted, where it came from, why it was saved, who approved it, and whether it was correct based on the information available at a historical point in time?”

That proof cannot come from a model response alone. It comes from the right balance between AI intelligence and deterministic engineering.

Co-Author

Huge thanks to co-author Aidan Caffrey for collaborating on this article, whose expertise and vision shaped this piece from the ground up.