How We Built an AI-Powered Data Governance Platform for 25,000 Financial Advisors

Using Amazon Bedrock AgentCore and FastAPI to automate exception tracking, reconciliation, and lifecycle management across financial data domains

Data governance in financial services is rarely glamorous. It’s a mix of exception queues that grow faster than teams can clear them, reconciliation breaks that reappear every Tuesday, and rule violations that get flagged but never quite get fixed. For a while, the answer was more people, more spreadsheets, and longer overnight batch windows.

We tried a different approach: build AI agents that could handle the routine work autonomously, surface only the genuinely hard decisions to humans, and do all of it inside a platform that the operations team could actually use.

This article walks through how we built the UDX Data Experience system — what the architecture looks like, the specific choices we made with Amazon Bedrock AgentCore, and what we learned along the way.

The Problem We Were Solving

Our data platform supports over 12000 financial advisors across five core domains: Account, Contact, Symbol, Allocation, and Transaction. Each domain has its own data sources, its own quality rules, and its own lifecycle patterns. When something goes wrong — a position value doesn’t reconcile between the custodian feed and the internal OMS, a KYC record is missing a required field, a transaction trips a daily limit rule — it creates an exception that someone has to look at.

At any given time, we were carrying hundreds of open exceptions. Most of them were low-risk and followed recognizable patterns: a pricing lag, a rounding difference, a stale vendor feed. But because every exception landed in the same queue, the team spent a disproportionate amount of time on issues that probably didn’t need human eyes at all.

The reconciliation picture was similar. We were running daily recon across all five domains against external custodians and data vendors. Match rates were solid — generally above 95% — but the remaining breaks required manual investigation, classification, and follow-up. It was repetitive work that rarely required judgment, but it took time.

Governance rules were managed in a configuration file. Adding a new rule meant a pull request. Testing it meant running it manually against a sample. There was no feedback loop, no visibility into which rules were generating the most noise, and no easy way for the operations team to adjust tolerances without involving engineering.

These weren’t catastrophic problems. The platform worked. But it wasn’t scaling well with the volume of data and the number of advisors we were supporting.

The Architecture We Chose

We settled on a three-layer design:

1. A FastAPI backend that handles all domain APIs — exception CRUD, reconciliation runs, governance rule management, lifecycle transitions, and domain entity access. It also manages WebSocket connections for real-time event streaming to the UI.

2. A set of specialized AI agents built on Amazon Bedrock AgentCore, each with a defined scope and a set of tools it can call. The agents don’t just generate text — they reason through problems, call tools to fetch or update data, observe the results, and continue reasoning until they reach a conclusion or decide to escalate.

3. An Angular-based UI integrated into our existing UDX platform, giving the operations team a single place to review exceptions, configure rules, monitor reconciliation results, and watch agent activity in real time.

The data layer underneath this is our existing canonical integrated layer — Apache Iceberg tables in S3, governed through data contracts and catalog policies, with Redshift and Snowflake serving downstream analytics.

The Four Bedrock AgentCore Agents

The most interesting part of this project was designing the agents. We built four, each focused on a specific operational area.

Exception Detective

This agent handles exception analysis and resolution. When a new exception is created — whether by a scheduled rule evaluation job or by a reconciliation run — the Exception Detective can be invoked to analyze it.

The agent’s system prompt gives it context about what kinds of exceptions it should auto-resolve versus escalate. It has access to tools that let it query historical exception patterns, look up the current state of the affected entity, apply a correction if confidence is high enough, and send a notification if escalation is needed.

In practice, it auto-resolves about 34% of daily exceptions. These are mostly pricing lags, rounding differences within tolerance, and timing issues where the custodian feed is a few minutes behind the internal snapshot. For the remaining exceptions, it adds an AI-generated analysis note — root cause hypothesis, affected entities, suggested resolution path — that gives the human reviewer a head start.

Reconciliation Maestro

This agent handles break analysis after a reconciliation run completes. It classifies breaks by type (timing, pricing, quantity, FX conversion, corporate action), looks for patterns across breaks, and generates a plain-language summary of what’s happening and what should be done about it.

It also flags breaks that are likely to self-resolve by the next business day versus breaks that need active intervention. This distinction matters a lot in practice — a settlement-pending break on a trade that was executed today doesn’t need anyone’s attention. A pricing break on a fixed income security where the vendor hasn’t updated the price in 18 hours probably does.

Governance Guardian

This agent evaluates governance rules against entity data and manages the rule lifecycle. It can run a full evaluation pass across all active rules, identify violations, classify them by severity, and trigger appropriate responses — auto-remediation for rules where that’s configured, notifications to relevant teams for rules that require human review.

It also monitors rule health over time. If a rule is generating a high volume of low-severity exceptions that keep getting suppressed, that’s a signal that the threshold might need adjustment. The Governance Guardian surfaces this kind of pattern and suggests specific rule changes, which a team member can review and approve.

Lifecycle Orchestrator

This agent manages entity state transitions. Moving an account from Active to Suspended, archiving a zero-balance position, deactivating a security that’s been delisted — these transitions have preconditions, downstream impacts, and audit requirements. The Lifecycle Orchestrator validates that preconditions are met, identifies cascade effects (closing an account affects its positions and pending transactions), and executes the transition with a full audit trail.

It also runs a regular scan for entities that are eligible for cleanup — zero-balance positions, inactive accounts that haven’t traded in over 180 days, securities that have been delisted but haven’t been marked inactive. It generates recommendations and queues them for team review rather than acting on them unilaterally.

How the Agentic Loop Works

All four agents use the same underlying pattern: the ReAct loop (Reason, Act, Observe, Repeat).

When you invoke an agent, it receives the task description and any context you’ve provided. It then reasons about what it needs to do, decides whether it needs to call a tool, calls it, observes the result, and continues. This continues until the agent has enough information to reach a conclusion, or until it hits the maximum iteration limit and escalates.

Here’s a simplified version of how this looks in our Bedrock invocation code:

while iteration < max_iterations:
    response = bedrock_runtime.invoke_model(
        modelId="anthropic.claude-sonnet-4-5",
        body=json.dumps({
            "anthropic_version": "bedrock-2023-05-31",
            "max_tokens": 4096,
            "system": agent_config["system_prompt"],
            "tools": GOVERNANCE_TOOLS,
            "messages": conversation_history
        })
    )
    stop_reason = response_body.get("stop_reason")
    if stop_reason == "end_turn":
        # Agent has reached a conclusion
        return extract_final_response(response_body)
    elif stop_reason == "tool_use":
        # Agent wants to call a tool
        tool_results = await execute_tool_calls(response_body["content"])
        conversation_history.append({"role": "user", "content": tool_results})

The tools themselves are straightforward functions — query the exception store, update an exception status, run a reconciliation comparison, evaluate a rule expression, send a notification. What makes the agentic pattern useful is that the model decides which tools to call, in what order, based on what it’s learned from previous tool results. A single agent invocation might call three or four tools before reaching a conclusion, with each tool call informed by the output of the last.

We built seven tools in total:

• analyze_exception — fetch exception details and historical patterns
• auto_resolve_exception — apply a resolution action with audit logging
• run_reconciliation — execute a comparison between source and target systems
• evaluate_governance_rule — test a rule expression against entity data
• manage_lifecycle_transition — validate and execute a state change
• query_data_store — general-purpose data retrieval across domains
• send_notification — route alerts to email, Slack, or dashboard

What the UI Needed to Support

The operations team needed a few specific things from the UI that shaped how we built it.

Exception review with AI context. When a team member opens an exception, they should immediately see the AI’s analysis — root cause hypothesis, affected entities, confidence score, suggested action. They shouldn’t have to dig for this information. We surface it inline in the exception detail view.

Bulk resolution with oversight. When the Exception Detective auto-resolves a batch of exceptions, the team should be able to see exactly what was resolved, by what logic, and with what confidence. We built a bulk action view that shows all AI-resolved exceptions with one-click audit trail access.

Rule management without pull requests. The operations team can now add, edit, and deactivate governance rules through the UI. Rules can be tested against a sample dataset before being activated. AI-enabled rules have an additional toggle that controls whether the Governance Guardian is allowed to auto-remediate violations.

Real-time agent activity. We added a live WebSocket feed that streams agent activity to the dashboard. When the Reconciliation Maestro finishes a run, or when the Exception Detective auto-resolves a batch, the team sees it immediately. This was more useful than we expected — it gives people a clear sense of what the agents are doing and builds appropriate trust in the automation.

Results After Rollout

A few numbers that are worth sharing:

34% of daily exceptions are now auto-resolved by the Exception Detective. These are handled end-to-end by the agent — analysis, resolution action, audit logging, notification — with no human involvement.

Cross-domain reconciliation match rate is 92.8%, up from a baseline of around 91.7% over the previous 30 days. Some of this improvement comes from operational changes the Reconciliation Maestro recommended (adjusting the custodian feed refresh timing), not just from the automation itself.

62% reduction in manual exception handling time for the operations team, measured against the pre-automation baseline. The remaining 38% is genuinely complex work — judgment calls, regulatory edge cases, client-specific situations — which is exactly what we wanted the team to be focused on.

Governance rule coverage is up. Before this project, the team managed about 30 active rules. We’re now running 47, because adding and testing rules no longer requires engineering involvement.

Things That Didn’t Go as Expected

A few things surprised us during the build.

Agent confidence calibration took more iteration than we expected. Our initial confidence thresholds for auto-resolution were too aggressive — the agent was resolving exceptions that, on review, should have had a human look at them. We went through two rounds of threshold adjustment before the auto-resolution set felt right. The lesson here is that you should plan for a calibration period after any agentic automation goes live, not treat it as a set-and-forget deployment.

The system prompt matters more than the tools. We spent most of our early design time on the tool definitions and comparatively little on the system prompts. In practice, the system prompt has more impact on agent behavior than the tool design. Being specific about escalation criteria, confidence thresholds, and what counts as a “pattern” versus a one-off exception made a significant difference in output quality.

WebSocket reliability required more engineering than anticipated. The live activity feed looked easy on paper. In production, managing connection drops, reconnection logic, and message ordering across a distributed deployment took meaningful effort. If you’re building real-time agent monitoring, budget time for this.

The operations team asked for more explainability, not less automation. We assumed the team would want a conservative auto-resolution threshold — let the AI handle only the obvious cases and defer everything else. What they actually wanted was higher automation coverage paired with richer explanations for every decision. They were comfortable with the agent resolving more if they could understand why. This shifted our UI design significantly toward surfacing reasoning, not just outcomes.

What We’d Do Differently

If we were starting over, a few things would change.

We would invest earlier in a shared session memory architecture. Right now, each agent maintains its own session history. There are cases where information learned by the Exception Detective would be useful to the Reconciliation Maestro, and we don’t have a clean way to share it. A shared memory layer with domain-scoped access would help.

We would also build the rule testing workflow earlier in the process. It was one of the last things we built, and by the time it was ready, the team had already been managing rules through the UI for a few weeks without being able to test them properly. Building the full authoring-to-testing-to-activation workflow as a unit from the start would have saved some friction.

Closing Thoughts

The honest version of what we learned is this: AI agents are useful in operational data governance not because they’re smarter than the team, but because they’re consistent and available. They apply the same logic to the hundredth exception of the day as they do to the first. They don’t get distracted. They log everything. For a category of work that is high-volume, pattern-heavy, and time-sensitive, that consistency has real value.

The work that genuinely needs human judgment — novel exception patterns, regulatory gray areas, client-specific context — still goes to the team. What’s changed is that the team is now spending their time on those cases rather than triaging a queue to find them.

That, more than any specific metric, is what a well-designed governance automation system should produce.