Beyond the Patchwork: Building an Intent Aware, Agentic Data Platform

An architecture blueprint to shift from passive data platform to an active, agentic data platform.

Photo by Markus Winkler on Unsplash

Today’s data platforms reflect a patchwork of historical standards and fragmented logic. The rise of Agentic AI provides a powerful lens to move beyond incremental fixes and fundamentally overhaul the entire pipeline lifecycle for maximum speed, quality, and cost efficiency.

By embedding Agentic orchestration into our core architecture, we shift from passive, deterministic pipelines bound by hardcoded rules to a next-generation platform that is inherently active and adaptive.

The result is a complete transformation of core data operations: ingestion, normalization, enrichment, lineage tracing, taxonomy management, and semantic layering.

A Six Layer Platform Blueprint for Agentic Data Engineering

This blueprint details a modern 6 layer architecture designed to address the intricate workflows of Revenue Cycle Management (RCM) and unlock significant improvements across critical KPIs. The governing principle of this model is the shift from passive, deterministic processing to active, context aware agentic AI orchestration.

Figure 1: Six layer Implementation and Data Flow Diagram

Architecture at a Glance: Mapping Data Flow Phases to RCM Business Value

The matrix below traces the progression of data through the six layers, highlighting the precise role agentic AI plays in optimizing key RCM processes.

Deep Dive: The Six Layers of Agentic Implementation

Here, we dive deep into the technical execution of an active, self-healing data ecosystem. We trace the journey of complex RCM data as they move through the modernized data lifecycle. From unstructured raw ingestion to secure semantic consumption, this layer-by-layer analysis outlines the precise responsibilities of each intelligent agent.

1. Ingestion

Data Types: Structured | Semi structured | Unstructured

Ingestion is where you recover the most cost, fastest. In healthcare, the biggest pain is semi-structured inbound data e.g. 835 remits, 277 eligibility responses, fax-converted EOBs. LLM-based document extraction with confidence scoring lets you route high-confidence records straight through and flag low confidence ones for human review, replacing armies of data entry clerks and brittle regex parsers. The agentic connector pattern is particularly high value. Instead of a 6-week integration project every time a payer changes their portal, an agent discovers the new structure and adapts.

• LLM driven schema inference: Auto detects schema, data types, and semantic intent from raw sources (JSON, EDI, HL7, CSVs) without hand coded parsers.
• Intelligent connector agents: Agentic crawlers that self discover new REST endpoints, paginate, handle with refresh, and adapt breaking API changes.
• Unstructured -> structured extraction: LLM pipelines that parse PDFs, faxes, emails, and clinical notes into structured fields with confidence scores, replacing brittle regex/OCR rules

2. Normalization & Transformation

Tech Stack: dbt | Spark | Glue

Normalization’s biggest win is the NL-to-dbt loop. Your data engineers are bottlenecked on translating analyst requirements into models. An agent that generates a dbt model draft + tests + documentation from a business description cuts that cycle from days to hours. You still need engineering review but the blank page problem is gone.

• NL-to-SQL/NL-to-dbt agent: Business analysts describe transforms in plan language; the agent generates, tests, and proposes dbt models for engineering review.
• Semantic deduplication: Embedding based entity resolution across sources (e.g. patient/provider records) that goes beyond exact match or fuzzy string matching
• Automated unit test generation: LLM generates dbt tests, great expectations suites, and anomaly thresholds from statistical profiling, removing manual spec writing.

3. Enrichment

Tech Stack: RAG | Knowledge graphs | External APIs

Enrichment should move from blanket to selective. The typical pattern is “call all enrichment APIs for every record.” Agentic orchestration that looks at record type, source system, and existing field completeness before deciding which enrichment calls to make typically cuts enrichment API costs by 40–60% with no quality loss.

• Contextual RAG enrichment: At write time, embeds records and retrieves relevant knowledge e.g. payer policy, clinical guidelines, ICD/CPT code hierarchies, to annotate each record.
• Graph augmented classification: Knowledge graphs provide relationship context e.g. provider -> speciality -> network tier that flat LLM classification misses.
• Agentic enrichment orchestration: Agent decides which enrichment sources to call per record type, avoiding blanket API calls. Reduces enrichment cost by 40–60% on sparse data.

4. Data Lineage

Tech Stack: OpenLineage | Unity Catalog

Lineage is the highest leverage governance initiative. Most platforms have lineage gaps undocumented notebooks, ad hoc queries, vendor ETL jobs. LLM based lineage inference from SQL/Spark code closes those gaps. The conversational lineage explorer then unlocks the value: instead of a data steward spending two days tracing a KPI back to source, an analyst asks in plain language and gets an answer in seconds.

• LLM powered lineage inference: Parses SQL, Spark DAGs, and notebook code to auto generate column level lineage graphs where no instrumentation exists, closing lineage gaps.
• Conversational lineage explorer: Allows users to ask, e.g. “Where does this denial rate KPI come from?” via an NL interface over lineage graphs, retuning human readable provenance chain with citations.
• Impact analysis agent: Before a schema change is emerged, the agent traverses the lineage DAG and generates a downstream impact report, replacing manual audit reviews.

5. Ontology & Taxonomy

Tech Stack: SNOMED | ICD-10 | FHIR | Custom ontologies

Ontology and the semantic layer are where you harden the ROI. The enrichment and NL-to-SQL patterns only work if the underlying ontology is clean and the semantic layer is governed. The cross ontology mapping agent (especially ICD-10 → payer-specific code → internal taxonomy) is directly applicable to your denial decoding pipeline and worth prioritizing early, since mapping errors there cascade into incorrect appeal strategies.

• LLM assisted ontology authoring: Domain experts describe concepts in plain language; the LLM proposes OWL/SKOS definitions, synonyms, and parent-child relationships for SME review.
• Cross ontology mapping agent: Auto maps between payer specific code sets, ICD-10, SNOMED, and internal taxonomies, replacing costly manual mapping spreadsheets.
• Ontology drift detection: Agent monitors new code releases (ICD-11, annual CPT updates) and flags impacted mappings before they cause downstream classification errors.

6. Semantic Layer

Tech Stack: dbt Semantic | Looker | PowerBI| AtScale

Serving as the technical counterpart to a clean ontology, a governed semantic layer ensures that NL interfaces remain accurate. Without this layer, AI driven data exploration suffers from hallucination and inconsistent metrics.

• NL-to-metric interface: Business users query the semantic layer in plain english. The LLM resolves intent to governed metric definitions, eliminating ad-hoc SQL sprawl.
• Metric definition generation: LLM proposes metric YAML (filters, joins, grain) from business logic descriptions, seeding the semantic layer directly from requirements docs.
• Anomaly narration agent: When a metric deviates, the agent retrieves lineage context and upstream data quality events to auto generate a root cause narrative for analysts.

Conclusion

The traditional model of data engineering, defined by static mappings, fragile rules, and reactive maintenance are obsolete. As enterprise data and AI platform leaders, we must pivot our entire perspective toward intent aware, self healing platforms.

Every layer of this six layer architecture follows the same governing principle: replace hardcoded rules with LLM generated artifacts that are proposed, reviewed once by a human (HITL), and then governed. By embedding agentic intelligence from ingestion to consumption, we transform the data platform from a passive pipeline into an active, adaptive ecosystem.

In the complex landscape of RCM, this agentic shift is not just an optimization. It is the definitive roadmap to unlocking sustainable speed, high-fidelity reporting, and significant operational cost efficiency.

Disclaimer: The views, architectures, and opinions expressed in this article are solely my own and do not represent the official policy, strategy, or position of my current or former employers. All concepts, technical designs, and workflows presented here are for educational and illustrative purposes within a generic healthcare framework.

The use of Revenue Cycle Management (RCM) as a contextual example is strictly conceptual and does not describe or reflect any proprietary business logic, internal data structures, or operational metrics of my employer. Furthermore, references to specific third-party tools, technologies, or platforms are included as industry examples and do not constitute a corporate endorsement or indicate an internal production stack.