Building an AI-Native Data Mesh with Metadata at the Core

Most enterprise data platforms fail for a familiar reason: they treat data movement, governance, quality, and consumption as separate concerns.

Data engineers build pipelines. Governance teams maintain catalogs. BI teams create reports. Domain teams define business meaning in their own silos. Then AI initiatives arrive, only to struggle because metadata, schemas, data quality signals, and business context are scattered across disconnected systems.

Data mesh architecture

The architecture in the deck points to a better model: an AI-native data mesh where metadata, schemas, quality rules, and consumption patterns are treated as first-class platform capabilities.

The principle is simple: keep the data plane open, scalable, and flexible; make the control plane metadata-driven and governed; and expose the platform through trusted SQL, BI, and AI interfaces. This shifts the platform from merely moving data to enabling trusted, explainable, and AI-ready data products.The platform vision

This is not just an ETL framework. It is a metadata-driven data platform where domain data products can be ingested, transformed, governed, queried and explained through a combination of pipeline DSLs, lakehouse storage, MongoDB-backed metadata services, SQL engines and AI agents.

The architecture has five important building blocks:

1. Connectors and parsers bring enterprise data from databases, APIs, files and event systems into the platform.
2. Spark-flow and Polar-flow define configurable pipeline execution paths for large-scale Spark processing and lightweight Polars processing.
3. Object storage acts as the lakehouse foundation for raw, curated and data-product zones.
4. MongoDB-backed metadata services store catalog metadata, data-product schemas, operational metadata and quality rules.
5. AI Agent + UC MCP Server enable natural-language queries, governed pipeline generation and quality-report interpretation.

Why This Architecture Is Needed

Traditional data platforms are often designed around moving data from source to destination. While that is important, it is no longer enough for a modern enterprise data platform.

• What does this dataset represent?
• Who owns it?
• Which schema contract does it follow?
• Which quality checks were applied?
• Can a business user query it safely?
• Can an AI agent explain it without hallucinating?

This is where the control plane becomes critical.

By placing Unity Catalog on MongoDB, metadata stores, schema registries, and quality rules at the center of the platform, this architecture ensures that every pipeline, dataset, and query operates with governed context. The result is a platform that does more than move data — it makes data trusted, discoverable, explainable, and ready for both business and AI consumption.

The Refined Architecture

The refined architecture organizes the platform into three distinct planes, each serving a clear purpose.

1. Metadata, Catalog, and Governance Control Plane

This plane acts as the intelligence layer of the platform.

Unity Catalog on MongoDB serves as the anchor for discovery, access control, and lineage. The metadata store captures operational details such as pipeline configuration, runtime status, ingestion history, and data quality metrics. The data-product schema registry manages contracts, ownership, semantic definitions, and domain-specific meaning.

This is the layer that prevents the platform from becoming just another data lake or file dump. It makes data assets discoverable, governed, reusable, and trusted across the enterprise.

2. Data Ingestion, Lakehouse, and Processing Plane

This is the execution layer of the platform.

Connectors and parsers bring data from enterprise systems and normalize it for downstream processing. Spark-flow and Polar-flow act as configuration-driven transformation engines, where transformation rules, data quality checks, validation logic, input/output contracts, and processing behavior are defined through JSON-based specifications.

Spark-flow manages large-scale distributed pipelines using pySpark, making it suitable for high-volume batch and lakehouse processing. Polar-flow supports fast, lightweight transformations using Polars, making it useful for smaller datasets, quick validations, file-based transformations, and high-performance local processing.

The configuration for both Spark-flow and Polar-flow is stored as part of the catalog, along with schema definitions, ownership, quality rules, lineage references, and runtime metadata. This ensures that pipelines are not hardcoded in isolation, but governed, discoverable, reusable, and explainable through the platform control plane.

Object storage forms the lakehouse foundation, organizing data across Bronze, Silver, Gold, and data-product zones. DuckDB and Spark SQL enable interactive querying and serving views, while the JDBC interface allows BI tools and applications to consume governed datasets without being tightly coupled to the underlying storage format.

3. AI, BI, and Consumption Plane

This is where the platform becomes accessible and useful to business users.

BI integrations allow users to consume dashboards, reports, and applications. More importantly, the AI Agent connects with the catalog through the UC MCP Server, enabling natural-language queries, pipeline specification generation, data quality explanations, and recommended remediation steps.

The AI Agent does not operate blindly. It is grounded in catalog metadata, schema contracts, lineage, and quality rules. This makes AI a governed interface to the data platform, rather than an uncontrolled shortcut around governance.

Key design principles

Metadata first

Every pipeline, schema and data product should be registered before it is consumed. The catalog and schema registry are not documentation after the fact; they are part of the execution contract.

Data products over raw datasets

The object store should not be treated as a generic dump. Bronze, Silver and Gold layers should eventually publish domain-oriented data products with ownership, contracts, quality scores and consumption patterns.

Multiple compute engines, one governance model

Spark, Polars, DuckDB and Spark SQL serve different use cases. The platform should not force every workload into one engine. Instead, the governance model should remain consistent across engines through catalog, schema and metadata services.

AI should be catalog-aware

The AI Agent should not directly guess table names, columns or business definitions. It should use the UC MCP Server to retrieve governed catalog context, schema contracts, lineage and quality metrics before answering questions or generating pipelines.

Implementation Roadmap

A practical implementation of this architecture can be delivered in clear phases.

Phase 1: Build the Metadata Foundation

Start by defining the MongoDB metadata model for data products, schemas, pipelines, source systems, quality rules, and lineage anchors. This creates the foundation for discovery, governance, observability, and AI grounding.

Phase 2: Standardize Pipeline Execution

Implement Spark-flow and Polar-flow as configuration-driven pipeline runtimes. Both runtimes should follow common contracts for inputs, outputs, schema validation, quality checks, error handling, and observability.

Phase 3: Establish Lakehouse Zones

Standardize object-store zones for Bronze, Silver, Gold, and published data products. Every pipeline output should be registered in the catalog with its schema, ownership, lineage, and quality metadata.

Phase 4: Enable SQL and BI Serving

Expose governed datasets through DuckDB, Spark SQL, and JDBC interfaces. This allows BI tools, reporting platforms, and applications to consume curated datasets without depending directly on the underlying storage layout.

Phase 5: Introduce AI Enablement

Introduce the AI Agent and UC MCP Server only after the catalog, schema registry, lineage, and quality metadata are mature enough to ground AI responses. This ensures that AI interactions are explainable, governed, and aligned with trusted platform context.

Final Thought

The real power of this architecture does not come from Spark, Polars, DuckDB, or MongoDB in isolation. Its strength lies in the clear separation of responsibilities: scalable execution in the data plane, trusted metadata and governance in the control plane, and governed self-service in the consumption plane.

This combination transforms the platform from a pipeline factory into an AI-ready data product ecosystem — one where data is not only processed, but also understood, trusted, discoverable, and ready for intelligent consumption.