Future of Enterprise Data Trust Need Different Layers for Governance and Execution

Microsoft Purview for Governance + Datagaps for Execution to build trust!

Note — Stuck behind medium paywall? Click here to read this blog for free!

Enterprises today manage hundreds of Data and AI products, both for internal operations and for their clients. From what I have observed over the last three years, projects in their initial stages hardly have any governance layer. In some cases, even older projects lack one. This essentially means that most enterprises do not have a dedicated data governance setup. Even if they do, they don’t invest much in it. A dashboard prototype or a simple audit table in the same warehouse is often the best they have in the name of governance.

Now, the enterprises that do have governance in place usually have their Solution Architects design the governance and observability layer around Microsoft Purview while working with Azure. Which is great, but the only issue is that Purview primarily helps with the governance and compliance aspects of data.

It is more policy-driven than execution-driven. It does not help you build trust in the data itself. Rather, it helps you answer questions like who owns a particular asset(data/artifact), who has access to which part of the data, whether the data is secure, or whether there is a random table in the database that isn’t related to any business use case.

But there are a lot more problems to solve when it comes to building trust in the data sitting inside your tables. For example, Purview provides health controls to observe the health of your data estate. If you look closely, almost 70–80% of those controls relate to the associated metadata rather than the data itself. Purview helps you manage , or you could say govern, your data platform. What it doesn’t control is the execution layer of governance and whether the defined expectations are actually being met.

It doesn’t actually go inside the data to find issues continuously. Purview was never designed to validate your ETLs. It doesn’t allow you to identify issues in your dashboards. Lineage helps you track the movement of data, but not the quality of that data throughout its journey. If you perform a migration, you cannot actually test whether it happened as expected. You need better validation.

At the time of writing this blog, I realized that both Microsoft and Datagaps had released major updates to Purview and DataOps respectively, introducing several advanced features. That makes this blog particularly relevant because it is a good time to discuss how these two platforms complement each other and why the overall data architecture becomes much stronger when both are implemented together using their native capabilities.

Knowing vs. Proving

There’s a humorous saying in the data world: “Metadata tells the story. Data tells the truth.”

You can know everything about how a system is designed, yet someday a random value out of nowhere can still bring your pipeline down. Enterprises simply cannot afford that kind of risk.

I want you to think about a real world scenario. Imagine you’re leading a project. After a few months, the governance layer starts taking shape. Assets get cataloged. Sensitive columns are classified. Business glossaries are created. Lineage diagrams begin to show how data moves across pipelines, warehouses, and reporting systems. Owners are assigned. Access policies are defined. Documentation slowly becomes richer with every sprint.

These are exactly the areas where Microsoft Purview excels.

So, if you’re an architect trying to answer questions like— Where did this dataset originate? Does this table contain PII? Who owns the asset? Which business glossary term maps to this column? Who accesses this data last week?

Purview answers all of these questions exceptionally well. These capabilities are foundational because governance starts with visibility. However, visibility alone does not create trust. Why?

1. Knowing that a customer table is certified tell you nothing about whether yesterday’s ETL loaded only 60% of the expected records.
2. Knowing the lineage of a sales dashboard doesn’t tell you whether a transformation accidentally duplicated millions of records. Trust me, the Silver-to-Gold pipeline creates more issues than you might think if the system isn’t designed carefully.
3. Knowing that a dataset has an assigned owner doesn’t tell you whether the pipeline dropped an entire partition because of a schema change.

These are execution problems. And these are the problems that ultimately determine whether business users trust the numbers they see. If anything is wrong with those numbers, the entire pipeline loses trust.

If your metadata tells you one thing but the actual data tells another story, that’s when you need to rethink your governance strategy.

You don’t build trust by documenting data. You build trust by continuously proving that the data deserves it. And for that, you need validation.

Here’s a quick visualization to help you understand the difference betwen these two layers of data governance (Share this to your manager if you don’t want to spend nights debugging issues lol) —

Comparison between Metadata and Operational Governance. (more about it in the next section)

Building the Execution Layer of Governance with Datagaps DataOps Suite

Now that you understand the problem, you know that what you need is not just visibility but also validation. In fact, the realization I had while reading more about this whole concept is that the term “governance” itself can be divided into two parts. (as shown in the iamge above)

The first is metadata governance. It involves everything surrounding the data. For example, cataloging, lineage, ownership, classifications, access policies, compliance, business glossary, and more. Purview is one of the best tools for this if you prefer the Microsoft stack.

The second is operational governance, which we just talked about. It focuses on whether the data itself satisfies the expectations that the business has defined. DataOps Suite by Datagaps is a tool that excels in this area. It offers everything from Data Quality to ETL and BI Validation.

When you think about operational governance, the questions you ask as a stakeholder, or maybe even as an architect, become very different.

Instead of asking, “Who owns this table?”, you start asking, “Did this table load correctly today?” Or maybe questions like, “Can we use the numbers shown on this dashboard in tomorrow’s board meeting?”

You become much more specific while asking these questions. The whole shift happens because you’ve now moved from documentation to validation.

I think this is exactly what enterprises demand, and DataOps solves precisely this problem. It doesn’t replace the governance layer, nor does it replace Purview. In fact, it complements it by continuously validating whether the policies and expectations defined by the business are actually being satisfied in production.

So, governance defines expectations. And, DataOps proves those expectations are being met through data quality rules, scans, ETL validation, BI validation, and trust score generation.

Just like lineage tells you where the data traveled, validation tells you whether it arrived correctly. That’s the simplest difference between these two layers of governance.

I’m more of an experimentation guy, and it would be wrong not to mention some product-specific features, especially when I’ve been experimenting with the tool for quite some time now.

If you’re planning to build the execution layer of data governance using DataOps Suite, I’ve explored some of its AI capabilities and documented them in the form of blogs. Check them out here.

[embed]End-to-End Data Migration Testing for Microsoft Fabric — A Practical Enterprise Guide Using… Note — Stuck behind a medium paywall? Click here to read this blog for free!uselessai.in

[embed]Implementing End-to-End AI-Powered Data Observability for Microsoft Fabric with Datagaps Difference between Data Observability and Data Qualityuselessai.in

[embed]Scalable Synthetic Test Data Generation using Datagaps for Microsoft Fabric — Stop Copying… How to use Datagaps TDM to generate synthetic data for testing Microsoft Fabric Pipelines?uselessai.in

Recently, Purview introduced its new portal with several advanced capabilities around governance, compliance, and data security. Datagaps also released a new version of DataOps, introducing AI across almost every major stage of data testing — from generating data quality rules to writing and fixing SQL queries, all with minimal manual effort.

If you are designing a solution that integrates Microsoft Purview and DataOps, you can use this sample architecture as a reference. It demonstrates how the capabilities of both platforms work together across various stages of the data governance and quality lifecycle.

Architecture combining Microsoft Purview and DataOps Suite for entire governance lifecycle

While I’ve focused more on data testing throughout this blog, that doesn’t mean you can’t perform metadata governance here as well. You can also track schema drift and any structural changes in your tables, whether it’s adding or removing columns or simply renaming them.

You can filter this information across all your data sources from a single place for any given time period. It also performs impact analysis, showing you the affected tables and columns across your data flows within DataOps Suite.

One area where DataOps Suite really shines in this context is transformation validation. Imagine applying 10–12 different transformations across multiple columns and tables while moving data to your dashboards or Gold layer. This stage determines what eventually becomes visible on dashboards or gets exposed through APIs. If anything goes wrong here, you’ll end up with incorrect numbers.

Instead of discovering these issues after a stakeholder reports incorrect values, the platform continuously validates the data throughout its lifecycle and sends alerts whenever required, before incorrect data reaches business users. That’s the real objective of operational goivernance.

It not only helps you create transformation validation pipelines for your data flows but also allows you to generate synthetic test data using DataOps TDM, so you can validate those transformations before moving anything into production.

When it comes to Data Quality, there are many other components that help build the execution layer of governance.

For example, you can separate bad or invalid records and process them later while allowing only clean, validated data to move downstream.

Once you start using the platform, you’ll notice that some checks become repetitive. Instead of recreating them every time, you can define reusable Data Contracts that standardize validation rules across similar datasets.

And these are just a few examples. The broader idea is to treat data validation as a continuous operational capability rather than a one-time testing activity.

What next?

Companies are increasingly moving towards building AI applications on top of enterprise data. As newer technologies emerge, the definition of governance continues to evolve. It can no longer be limited to visibility alone — knowing where the data came from or who owns it is no longer enough.

I believe the future of enterprise data trust will be built on two complementary layers. The first layer focuses on visibility, while the second focuses on continuous validation.

I’m also optimistic about a headless approach to building trust using these tools and pipelines. By that, I mean we’ll soon see more automation and fewer interfaces.

Today, we have interfaces for almost everything we do — reading files, editing them, making changes to data flows, modifying business rules, adding new data sources, and so on.

I feel that, in the near future, most of these operations will happen behind the scenes. Instead of humans manually performing these tasks, AI Agents will automatically handle them while humans simply review and approve the generated configurations.

Building and maintaining the governance layer will gradually become an automated backend process rather than a manual one. DataOps already introduces AI capabilities to simplify many of these tasks today, but I believe the future, maybe five or ten years from now, will look quite different. We’ll spend less time configuring governance and more time defining business intent.

Till then, see you again.If you’re working in a leadership or senior technical role, I think it’s worth spending some time evaluating which governance strategy best fits your organization’s journey. I’m sure the mixed approach of both layers would be helpful in building trust with your enterprise data. Cheers :)