Microsoft Purview Data Map: Your Ultimate Data Navigator

Picking up where we left off in my previous article on governing data with Purview’s Data Catalog , it’s time to dive into the next big thing — Data Map! Think of it as Purview’s brain 🧠, mapping out your data landscape so you can keep everything organized, secure, and under control. Ready to explore? Let’s go! 🚀✨

Link for free access: https://medium.com/@yashigoyal1927/microsoft-purview-data-map-b7a2b012ae60?sk=2fe3beaeb9856bf95ebbf0e243007c61

Data Map to the rescue!🗺️

Microsoft Purview is like your data bodyguard 🛡️ — helping you discover and protect sensitive information with ease.

This article is your crash course on Scanning and Ingestion in Microsoft Purview! 🚀 Here’s the scoop:

• Scanning: Purview’s version of data paparazzi 📸 — it connects to your data sources, grabs metadata like names, file sizes, and schemas, and applies classifications. You can scan everything or just the juicy bits (specific folders or tables).
• Ingestion: This is the cool part 😎 — metadata is processed and stored in the Data Catalog, neatly organized in the Data Map.

Let’s make it fun with an example! 🎉

Imagine saying, “Hey Purview, here’s my data lake. Inside, we’ve got a curated container with loads of directories. Can you scan it, tag the data, and tell me what’s sensitive? Oh, and I’ll pick what to register in my domain later! “

To register and manage an Azure Data Lake Storage (ADLS Gen2) data source in Purview. Follow these steps:

Register the Data Source:

• Head to Data Map → Sources

• Select Azure Data Lake Storage Gen2 → Hit Continue.

• Give it a cool name 😎, pick the Azure subscription, the Data Lake account name, and the collection, then hit Apply.

• Boom! Your ADLS Gen2 storage account now appears under the collection.

What’s next after registering data sources? 🗂️

Once you’ve registered your data sources, the next step is scanning. Here’s how it works:

1. It sets up a connection to your data source.
2. Captures technical metadata like file names, sizes, columns, and schemas.
3. Applies classifications on structured data sources.

Pro tip: You can scan everything or just focus on specific entities (folders/tables). Plus, scans can run instantly or be scheduled for regular updates. ⏰

Create the Scan:

• Open your Microsoft Purview account → Go to Microsoft Purview governance portal.
• Navigate to Data Map → Sources to see the collection tree 🌳.
• Under your registered ADLS Gen2 source, hit the New Scan icon 🎛️.

Now sit back and let Purview do its thing — keeping your account up-to-date and your data in check!

Why Scanning Rocks🌟

By scanning your data estate, Purview automatically collects metadata for data assets like tables, files, folders, and reports. Here’s why it’s awesome:

• Ownership: Registering a data source and scanning it establishes technical ownership of the sources and assets displayed in the catalog.
• Control: You decide who can access which metadata in Purview. 🎛️
• Hierarchy: By registering sources and assets at the domain level, they’re stored at the highest level of access hierarchy.

Pro Tip: Create collections to scan metadata and set up the right access hierarchy for your data. This ensures your data stays organized and secure!

Ingestion Simplified

Let’s talk about Ingestion — Ingestion is the essential bridge 🌉 between scanning and having a fully functional Data Map. Without it, the metadata collected by scans would just sit there, unorganized and inaccessible. So, the next time you see your beautifully populated Data Map, remember to thank the humble ingestion process!

How Does Ingestion Work?

Ingestion kicks in after a scan is completed and takes care of these essential tasks:

Processing Metadata: It takes the technical metadata and classifications identified during the scanning process (like file sizes, column names, schemas, etc.) and gives it a once-over.

Applying Resource Set Patterns: Ingestion recognizes patterns in your data sources, making it easier to organize similar types of data assets. Think of it as grouping all mystery novels together in the library 📚.

Populating Lineage Information: If there’s information about how data flows between systems (data lineage), ingestion captures and stores it, so you get a clear picture of your data’s journey.

Loading into the Data Map: Finally, it neatly places all the processed metadata into the Data Map, ready for you to explore, manage, and govern.

Why Wait for Ingestion?

Here’s a simple example:

• Imagine you’ve scanned a data source containing 100 files in your Azure Data Lake. The scan finishes, and you’re excited to see your data assets in the Purview Data Map. But… they’re not there yet. 😟
• That’s because the ingestion process is still working in the background to analyze the scanned metadata, apply patterns, and organize it.
• Once ingestion is complete, voilà! 🎉 Your assets and schemas appear in the Data Map, ready for discovery and governance.

Pro Tip — If you’ve completed a scan and don’t see your assets yet, don’t panic! Just give the ingestion process a little time to do its job. 🕒

And that’s a wrap, folks! 🎉 We’ve explored how Microsoft Purview’s Data Map helps you scan, organize, and take ownership of your data estate like a pro. From registering data sources to setting up scans and managing access hierarchies, you’re now equipped to master your data governance game!

If you found this helpful (or just enjoyed the ride 😄), don’t forget to clap 👏 and follow me for more cool insights, tips, and tricks on data engineering, governance, and beyond. Your support keeps the content flowing! 🚀✨

If you’re just joining the series, don’t miss out on my other articles:

• Data Governance: A Fun and Simple Guide to Getting Started *https://medium.com/@yashigoyal1927/data-governance-a-fun-and-simple-guide-to-getting-started-d09537a9016b*
• Microsoft Purview Data Catalog https://medium.com/@yashigoyal1927/microsoft-purview-931bb9e67946