Metadata Sync from within a Fabric Pipeline!

Before I start with the great news, I’ll say this upfront: we still have a long way to go with metadata sync (MD Sync), but we’re getting there. It’s just a shame you can’t see all the hard work that’s going into it behind the scenes.

So let’s start with the problem.

When adding tables to a lakehouse, there is a delay between writing the files and the data becoming available in the SQL Analytics Endpoint (SQL AE). The SQL AE is powered by the Fabric Warehouse engine in Microsoft Fabric.

Until now, the workaround has been to update the tables and then, before querying them, execute a REST API call to refresh the metadata.

How you call the REST API has always been dealer’s choice: a Spark notebook, a Python notebook, a pipeline web activity, etc. After all, it’s just a REST API — you can call it from anywhere.

That is, until today.

Today there is a new activity in Fabric Pipelines: Refresh SQLEndpoint. (Now, I don’t love the name — but I love what it does!)

Just provide the connection details, workspace, and SQL endpoint name (don’t worry — it’s the name, not the ID), plus an optional timeout.

That’s it.

Behind the scenes, it simply calls the REST API for you.

This should make things much easier for people working in Fabric Pipelines who aren’t fans of notebooks or don’t want to spend time wiring up REST API calls themselves.

Written by the human with human mistakes.