Metadata-Driven Framework for Scalable Data Engineering

(Sept 27, 2025) This article talks about metadata driven framework for scalable data pipelines

Creating scalable data pipelines for ingesting multiple tables at one go is key and an important perspective for data engineering. Hardcoding values and creating tons of pipeline for the same job isnt really functional, as a good data engineer one should always think for a scalable solution which is resilient and also fail proof. In this article, we will cover the aspects of creating a scalable and fail proof pipeline.

Pre-requisite:

Fabric

• SQL DB — A SQL server for storing meta-data. The reason of choosing SQL DB is it works well for storing and pulling meta-data information and also doesnt cause a major impact on the warehouse.
• Warehouse — Storing Data
• Data Pipeline — Orchestration

Synapse

• Azure SQL — For storing meta-data
• Synapse Warehouse — Storing Data
• ADF — Orchestration

Scenario: We own a CRM tool and we want to store all the CRM data inside the warehouse for analytics and ML purpose. So, here we have customer related data from different different countries.

Step 1: Choosing the right columns for meta-data driven framework. Since, its customer data we will need data classification, country, name etc.

CREATE TABLE elt_orchestration.config_table
(
table_id int not null, 
table_name varchar(100),
country varchar(100),
data_classification varchar(100),
run_sequence smallint not null,
isenabled boolean
)

Step 2: Creating data pipeline/ADF pipeline

• Lets start with using the lookup activity, this will be used for pulling the information stored inside the meta data table.
• Once the information is pulled, we will use a for loop and filter the configs which are matching the current config
• Run the activity of SQL procedure, which will run the SQL statements for ingesting data from one layer to another layer

This framework is very basic and for the idea of creating a scalable framework.

Congrats! You have created a meta-data driven pipeline

Thank you for reading! Have a good day!

Enjoyed the article? Tap the clap and like buttons! And don’t forget to hit follow to stay updated with more great content!

Follow me on YouTube: https://www.youtube.com/EngineerOnTheMove1