Data Fabric or Datamesh .. next Data Architecture Frontier?

If you are data professional, you might have already encountered some of the following questions: what type of data architecture is to be implemented in your business.. what would be beneficial for long term keeping close eye to your business requirements.. whether our business is ready to implement a particular data architecture paradigm .. whether to implement Data Fabric or Datamesh and many more..

And I am sure, you have not got a easy solution after discerning through various discussion, internally and externally.. I humbly echo with you view. In this context, I would like to share my experience, probably this would be useful for some of you.

As a Data Analytics practitioner, I have personally observed evolution of data architecture over two decades. This evolution or change is really fascinating if we understand the need for change. Would like to touch upon some of the drivers of this evolution:

a. Change of overall characteristics of data: Currently, more unstructured data from various sources (such as, IoT, GPS, social media etc.) are available in enterprise ecosystem. Proportion of such data has been grown exponentially in last two decades. Enterprise Data Architecture should be able to accommodate and consume this.

b. Data as value driver: It’s reality that data is new oil. Hence, business would like to use data as a value driver for various purposes including product development, improve operational efficiency and enhancing customer experience. In such case, business would like to use data more self-service way that than over-dependent on central team. That demands change on enterprise data architecture patterns.

c. Expected agility in business environment: Characteristics of businesses are changing… and that is changing quite fast. Businesses are struggling to up to the pace. Now-a-days, no business is only core-competency-only business. Enablers and force-multiplier of any business have been done by digital ecosystem powered by data. This leads to urgency to access data assets more frequently at scale and certainly, this is to be enabled by enterprise data architecture.

We have mentioned some of the important factors. Non compatibility of existing data management solutions with emerging use cases are also contributing towards that change. However, common theme remains the same …future architecture should be able to address current and future business needs at scale supporting all transformation programs that enterprises would like to pursue.

At this point you might be wondering that we have already decided to implement data lake or data lakehouse, addressing current need of the business .. why to worry about certain methods of implementation such as Data Fabric or Datamesh..

If you like to see how enterprise analytical data management solution has progressed with time, below diagram shows bird’s-eye view of that ..

Evolution of Enterprise Data Management Solutions

Over the period of time, controls are slowly decentralized and more automation thus the use of metadata increasingly used. This provided flexibility of use of data assets with more fluidity and certainly these changes are aligned to growing need of business.

And increasingly, two major methods of enterprise analytical data solution are becoming popular and those are 1. Data Fabric and 2. Datamesh

1. Data Fabric:

Data Fabric is one of emerging data management solution to attain flexible, reusable, enhanced data integration solution at reduced cost. This solution is just a natural evolution of current logical data warehouse model as it leverages existing technology and metadata in a cohesive and modernized way.

Technology Pillars of Data Fabric

Data Fabric provides

a. Enables less technical uses to discover, integrate and analyze data and thereby share insights

b. Enables more productivity gain owing to extensive use of metadata

c. Faster time to insights at reduced cost

Kindly note that all the components required to implement an effective and complete Data Fabric solution is not completely mature. Components such as Data Integration, Data Preparation, Data Orchestration etc. are more mature than that of the other components mentioned in above diagram

Further, if you are a Azure customer and would like to implement Data Fabric solution using Azure Synapse, then a lot of external tools are to be integrated. More modern solution such as Microsoft Fabric would be an effective tool; however, for discovery engine, some other suitable is to be integrated. Some tools such as Stardog for Enterprise Knowledge Enrichment would be useful.

Finally, effective implementation of Data Fabric solution is dependent on completeness and effective usage of metadata.

2. Datamesh:

Datamesh brings product centricity to manage data assets where more decentralization is enabled and best practices of automated provisioning using metadata while governed federally by domain teams.

4 pillars of Datamesh implementation are as follows:

4 Pillars of Datamesh

Implementation of Datamesh can be done as shown in following diagram.

Logical Diagram of Datamesh Implementation

Datamesh enables more autonomy to domain / department team and less dependency on central IT. However, this seems to be quite appealing, success is dependent on a lot of factors including organisational maturity, type of business , skill availability and organisational agility to change.

Datamesh ecosystem tools are still evolving. Best practices are yet to be matured. Nevertheless, product orientation of this implementation concept would appeal to a lot of agile-savvy customer.

With this I would rest my views! Would be really happy to see your comments!

Thank you for reading this article!