Ontology Matching — Still unsolved problem ??

To start discussing about this topic , i need to go two years down the line one random day one of my mentor and manager introduce us to this company called Palantir , for me atleast thats first time me listening that name ( but for most of you i dont think so, because market cap wise 30th world most valuable company ₹34.231 Trillion ) we went back home started researching about company and heard lot of things ( good/bad , strange/weird ). You must be wondering why iam discussing all this … Here is the answer after going through all of these i realised core of the Palantir products are Ontology creation and ontology matching and understood why data and analytics leader talked to us about Palantir.

Lets understand what’s ontology , starting with google definition A data ontology is a formal framework used to organize and structure information by defining specific concepts, entities, and the relationships between them within a given domain. Instead of just storing data as raw text or numbers, it explicitly maps out what the data means, acting as a shared vocabulary for systems and users.

For example in simpler words consider small start up company ontology is like having detailed tree / graph structure showing each person involved , their relationship and their responsibilities ( i.e organisation chart ).

Seems like easy thing why much hype around it????

Lets take the above example and consider you wanted to revamp your organisations role names based on some standard roles which are already followed by some big companies , now to do this mapping of which standard role name suits to our existing roles in the company is ontology matching.

Still nothing complex right???? But lets take simple example in your company entry level software engineer is Software Developer — 1 and in may be the reference you are taking its called Software Development Engineer — 1 how to map these????

This might be simpler scenario , where it will take only less time to create that manual mapping, But we can consider many such huge scenarips where we will be dealing with hundreds of databases , thousands of tables and more more columns sample scenarios include having standarised data platform for all business units in the company and here in most of the scenarios each business units follow their own database , column naming conventions simple examples will passing the customer id as cust_id in one BU and customer_id in another BU is simple example of this it can be much more complex.

Having standardised data always helps and this is much more important to get some insights and analysis on broader level.

Iam still getting information on whats the better way to do ontology creation , and how to do this especially when existing data sources dont have one , i.e for sure we need ontologies to match them first. Recently when iam talking with one of the fellow ex-hockey player and currently leading an Data & AI team i realised they are looking to solve the similar problem , and its quiet common across the companies (may be thats why Plantir is that big ).

In this article i will try to explain one of the approach iam thinking and expecting some ideas from all you in comments.

Idea is to leverage the LLMs to help us do this matching. Idea is to use LLMs to create the description and some relationship information and using the similarity matching may be using BertBased similarity matching or may be using the simples cosine similarity ( or ofc can directly use LLM too ) .

Idea is simple take all the tables and all necessary informations like column names , data types and other information db holds basically taking ER diagram information too and generating the detailed description about each column and repeating it for both source and destination.

Flow diagram

Above sample flow diagram shows highlevel idea which is in my brain , implemented and tested on smaller datasets but can we scale it up and use it for production ready system ?? No i don’t think still reading articles and gaining knowledge on few things appreicate any clues and links to some resources.

See you all with some other articles around and if incase you don’t know about palantir just search it up you will understand the real power of DATA.