Is Data Vault worth the investment?

Data Vault is a durable, very reasonable path to an authoritative enterprise view but it is by no means the only path. Beware the learning curve and the cult-like advocacy.

Read on for why but note that this article is not a primer on any of the modelling techniques discussed.

A brief history on the competition

The idea of a single corporate source of enterprise data was conceived in the 1980’s. One of the grandpappies of the enterprise data warehouse, Bill Inmon, popularised E.F. Codd’s 3rd normal form as a means of standardizing corporate data into authoritative, reusable representations of the enterprise.

Inmon’s 3rd normal form (3NF) model of the enterprise was the gold standard for nearly 20 years, only to be widely challenged when Dan Linsted released his Data Vault modelling standard in the year 2000.

While other techniques don’t come close to 3NF and Data Vault for delivering authorative, reusable representations of the enterprise, many use case driven modelling techniques have found a place in the enterprise data warehouse, including from another grandpappy of warehousing, Ralph Kimball, whose dimensional modelling techniques are still widely used today to simplify the datawarehouse for BI reporting.

The data industry has also largely expanded its focus, stacking and iterating patterns to deliver a multi-purpose enterprise data platform, including feature stores, object oriented services, federated delivery through Data Mesh and Agile, and now, ever more so cautiously, exploring the promise of virtualized data services such as Data Fabric and the universal semantic layer.

Are 3NF and Data Vault still relevant today? Do we need either?

Yes (so far). If working in data generally hasn’t, the early years of Big Data taught most data practitioners that clean, standardized data is at least 80% of the work when mining for actionable insights or developing data products. We tend to agree that:

• The datalake, with its varying degrees of swampiness, forces a tedious 80% (approximately) of effort before the fun, incisive stuff can begin. Most data scientists work to alleviate the 80% in some way — usually through standardized, reusable data sets.
• To move fast, drive value, comply with data privacy and security laws, most teams work to reduce the number of people who have to do that 80% over and over again.
• Centralizing and standardizing data engineering efforts, by way of an enterprise data warehouse, eventually always makes sense.

There are of course some exceptions. For example:

• Small to medium businesses with relatively low volumes of data, low levels of complexity, spread across a small numbers of systems, generally won’t get the large scale benefits that outweigh the costs of building and maintaining an enterprise data warehouse.

So which one? Data Vault or 3NF?

On the face of it, a newer system of modelling should be the obvious choice. Especially one that’s been developed to address the deficiencies of previous methods, has been stressed tested through 25 years of enterprise development and has continued to be improved over the years.

Yet, it’s not a resounding yes from me. More of a cautious yea uh huh.

Let me explain.

The commonalities

Both modelling methods address the basics:

• Definitive identification of the things that matter to your organization: Data Vault conforms business keys into subject oriented Hub tables and associated descriptive history into satellite tables. 3NF conforms systems oriented data into subject (or business) oriented, normalized set of tables.
• Recording key events from your organization’s business processes: Data Vault models these into transaction satellites (attached to links) while 3NF models these as transaction tables within its subject oriented, normalized set of tables.
• Systematic storage of organizational history: Both modelling techniques apply CDC to accumulate history (which hopefully has been programmed by data teams to accumulate in line with the organization’s data retention policies).

Where they differ

Data Vault 2.0, released in the year 2013, is highly prescriptive in its physical implementation to ensure solutions scale towards a real-time “operational” warehouse, including:

• Hash keys as primary keys
• Rigorous application of set theory in CDC operations
• Separation of soft and hard rules to maintain a semblance of transparency and traceability
• Bridge and Point-In-Time tables as assistive objects for high volume (BIG) data
• Virtualized Business Vault as much as possible (with the exception of Bridge and Point-in-Time tables)

Doesn’t that place Data Vault ahead?

Not yet. All of the above can be applied in 3NF modelling as well.

But isn’t that cheating? I’d like to think of it as imitation being the highest form of flattery. Afterall:

• Data Vault’s Bridges and Point-In-Time tables are implementations of join-indexes from Teradata, one of the first widely available MPP databases.
• Hash keys were in use well before Data Vault, originating in IBM, and then adopted by databases such as Teradata and Cassandra to distribute data.

So if most of these differentiators can be applied through 3NF modelling as well, what actually places Data Vault ahead?

Simply, it doesn’t insist that multiple systems conform into a single authoritative business record.

For example, a customer with descriptive data in 3 different systems would have its attributes stored in source-specific satellites attached to the customer hub record.

A customer hub with 3 satellite sources

3NF doesn’t have the same luxury.

• Modellers have to conform these systems in order to maintain the correct grain. It’s pretty arduous and time consuming.
• While 3NF could accomodate more attributes (for example, 3 different name attributes qualified by system specific suffixes?), it can get out of hand with wider tables and more extensive CDC rules. It’s rarely easy to justify.

Data Vault’s ability to accomodate attributes from more systems through its source-aligned satellites also means:

• More nuance from across the organization being captured in the enterprise data warehouse
• Fewer decisions for a conformed view and therefore faster progression to build phase

So what’s the catch

The first is perhaps an accepted trade-off with Data Vault.

• A business process can be inferred from travelling a well built 3NF model. The same is far less intuitive in Data Vault. The very designs that make Data Vault easier to build and tantilisingly promises to be easier to use, abstracts it further from business processes and makes it initially quite baffling to a data community.

The second, is far more significant:

• The cost to build can be high: Data vault is a proprietary methodology. An entire industry revolves around it. Trainers, project coaches, certifications, workshops, courses, books. Resources come at a premium.
• Poor implementations of Data Vault have been fairly common: The IT industry doesn’t talk as readily of its failures so perhaps this isn’t commonly acknowledged. Despite the highly prescriptive methods, employing certified resources and adoption of best practices for project delivery, incorrect implementations still occur. Why? For a myriad of reasons, including insufficient business subject matter expertise and engagement, teams relying on work-arounds to compensate, software engineers well versed in Data Vault rules yet not particularly well versed in Data.

So yes — I’d recommend Data Vault, by a whisker, but make sure your project has deep pockets. Your organisation will essentially find itself paying a premium to build the solution because your modellers and engineers will need substantial support and iteration to execute well.