Recently I received a contact inquiry from an interested individual. In this contact there are many different questions, one of which is specifically about the architecture or design side of the house – as in: are there two design patterns that are followed or only one, and if there are two – which one is correct? In this entry I will address the nature of business keys, the singular design pattern of the Data Vault and try to correct what thought processes you may or may not have around constructing the model from the top.
This is not something I’ve written much about, although you can catch glimpses of it in my presentations, more glimpses of it are available in my classes, and of course – I will elaborate at great length within the coaching area of my site (going forward). In other words, in the coaching area, I will share with you *how* to make this work for the best light of the business.
The question I received is related to the following two statements (as quoted in his words):
1. First of all, it seems non debatable to me that the data vault pattern equals in terms of a software artifact to a “canonical data modeling pattern”. Hubs, Satellites and Links are the components of the pattern and there are more or less formal rules and degrees of freedom to instantiate the pattern. It is like a higher, BUT semantically STRUCTURED! normal form.
2. The pattern clearly separates ‘identity concerns’, the Hub (with its business key), from temporal-variant and temporal-invariant aspects (different satellites) and offers an extendable mean to evolve the schema, by giving a uniform representation.
So the first question from the individual is: What SHOULD BE actually the CONCEPTUAL instantiation of the HUBs ?
My Thought Process:
I’ve always been quirky and off the wall. My wife says I never see things the same way that everyone else does – and even my Dad said to me: I knew you were different when, at the age of 2 I’d ask you to point to the elephant picture, and you would point to the hippo. I knew you understood me, because later I would see you pointing to the elephant and repeating the words.
Ok, with that out of the way… Instead of providing a narrative (like usual) I’ll try to explain this in a step-wise fashion.
- Businesses create source systems (or purchase them) in order to hold data, and operate along the perception of what they think (they perceive how their business works).
- Source systems get old, people move on, but the rules in the source systems usually change at a much slower rate than the business does… So to compensate, business overloads data values (creates what might be perceived as dirty data by other business units) in order to force the system to hold data it isn’t engineered for.
- Businesses require keys to the information in all the systems, in order to identify specific sets of records. This is where the notion of customer, product, part, employee, etc… comes in to play. The business believes they have uniqueness in their data.
- New systems are acquired or built to make up for gaps in the existing or old systems. New systems often create new keys, and have new business processes that differ from the old systems. Frequently these new systems have overlap in functionality.
- Some (not all) business units begin using the new systems, while other business units continue using the old systems. Once this happens, the business units using the new systems move or re-create their data sets in the new systems, causing: duplication, new keys for the same data, different representations, and on and on and on.
- The BI systems try to reconcile these systems and end up in a pickle (with different results every time)
- REMEMBER: both the old and the new systems have different execution rules that currently don’t align with existing perceptions of how the business is supposed to operate.
- Now, enter the Data Warehouse… Part of our job is to align the data sets across business units (lines of business), internal and external systems, and multiple business processes. OUCH that hurts! This is probably the single toughest challenge a corporation can throw at us (and they think this stuff should be easy!)
- What’s the one single consistent business element that all business users access when they want to retrieve their data from their systems and fit it in to their business processes? RIGHT! The Business Key.
- Oops, I almost forgot – On top of the business processes, and the data in the systems exists a hierarchy of terminology called an ontology. These ontologies are organized in to taxonomies. Basically what I’m saying is: they categorize and prioritize their business terms, and they tie these to their business processes. Unfortunately many of these ontologies are not written down anywhere. If they were, it would make the process of building a master data system 100x easier!
Ok, now you understand the background – and I hope this has given you a bit of insight as to how to approach modeling business keys in the Data Vault, or even the Data Vault itself. It’s a necessary process – of course, however it gets’ deeper from here. Those of you not wanting a challenge, or wishing to stick your heads in the sand and still claim that “building a data warehouse is easy” should stop reading now and go back to your easy-chair.
The next Real Question in Integration Architecture is…
Really a series of questions that all have the same answer (unfortunately or fortunately depending on how you look at it). This is really why if you ask a true BI / EDW consultant a question, the answer most often is: “it depends.“ Well, for the following questions, the answer is singular in nature (due to the mathematics of the problem, and the representation of set logic, and the implications of MPP scalability).
The questions are:
- How do I effectively tie the current perception of the executing business process to the actual data process in the source system? How do I establish the gap between the two?
- How do I watch, monitor, and view the data that is used in the business process as it flows from one process to the next?
- How do I track the data that is used by the business external to the system that it is stored in? There are plenty of manual processes that take place beyond system boundaries that alter data sets that we don’t see inside the capture systems.
- How do I tie the same data set to the business processes and system process that are ocurring?
- How do I map similar data sets across multiple systems yet maintain auditability of each set of information?
- What is the one mostly consistent piece of business terminology that is used to represent data sets in the business users’ world?
- How do I represent this information in an ontology that is understandable by the business at the end of the day?
And of course, then there’s the element of: How do I track all these sets of data over time? And of course, all the technical questions like scalability, auditability, maintainability, simplicity, performance, etc… We deal with a LOT of issues – and quite frankly, our knowledge is often taken for granted… Anyhow, back to the point.
The answer to all of these questions is: the business key. The business key is a representation of a single data element that is captured, stored, used, applied, printed, memorized, and searched on across all these situations. It is a hard-data element that should not change (but often does, because business doesn’t understand that they hemmorage money when they do change it). It is an element that should be consistent throughout systems, across business processes, and everywhere else.
It is a conceptual representation that also happens to have a physical instantiation. It ties the conceptual world to the physical world – this is why the Data Vault model is based on business keys. IE: Hubs. It allows the model to follow the structures, processes, systems, and “perceived business rules” that the business users are under the impression they are using. It allows us to find and measure the gap between the perception and the reality of capture of the data held in the systems.
It allows us to trace the data flow, and measure the time it takes for critical path information to make it through the business. The sooner the business realizes that this is a serious level of power, the sooner they will begin to understand that this modeling technique can save the business millions of dollars and thousands of hours in integration, and consolidation, and other elements.
Ok – off the soap box. Let me answer the original question(s) above.
QUOTE from the inquiry:
Obviously, a DV can be CONCEPTUALLY instantiated in at least two major ways: – Source-affine, meaning that the DV gets instantiated along the native concepts (=HUBS) of the sources; if it’s Siebel CRM, than Siebel model will show the way, etc…I don’t want to elaborate on this as it not *my case*, it has however limitations and problems. – EDM-affine (EDM=Enterprise Data Model), meaning that the DV gets instantiated across a neutral, business-oriented representation (Enterprise data model) of the *relevant* part of the company.
The problem domain of the EDM are obviously the “business activities” (processes, events) that define the business and of course their “context” (involved parties, products, etc. = “master and reference data” in the solution space). Now if one would like to build a DWH based on this EDM pattern, which is *my case*, then a few essential aspects have to be mentioned: 1. First and most important, the EDM must be projected onto the data offering space; it doesn’t make any sense to specify a cube with facts and dimensions that cannot be fulfilled from the sources.
Ok, here’s the long and short of it: The person asking the question is really struggling with this question: What’s the best modeling approach (using Data Vault) to help identify and close the gap between the systems and the businesses?
Do you see where I got that from? It’s a gap analysis question. The point is: if your systems are truly out of alignment from your business processes, then you will find the statements in the quote above to lead you to the same question. “Do I use the source systems to model, or the business processes?”
Enough already, I hope I’ve given you some insights to the background of business keys and Data Vaults. So here’s my answer, and at the end of the day, you need to decide what works for you and follow it. Why? Because the DV model is flexible and fluid – if you build it right, you will never be completely finished with the model, you should be changing it as you learn more, and as you close additional gaps between business and systems and data sets.
Conculsions and thoughts:
My answer is (obviously): It depends. My preference is: always start with the business.
The Data Vault is supposed to be: of the business, by the business, and for the business. There isn’t supposed to be anything in the Data Vault that is not business driven. If it doesn’t have business value, don’t model it in the Data Vault – it should literally be chopped/removed from the source systems.
Now, that said: here’s the flip side of this point:
IF the source systems were too far out of sync from the business processes, then the business would be losing so much money they might go out of business or replace the source systems and applications entirely. Therefore, when worse comes to worse and you don’t have time to learn the business processes, or can’t find business users willing to give you the time of day (that’s a real shame) – then sometimes you have to resort to what you have…. Which of course is the data in the source systems, and the hardcore “data profiling” to help give you some basic foundational footsteps on which to begin building and producing results right?
I know which side of the bread is buttered… I know where my paycheck comes from, and I understand the value of releasing something quickly to the executive staff.
HERE’S the MAGIC folks in case you may have missed it:
- The Data Vault Model is based on raw data – auditability
- The Hub construct is based on the foundations of business keys – traceability/gap analysis
- The Link construct is based on the notions of change – flexibility / MPP scalability
- The Satellite construct is based on data changing over time – audit-trails, massive disparate systems/gap analysis
So, can you start with one or more source systems and it’s data? YES! Can you start with the business processes? YES! Can two different teams start with different systems or even different approaches and still link the data together at a later date? YES!
So you see, it depends. If you missed the answer, go back and read the BLUE statement above.
Oh yes, I almost forgot: The DATA VAULT should NOT be seen or used directly by business users – so any questions of “interpretation” or “understanding” should be moved downstream to “data marts” – including the Business Vault.
Do you have other thoughts? Opinions? Don’t be a stranger, I want to hear from you! Register for FREE and leave a couple comments…. Tell me what you’ve found, what you experience, what your pros and cons are of using each method of approach.
PS: By the way, you can get lessons on HOW/WHY on these things inside my coaching section: http://danLinstedt.com/my-coach