This post is in response to others on the web. Lately there have been some suggestions by individuals to break the standards. This post is a reminder as to why you should NOT break the standards, what went in to the standards in the first place, and IF you are to suggest changes, please put your testing in place before suggesting these changes to others.
First things first…
Apparently there are individuals out in the market place now who would have you believe one of the following statements:
- You do not need link satellites
- Link Satellites are bad
- Link Effectivity is not necessary
- Some Links become Hubs
- Some Hubs are composite with Link Keys
- Hub to Hub Relationships via foreign keys are OK
The problem with this, is none of these statements have been thoroughly vetted or tested. None of them hold the test of time. In fact, these statements are completely and utterly inaccurate. They can and will cause your data vault solution to fail.
The fact of the matter is: around 1996 I already went down this road. I already tested these statements when I was designing and building the standards for Data Vault. I tested these statements in the Model, Methodology, and Architecture – as well as the implementation standards.
The fact of the matter is every single one of these statements failed the tests that I put them through. Some failed in agility, some failed in maintenance cycles, some broke the model (made it brittle and inflexible), some failed in Big Data, some failed in NoSQL, and some failed in a combination of these.
The point to this is: following these statements leads you to construct something called: “CONDITIONAL DESIGN”
Conditional design is:
IF <this condition>
THEN <apply this design>
ELSE <apply some different design>
You will NEVER have a successful, scalable solution that works for real-time and batch, big data and small, NoSQL and relational, and is repeatable, consistent, and redundant when you build conditional design.
Conditional design ALWAYS leads to RE-ENGINEERING based on SOMETHING in your environment breaking your condition!!!
The root standards of the Data Vault as put forward in my SuperCharge your Data Warehouse book, and in my Building a Scalable Data Warehouse with Data Vault 2.0 book, are vetted, tested, and NON-CONDITIONAL. It’s what makes them so robust. It’s why they’ve stood the test of time, it’s why they can apply to EVERY solution build in data warehousing without failure.
Background and previous History…
I discussed a lot of the background and history of the Data Vault standards here: http://danlinstedt.com/allposts/datavaultcat/datavault-standards-what-really-matters/
In this post you will find a lot of the questions I ask about how standards come to be, and why the standards in the Data Vault world are so important. I will add to that post, with the following statements:
Data Vault 2.0 includes Agility, IT cycle time reduction, better faster cheaper, six sigma, and TQM components. That said, not only are the standards for the data modeling important, but also the people and the processes they go through to build data warehouses. So in addition to the questions in the original post I would ask you to consider this:
- Does your newly proposed standard affect Agility in a negative fashion?
- Can your new standard be repeated for all cases?
- Does your newly proposed standard increase maintenance costs?
- Is your standard be pattern based?
- Can your standard be applied in all parts of the Data Warehouse and BI lifecycle?
- Does your newly proposed standard negatively impact the flexibility of the processes or queries?
- Does your newly proposed standard need to be re-engineered based on volume? real-time? number of tables in the model?
I appreciate your Zeal…
I really do. I challenge the Data Vault standards every year. It’s how and why Data Vault 2.0 came around, and how and why Data Vault 2.0 is subtly changed and expanded – to meet the needs of cross platform MPP, Big Data and NoSQL solution sets. If you don’t believe me, just ask my friends Kent Graziano, Sanjay Pande, Michael Olschimke, and Roelant Vos.
There is a need to challenge the standards, there is always a need to innovate… This I don’t deny. It’s HOW and WHY we make a change to the standards that matters to me most. Please don’t take any of this the wrong way: I welcome you to challenge the standards too, but don’t just be a lemming and follow someone elses lead just because they said: “hey do this – it’s ok to break the standards”.
No, do it right… Bring your tests and your scientific controlled experiments to the table, bring evidence, test it. IF you can answer all the questions appropriately (from above and the original post), THEN I’m happy to entertain the ideas that the standards need to be changed. I’ve always felt this way.
I will tell you this: I have had many discussions around the challenges to the standards along the way, some of them are very fruitful. That said: my number one concern for changing standards is: Conditional Architecture / Conditional Design. I refuse to put forward a standard that requires re-engineering the base design just because Condition X appeared on the radar.Happy to entertain your thoughts and comments on the matter, that said: I can point out customer cases where standards have been broken, and a year later – the Data Vault “broke” in the customers eyes, I am hearing about some of these right now.Thoughts? Comments? feel free to reply to my post.Thank-you kindly,
(C) Copyright 2016 Dan Linstedt, All Rights Reserved