as you know i’ve been chatting with people about the need for real standards, consistent and repeatable standards in the data warehousing and business intelligence industry. i’ve been harping on and on about how you must stick to the data vault standards as i’ve defined them. i’ve talked a little about what happens when you leave or depart from the standards. if you want the business perspective on all of this, read on…
well, in this entry i’m going to explore the business purposes for applying standards, and we will explore why i am so adamant about sticking to the defined, well tested, well vetted standards.
a brief introduction…
data warehousing and business intelligence have suffered over the years from a lack of proper maturity. what that translates to are statements like this:
- we (business) built our own (access database) because it took too long (to implement our requirements)
- we (business) built our own (access database) because it costs too much…
- we (business) built our own (hadoop / nosql database) because we needed customized information (self-service anyone?) and it was unable or unwilling to provide it.
well, if you’ve heard these things, or have been guilty of saying these things, then you know, see and feel the real pain in business. perhaps, you’ve heard this one? “we had to scrap everything we just finished, green-field the entire project, and build it all brand new because the costs and time to sustain and maintain were too much for business to bear….”
if you’ve not heard or experienced this in your it landscape then you’ve been living under a rock or in a fantasy world for the past decade or so.
the fundamental reason for this…
there are reasons why this happens, and it’s the same thing, every time, all the time.
i say: organizations simply have not adopted proper or well founded standardized principles which allow their it staff and the corresponding practices to evolve along a maturity curve.
the impacts are everywhere…. the problem is: corporations and executives (is this you???) fall in to these traps:
- did we invent this? (not invented here syndrome) so it must not be good
- is anyone else doing this? (afraid to try new things, even in a sandbox)
- my buddy at the <insert your favorite big 6 consultancy here> said he’s never heard of it, so it can’t be valuable.
- i heard about him/her , and i don’t like them based on what i heard.. even though i don’t personally know them and have not taken the time to talk to them myself…
and the list goes on….
so how are maturity models, efficiency, and standards related?
you need to do a little research here, but you don’t have to look very far. maturity models are frameworks (like cmm/cmmi) that allow an organization to move ahead in a level based system. the levels along the way, provide the organization (you and your team) a solid set of instructions for what you need to do to: yep, you guessed it: improve your current operations. ultimately to reach the top level (which is where automation and efficiency truly live).
a maturity level is a well-defined evolutionary plateau toward achieving a mature software process. each maturity level provides a layer in the foundation for continuous process improvement. http://www.tutorialspoint.com/cmmi/cmmi-maturity-levels.htm
it just so happens that cmmi is already “baked in” to the data vault 2.0 solution, so that teams leveraging the methodology can actually mature their own it practices extremely quickly. i also want to point out, that enterprise data warehouse solutions which don’t follow standards, never reach maturity, and very rarely can take advantage of automation tooling!!!
so, let’s take a look at efficiency, automation and optimization for a minute. i learned long ago (while being trained in cmmi all the way to level 5, six sigma for error reduction, and tqm for total quality management) that the following mantra must be applied to everything we do!! at home, in your personal life, or at work – it doesn’t matter… these rules of maturity are inescapable.
- you cannot “automate” what you don’t “optimize” (agility and lean initiatives)
- you cannot optimize what you don’t measure (kpi and success measurements).
- you cannot measure what you don’t define (kpa and standards)
- you cannot define what you don’t understand (non-standard approaches)
these rules are the cmmi maturity curve in reverse (starting with level 5). automation is the cherry on top of the icing, which covers the cake!! the icing is the optimization, the cake are the right measurements for all the ingredients and the right temperature to bake the cake at, and how long to bake the cake (all kpi’s). putting the cake batter together, mixing the ingredients is a process – defined on the box, known as the recipe.
- if you don’t understand an oven and how to use it, you can’t bake the cake.
- if you don’t understand what flour, sugar, yeast, flavoring, and raw ingredients are, you can’t put the cake batter together).
- if you don’t have a well-defined recipe with a standard set of instructions to follow, your cake won’t “turn in to a cake” at the end.
now, all of that said: with all the right pieces well defined, and put in place, can you build a machine that “automates” making cakes?? of course you can!!!
this is precisely what i’ve done with data vault and data warehousing / business intelligence. i’ve provided the recipe, the know-how, the list of ingredients, the temperature, time, and vetted process in order to bake the cake (edw) properly.
efficiency in the maturity model is achieved in two manners: a) by measuring, and then improving or correcting broken or misaligned processes (not just data loads here, but the business processes involved in building a data warehouse solution), b) by applying a tool to leverage repeatable, consistent and standardized templates (recipes) that follow the rules.
level 4 of the maturity model creates an optimized business environment that uses real-time operational information to align employee activities with business goals. the last step of the maturity process is to automate fundamental decision-making processes. this allows personnel to redirect efforts from mundane tasks to efficiency-driven activities that continually improve operations and lower costs. at this level, enterprises rely on automated decisions to provide key benefits such as reject handling, work order generation, and supply chain management. complex and crucial decisions incorporate authorization workflows that integrate directly with business systems. http://www.controleng.com/single-article/understand-the-maturity-model-to-better-manage-integrate-plant-floor-enterprise-systems/a3c89275a85f5b2e079c2683e2a4a90a.html
why, as an executive or a manager should i care?
you should care because:
advance capabilities of enterprise information systems with the maturity model explanation provided. decision efficiency is measured by the cost (money, time, and effort) to make a decision. greater decision efficiency = lower costs = higher profits. consider 5 factors when measuring decision efficiency. http://www.controleng.com/single-article/understand-the-maturity-model-to-better-manage-integrate-plant-floor-enterprise-systems/a3c89275a85f5b2e079c2683e2a4a90a.html
you should care because: automation can make your teams 70% more agile, more efficient, and reduce your costs and time to deliver by 70% to 90% or better.
you should care because: skills become transferable, and shareable across the it teams around the world – both internally to your organization and externally to other organizations. meaning: you don’t have to spend time ramping up the new guy whether he’s internal or external resource. if they understand the standards, then they hit the ground running (including usage of automation tooling)!!
you should care because: it impacts your bottom line – whether you are an executive or a consultant, you can improve your own personal profitability! i once bid a competitive project for $30k and 2 weeks with 2 people, where my competitor bid 15 people 90 days and $250k. i won the bid, we completed the entire project in two weeks (yes, with automation, yes with data vault, yes with standards!!!)
the customer thought i was low-balling them, i said: give me a chance to prove a highly mature method for building your edw. they said: ok – because even if you fail we have enough time and money to hire the other competitor. needless to say, we succeeded, the customer hired me on repeated occasions for the next two years.
this is the power of a mature solution build, a mature architecture, a standardized methodology with tried and tested and proven (optimized) processes.
the goal of the maturity model is to address the complexity of enterprise systems as you would any complex problem. it simplifies the problem into small, more manageable chunks. the model is based on the overall purpose of an information system: how well does it increase decision-making efficiency? http://www.controleng.com/single-article/understand-the-maturity-model-to-better-manage-integrate-plant-floor-enterprise-systems/a3c89275a85f5b2e079c2683e2a4a90a.html
what about innovation? why should i stick to standards in the first place?
really? seriously? you’re still asking me why you should bother with standards??? if this is truly the case, go back and re-read the top half of this post until you understand that:
- you cannot be agile unless you are optimized
- you cannot be automated unless you are standardized
- you cannot do things better, faster, cheaper unless you are engaged in a maturity model for the solution.
ok, but innovation says:
break the standards, think outside the box!
true true true, in order to innovate you must push the boundaries. that said, are you familiar with the scientific experiment? it states you need a test bed, a control, and a test environment, you need to seek out false positives, and run impact analysis studies. innovation is great, it’s wonderful – i engage in it all the time – it’s a part of who i am.
that said, truly great innovations: a) stand the test of time, b) are unconditionally applicable (this is how it becomes a standard in the first place), c) are proven to work backed with scientific evidence or mathematical evidence based on many many trials. if you’re lucky, or it’s by design, these trials are non-biased.
new innovations must be tested, vetted, and cleared. this is the responsibility of the innovator whom is challenging the standard!!
note: the results of my innovations (ie: all the standards) are now fully documented in “building a scalable data warehouse with data vault 2.0” – it is far more than just a technical book, i would recommend everyone (including you and your executive friends) read at least the first three chapters.
let’s take a look at where someone “forgot” to follow standards, or “broke them” or “followed their own path because they thought they new better”…
just like any other type of document, build setup documentation can be out of sync without people realizing it. a new module may be added last week, which suddenly implies a new dependency. an important configuration file has changed and therefore simply following the outdated wiki leads to a mysterious failure.
to overcome this problem, consistency must be mandated by a defined process. https://ariya.io/2014/01/a-maturity-model-for-build-automation
the text goes on to talk about how building software requires a well defined (and followed) maturity model. but the author continues with statements about how optimization made a difference in their software builds:
there are two more benefits of this managed automation level. firstly, a multi-platform application is easier to build since the process of creating and provisioning the virtual machine happens automatically. secondly, it enables every engineer to check the testing/staging environment in isolation, i.e. without changing their own development setup. point of fact, tools like vagrant are quickly becoming popular because they give engineers and devops such power. https://ariya.io/2014/01/a-maturity-model-for-build-automation
wait, building a data warehouse is not building software!!
wrong wrong wrong… not true at all. we have development, test, and production enviroments, we have requirements, metrics, management, and releases to manage. we have iterations, revisions, modifications to apply, we have processes (some parallel, some sequential) that have to run, we have data to manipulate and move, we have inputs and outputs to setup and manage, we have deliverables, sign-offs, and buy-offs to handle.
if you believe that a data warehouse is not a software build then you need to go back to school! really!! go read the very definition of the agile manifesto – if you want to be “optimized and be agile”, then according to the manifesto itself – you must be applying software builds and best practices!!
http://agilemanifesto.org/principles.html – refer back to the principles.
why shouldn’t we apply software engineering institute (sei) and it’s cmmi principles to data warehousing? why shouldn’t we follow well-established best practices and proven standards that work??? we should.
with the data vault 2.0 system of business intelligence, you get:
- methodology – matured, optimized, and ready to build at cmmi level 4 (level 5 is achieved by the people involved in executing the optimization)
- model – matured, repeatable, consistent, standardized, and enterprise driven. vetted, and tested to reach petabyte scale data stores without re-engineering.
- architecture – flexible, multi-tier, de-coupled for specific purposes (modular, broken down to be optimized), and applicable to nosql, big data, and real-time (i.o.t – internet of things feeds).
- implementation – best practices, consistency, standards, implementation documentation, agile, and modular. again, vetted, tested, and optimized over the last 25 years.
you owe it to yourself to visit the data vault 2.0 solution, understand truly what the standards are, and why this is so ground breaking for enterprise bi and data warehousing solutions. it is an open standard – fully documented and published in my book on amazon.
you deserve success that comes with following the standards that i’ve provided. these standards were not created in a box all by myself, no – these standards were created with countless person-hours, and teams of people along the way helping me with testing, vetting, controlled experiments at places in the federal government, the nsa, the dod, and others… all to meet government auditability regulations and compliance initiatives.
lastly, if you have any thoughts or comments, i welcome your feedback. feel free to post a comment.
(c) copyright 2016, dan linstedt all rights reserved.