Hello everyone, I have been able to help teams achieve one day sprints for delivery with Data Vault 2.0! I am happy to say, that following Disciplined Agile Delivery (DAD by Scott Ambler and Mark Lines) that we can reduce the sprints to half day or one day. In this entry I will talk about the benefits of getting there.
In my recent travels I worked with quite a few teams. In particular, I was able to work with a highly qualified and specialized team at a particular customer. Together, we were able to put together one day sprint cycles for moving data from source all the way through the Data Vault, and out to the BI tooling layer. This requires special focus by the team, by the scrum master / team leader in order to accomplish. That said, there is no “magic fairy dust” involved!
How did we do it?
Number one issue for Data Vault teams is: lack of understanding of how to scope properly. By scoping down to the proper level, we were able to get the data processed all the way through to the BI layer. By the way, I have done this before with a number of other teams as well, this is not unusual for Data Vault 2.0 methodology practices!! This is the normal methodology implementation when done properly… Generally these types of things are taught in workshop settings and Kick Start packages on-site.
- split tasks in to different sprints
- scope tasks down to single delivery points (see #6 below).
- SCOPE should be limited to 3 to 4 source systems at most, and ONE reporting output
- Critical!! have the Data Vault model built before starting (just the portion that will be utilized)
- Have the right team (5 or 6 individuals, maybe 10 at most)
- Involve a Data Vault 2.0 Certified Team Coach / agile delivery master
- REQUIREMENT UP FRONT: Find a SINGLE output and build to that. The team must:
- already understand the data (already have done the profiling if needed)
- already have access to the source system
- be able to stage the data in a matter of minutes (10 minutes at most)
Steps to build:
schedule a war room – get the entire team in the room, get them focused, and build. Use the white-boards to iterate, use the overhead projector to iterate.
Note: Automation tooling can dramatically increase productivity, reducing build time from an hour or two, down to minutes. I recommend either WhereScape 3d & Red or AnalytixDS Mapping Manager as my preferred Data Vault automation solutions.
- Load data to stage
- Build Hub, Link, Satellite Loads (generate if possible)
- Execute Loads (Unit Test!!!)
- Design PIT and Bridge Tables in Data Modeling Tool, generate to Business Vault
- Either: Generate PIT & Bridge load code, or write it by hand (Unit Test!! Load Data!!)
- Design Virtual Mart Views (virtual dimensions, virtual facts) UNIT TEST
- Open BI Query Tool, ingest views, and build report (UNIT TEST)
- Then, begin Data Testing (yes there are DAD steps to testing as well)
Deliver! Deliver! Deliver!
By the end of the day, the team should have output to the BI query tool. If this doesn’t happen in a single day, then either the scope was too large, or the prerequisites I listed were not followed! That means, go back and do it again. It may be helpful to hire an authorized Data Vault 2.0 coach or trainer to get this process under way quickly. My authorized trainers are all trained in Agile Data Vault delivery practices, and can guide your team to successful day-long sprints.
Not only did we deliver, we also iterate during the day long sprint as well!!
Separate the Sprint Cycles folks, but always always always, demonstrate data value in the BI layers!! as fast as possible. In 1997, with a team of 3 people that I managed, we had 2 hour turn-around times for the BI delivery cycles, serving 5000 business users*** (qualified: this was the information delivery cycles only, all data was identified and loaded already in the Data Vault).
Other Suggested Day Long Sprint Cycles:
Remember: these are all targeted at a SMALL SCOPE for rapid delivery…
- Developing the Data Vault Model (Identifying Business Keys)
- Gathering Requirements (should be a 45 minute sprint!!)
- BI Report layout & build (this should be done BY the business users, NOT by I.T.) – this is self-service BI.
- Development of Business Vault pieces
Common Misconceptions in Data Vault Landscapes
There are a couple of common misconceptions that need to be dispelled. Bad practices that are followed and need to be stopped immediately. If you are doing these things, please readjust and optimize your flow, or call an authorized and certified individual (coach or trainer) to assist. These misconceptions drive LONG delivery cycles, blow budgets, blow expectations, and these projects never finish properly or on time, or in budget.
- Thinking you have to build the entire Data Vault model before you build any output.
- NO NO NO… please don’t do this!! This can lead to catastrophic failure of the entire project. The DV model is designed to be built incrementally and in pieces. It is designed to be agile, and to be flexible to change going forward.
- Waiting for business to sign off on requirements BEFORE you start building and loading the Data Vault model.
- No, you don’t have to wait! If you have a few Subject Matter Experts or a BA on staff who can attend the design session for the limited output scope, they should be able to give the team enough information about the business processes and business keys to get the Data Vault model built, and to identify the source systems where the data comes from. DON’T WAIT for requirements!! Get the DV Model built and loaded as fast as possible.
- That a Business Data Vault is required to get data out.
- Nope, not true!!! BDV’s are sparsely built – and only built for several reasons (covered in CDVP2 certification class). What needs to be built are the Point-in-time and Bridge Table structures.
- That you can *never* show a development prototype to business users
- Wrong again, during the day long sprint, the team should have a BI dashboard by the end of the day on the virtual facts and virtual dimensions. There should be a business analyst in the room to help qualify / examine the output and make corrections. A demo should be scheduled as soon as possible with at least the stakeholder (possibly the next day) to show output!! Even if it is a development data set.
- Building Virtual Facts and Virtual Dimensions or Views directly against the Raw Data Vault without Point-in-time or Bridge tables.
- Please don’t do this!! Please build Point-in-time and Bridge Tables, your queries will run very very fast if built properly, and should handle extremely large data sets with ease.
Day long sprints for different components are achievable. I have helped many teams in just the past year get to this point. Better faster cheaper is built in to the Data Vault Methodology!! This is only available in Data Vault 2.0 Training and coaching from authorized partners. Utilizing automation tools from WhereScape and AnalytixDS decreases time to market dramatically.
If your “data vault effort” is stagnating, and not delivering, then something is seriously wrong. Time to get it fixed!! Time to change the way the team is working, time to hire an authorized Data Vault 2.0 Trainer / Coach.
Have a success story from your Data Vault project? Have a problem with your current efforts? Please comment and share below.
Hope this helps,
Dan Linstedt (C) 2017 All Rights Reserved.