Technical Modeling Book is Coming….

I’ve talked about it for the past 2 years (at least).  I’m posting the Table of Contents, so you know it’s being written and worked on.  I’ll also post the current list of figures, and a couple of excerpts going forward.

I am Considering: offering subscription levels for access to the book WHILE it’s being written, if you think you are interested, then send me an email telling me: 1) how much you might pay per month for access, 2) what you expect to get in return.

Please keep in mind, that this is a temporary table of contents, and the titles as well as the content is subject to change.  This book is HIGHLY focused on DATA MODELING for the Data Vault.  (Not yet available, I’m hoping it will be available before the end of 2010)

If you think I missed anything, please comment on this blog entry.  If you think there should be a specific example case to use in the book, please forward the details to me.  IF YOU CONTRIBUTE A WORTHY CASE STUDY + DEFINITIONS / EXAMPLE AND I USE IT IN THE BOOK, YOU WILL RECEIVE A FREE COPY OF THE FINISHED WORK.

Abstract:

The purpose of this book is to present and discuss the technical components of the Data Vault Model.  The examples in this book provide a strong foundation for how to build, and design structure in Data Vault modeling.   This book is a second in the series of books surrounding the Data Vault model and methodology (approach).  The target audience is anyone wishing to implement a Data Vault model for integration purposes whether it be an Enterprise Data Warehouse, Operational Data Warehouse, or Dynamic Data Integration Store. 

Table of Contents

1.0     Introduction and Terminology. 7
1.1      Do I need to be a Data Modeler to Read this Book?. 7
1.1.1      Foundational Data Vault Certification. 7
1.2      Review of Basic Terminology. 7
1.3      Notations used in this text 10
1.4      Data Models as Ontology’s. 10
1.5      Data Model Naming Conventions and Abbreviations. 11
1.6      Introduction to Hubs, Links, and Satellites. 13
1.7      Flexibility of the Data Vault Model 14
1.8      Data Vault Basis of Commutative Properties and Set Based Math. 16
1.9      Loading Processes: Batch Versus Real Time. 17

2.0     Architectural Definitions. 18
2.1      Staging Area. 18
2.2      EDW – Data Vault 19
2.3      Metrics Vault 19
2.4      Meta Vault 19
2.5      Report Collections. 20
2.6      Data Marts. 20
2.7      Business Data Vault 20
2.8      Operational Data Vault 21

3.0     Common Attributes. 22
3.1      Sequence Numbers. 23
3.2      Sub Sequence Numbers (Item Numbering) 24
3.3      Load Dates. 24
3.4      Load End Dates. 26
3.5      Last Seen Dates. 27
3.6      Extract Dates. 29
3.7      Record Creation Dates. 30
3.8      Multi-Temporal Date Structures. 30
3.9      Record Sources. 30
3.10   Process ID’s. 31

4.0     Hub Entities. 32
4.1      Hub Definition and Purpose. 33
4.2      What is a Business Key?. 34
4.3      Where do we find Business Keys?. 35
4.4      Why are Business Keys Important?. 36
4.5      Why not Surrogate Keys as “Master Keys”?. 37
4.6      Hub Smart Keys, Intelligent Keys. 38
4.7      Hub Composite Business Keys. 38
4.8      Hub Entity Structure. 39
4.9      Hub Examples. 40
4.10   Dependent and Non-dependent Child Keys. 41
4.11   Mining patterns in the Hub Entity. 42
4.12   Process of Building a Hub Table. 44
4.13   Modeling Rules and Standards for Hub Tables. 45
4.14   What Happens when the Hub Standards Are Broken. 45

5.0     Link Entities. 47
5.1      Link Definition and Purpose. 47
5.2      Reasons for Many To Many Relationships. 47
5.3      Flexibility. 50
5.4      Granularity. 52
5.5      Dynamic Adaptability. 52
5.6      Scalability. 53
5.7      Link Entity Structure. 55
5.8      Link Examples. 55
5.9      Link-To-Link (Parent/Child Relationships) 55
5.10   Link Applications. 56
5.11   Hierarchical Links. 56
5.12   Same-As Links. 56
5.13   Begin and End Dating Links. 56
5.14   Low Value Links. 56
5.15   Transactional Links. 56
5.16   Computed Aggregate Links. 56
5.17   Vector Links (Directional) 56
5.18   Strength and Confidence Ratings in Links. 57
5.19   Exploration Links. 57
5.20   Capturing Changes to Business Rules Over Time. 57

6.0     Satellite Entities. 57
6.1      Satellite Definition and Purpose. 57
6.2      Satellite Entity Structure. 57
6.3      Satellite Examples. 57
6.4      Satellites Arranged by Source Systems. 57
6.5      Satellite Applications: 57
6.6      Record Tracking Satellites. 57
6.7      Status Tracking Satellites. 57
6.8      Computed Satellites (Quality Generated) 57
6.9      Splitting Satellites. 57
6.10   Consolidating Satellites. 57

7.0     Query Assistant Tables. 57
7.1      Point in Time Tables. 58
7.2      Bridge Tables. 58

8.0     Reference Tables. 58
8.1      Code and Descriptions. 58
8.2      National Drug Codes. 58
8.3      ICD9 Diagnosis Codes. 58
8.4      Calendars (Financial and Gregorian) 58

9.0     Additional Data Vault Thoughts. 58
9.1      Introduction to a Business Based Data Vault 58
9.2      Metadata and the Data Vault Model 58
9.3      Master Data and the Data Vault Model 58
9.4      Growth Patterns and the Architecture. 58 

Table of Figures

Figure 1-1: Example E-R Diagram (Elmasri/Navathe) 9
Figure 1-2: Crows Foot and Arrow Notation Example. 10
Figure 1-3: Small Example Ontology for Vehicle. 11
Figure 1-4: Abbreviations and Naming Conventions. 12
Figure 1-5: Example Data Vault 13
Figure 1-6: Flexibility of Adapting to Change. 15
Figure 1-7: 3rd Normal Form Product and Supplier Example. 15
Figure 1-8: Applied Set Theory for the Data Vault 17
Figure 2-1: Enterprise BI Architectural Components. 18
Figure 3-1: Time Series Batch Loaded Data. 22
Figure 3-2 Real-Time Arrival, Data Geology. 23
Figure 3-3: Load Date Time Stamp and Record Source. 25
Figure 3-4: Example Load Date Time Stamp Data. 25
Figure 3-5: Load End Date Computations, Descriptive Data Life Cycle. 26
Figure 3-6: Structures containing Last Seen Dates. 27
Figure 3-7: Left, Scan all data in EDW, Right: Scan Reduced Set 28
Figure 3-8: Reduced Scan Set after Applying Last Seen Date. 29
Figure 4-1: Business Key Changing Across Line of Business. 32
Figure 4-2: Hub Example Images. 33
Figure 4-3: Hub Example Data. 34
Figure 4-4: Smart Key Example. 38
Figure 4-5: Composite Business Key Hub Example. 39
Figure 4-6: Example Hub Entity Structure. 39
Figure 4-7: Example Hubs from AdventureWorks 2008. 40
Figure 4-8: Example of National Drug Code Data Vault 41
Figure 4-9: Dependent Child Relationship Modeling. 42
Figure 4-10: Typical Hub Row Sizing. 46
Figure 5-1: Relationship Changes Over Time. 48
Figure 5-2: Link Table Structure Housing Multiple Relationships. 49
Figure 5-3: Starting Model Before Changes. 50
Figure 5-4: Data Vault After Modification. 50
Figure 5-5: Additional Data Vault Model – More Changes. 51
Figure 5-6: Global Data Vault Linking. 51
Figure 5-7: Uncovering Fact Table Grain. 52
Figure 5-8: Data Vault Grain, Representing Star Schema. 52
Figure 5-9: Traditional Data Vault Storage Layout 54
Figure 5-10: Performance Physical Split Version 1. 54
Figure 5-11: Performance Physical Split Version 2. 54
Figure 5-12: Performance Physical Split Version 3. 55

_________________________________

Thank-you,
Dan Linstedt

DanL@DanLinstedt.com

Tags: , ,

5 Responses to “Technical Modeling Book is Coming….”

  1. av1234 2010/06/08 at 3:14 am #

    Dan,
    I notice that there is a section in your proposed book on end-dating links. This is something that I am considering for a Data Vault Application, but I have not found any previousl mention of end-dating links in the literature, only of end-dating satellites.

    Is the principle of end-dating lins the same as for end-dating satellites, and are there any catches that I should watch out for?

  2. dlinstedt 2010/06/08 at 4:19 am #

    Hi,

    End-Dating Links is NOT something you should do, ever. But be that as it may – the section covers one (only one) exception of when you can do it and why. The section also covers what happens when you DO end-date links, and the risks you take by implementing the solution (breaking standards). Technically it is a similar principle to end-dating Satellites.

    To put it shortly: only transactional links (links with no children) can be end-dated. But this is NOT the best practice. The best practice is to add a time-series Satellite that shows the start and end relationships over time.

    Cheers,
    Dan L

  3. av1234 2010/06/10 at 2:41 am #

    Dan,

    Thanks for the clarification. Referring to the example in Data Vault Basics (http://danlinstedt.com/about/data-vault-basics/), if for example the shipper of an order changes, then you are saying that the SAT_SHIPMENT record should be end-dated, and a new LNK_SHIPMENT and SAT_SHIPMENT record created?

  4. dlinstedt 2010/06/10 at 3:57 am #

    Hi,

    Yes, I am saying that if something in the link changes – and it’s NOT a driving key, then the OLD record in the satellite needs to be end-dated. In this case, Order number is the driving key (doesn’t change/shouldn’t change) – where Shipper CAN change (because of mistakes or other reasons), you are correct in your assumption.

    Hope this helps,
    Dan L
    Please don’t forget to tell people about my site & refer them here.

  5. av1234 2010/06/10 at 7:03 am #

    Dan,
    In that case, wounldn’t new LNK_ORD_ITEM, and consequently SAT_ORDITM records also have be be created to link to the new LNK_SHIPMENT record?

    Could this knock-on change be avoided by joining the LNK_ORD_ITEM to the HUB_ORDERS record instead of the LNK_SHIPMENT, or is there a catch with doing this that I haven’t spotted?

Leave a Reply

*