In-Database Analytics and your EDW/BI

Well well, we’ve come full circle haven’t we?  There’s an interesting (yet long and dry and somewhat technical) explanation of in-database analytic technology here. (Sybase IQ, Forrester, Fuzzy Logix).  But I have my own opinions.  I’ve blogged and written about these topics for years on and in Teradata Magazine.  In this entry I will explore the meaning, the consolidation, and the relationships to the next generation EDW – which I have also dubbed: Operational Data Warehousing. 

Full circle, what the heck are you talking about?

In the beginning there was OLTP and Applications, and there was a huge debate over where to put application logic: should it be in the database or in the application outside the database?  Over the years, people have realized that a balance of functionality is necessary.

Then came history of data, and up popped data warehousing.  You know this story…  The EDW split off from the OLTP systems, and for what?  REPORTING PURPOSES ONLY.  Then, came the principles of “trending, analyzing, and grouping” the data – pivoting, etc… which lead to application development (and logic / manipulation of data sets within applications).  Including Microsoft Excel which is still used today (along with macros, etc..).

I wrote articles years ago about the “future of data warehousing” in which I stated my belief that eventually, the applications, the logic, and all the controls would be consolidated into a hardware/software platform that had some sort of query logic built in…  along came the appliance market. This market is still in flux, and hasn’t yet matured to the point I envisioned.  None-the-less, here’s Sybase, Fuzzy Logix, and a few other vendors (not to pick on Sybase specifically) that say “in-database analytics” or “deep analytics” is the next best thing since sliced bread…  They discovered that push-down optimization of SQL and APPLICATION LOGIC is more powerful than moving the data out to a middle tier or an application, and then manipulating it.  Go figure!

Anyhow, they’re right…  It IS more powerful, and by far a much better approach (the ETL vendors learned that lesson 8 years ago… why has it taken so long for the reporting and database vendors to reach this conclusion?)

By the way, that was a very unfair question.  Technology had to advance to the point where it is feasible (as it is now) to control the temperature of data, and be able to store information sets in sizeable RAM caches for manipulation.  Compression algorithms, data mining algorithms, faster hardware, advances in SSD storage, and of course: 64 bit OS technology.  All these things make the underlying platform (along with MPP) very attractive for this type of proposition.

So, the BI market is all about: consolidation, consolidation, consolidation?

Yes, yes, and YES.  In other words, the sooner these database vendors realize that moving more and more “data manipulation logic” down to the database layers is valuable, the faster they will move into providing what the users want.  The presentation I mentioned (while dry), does a great job (technically) of telling us what we already knew – Application logic must begin to make it in to the database layers at the core-engine level, along with operational data, and historical data – all combined or consolidated in to a single platform.

The Result?  An OPERATIONAL DATA WAREHOUSE, not just operational BI (this is the output of the operational data warehouse).

How do I get there?

Ahh well, the easiest way is to construct first, a Raw Data Vault – then, begin to feed real-time data to it (pushing the latency cycles of arrival timing ever lower).  From that point, build a Business Data Vault which contains operational data, a vision of master data, and strategic data.  WAIT A MINUTE….  HOLD THE HORSES – (did you miss the discovery here?) 

Use in-database analytics routines to process the data from the RAW DATA VAULT to the Business Data Vault in real-time through data mining and statistical algorithms (deep analytics), resulting in a HOT Master Data store…  Here’s another discovery: use some advanced (new) BI tooling that empowers the business users to build the business logic that runs the analytics on the data being moved from the Raw Data Vault to the Business Data Vault.  DON’T USE ETL FOR THIS LAYER!!

I hope the gravity of these statements sinks in.  What I am proposing is back-end ETL data movement, with sophisticated front-end analytics driving the rules that move data from a raw store to a Business Data Vault.  Throw in MPP, or column based, or SSD, or other new technology and you’ve got yourself a wicked BI system (I think this is more in tune with the next generation of EDW).

IT is no longer responsible for “interpretation layers”, and business users have “nearly” free reign on creating the logic that makes sense of the information.

Yea, yea, I know – there are some engineering challenges and technical hurdles – but this is the fun part!

Deep Analytics, or In-database analytics is one step toward moving more business logic in to the database.  I hope that in the future, the database vendors continue to push these advances (absorbing more and more functionality).  This would make the job of BI applications what it should be: responsible for visualization and manipulation.

Would love to hear your thoughts on the subject…  Did I get it all wrong?  It’s possible….  Register for FREE, and post a comment!

Dan Linstedt

Tags: , , , ,


  1. Tweets that mention In-Database Analytics and your EDW/BI -- - 2010/09/01

    […] This post was mentioned on Twitter by johnlmyers44, Daniel Linstedt. Daniel Linstedt said: In-Database Analytics and your EDW/BI: Well well, we’ve come full circle haven’t we? There’s… […]

Leave a Reply