Data Science

#bigdata #nosql #dv2 and Managed Self Service BI

maybe you’ve heard the term: “self-service bi”?  maybe not.  it’s a blind term in my opinion, and lacks quite a bit in it’s simplistic title.  it truly needs to become: “managed self service bi”.  why? because i.t. (like it or not) is still in the picture, but more than that, business needs / demands a governed or managed process (just like i.t.) in order to affect data and processes in a production environment.

well, well – look how far we’ve come!  not.

does anyone remember eii? (enterprise information integration)  it’s been around a long long time, don’t believe me?  just check my blog entries on

and my personal favorite:

what’s even funnier, is to read the comments from 2005 when i wrote those entries.  the comments (many of them) predicted that eii was a huge success, and would supplant the data warehouse, replace the need for integration & data architecture and data modeling, and so on.

what’s the point?

the point is: eii was on-demand federated query.  the only engine still around today is composite software, and they are doing a great job of improving the premise and the platform.  the point is: we went through all of this “self-service bi” without management, without governance, without data modeling, and supposedly without it…  we’ve been here before!

the point is: self-service bi as a notion / concept is not new.  business users have been trying desperately to return to this state ever since dss (decision support systems) and reporting systems were split from the operational applications.

the point is: eii, federated query, on-demand query, self-service bi is a pandora’s box without good governance, good models, good management and it participation.

the point is: self-service bi isn’t!

the proper term should be: managed self service bi.

ok, read the wikipedia definition: wikipedia defines eii

enough of the rant, back to the topic…

there are a number of tools in the #nosql space that do some really cool things, things that eii had hoped to achieve, but either never got there, or faded out of existence before it could truly mature (again, my hat is off to composite software for sticking around and making a go of it).

today, the terms i find include:

  • data munging (or data mugging as my good friend doug needham would call it)
  • data preparation
  • data refinery (as per claudia imhoff) – although this one takes on new meaning
  • data exploration
  • data blending
  • data wrangling (yee-haw, get your data cowboys/cowgirls ready)

there are a number of good things, and a host of serious innovations that have come from the tools in these spaces – some really cool stuff… but does really cool stuff make for good governance?  how about good data warehousing?  no?  well, hmmm – maybe some more thought is needed in this inventions.

i’m reviewing and evaluating a list of tools in this space, my personal favorite thus far is datameer.  you should check them out.  it has governance, change management, lineage, check-in/out, sql generation, map reduce code generation, and a whole lot more – including neural net analysis visualization.  it’s simple and easy to use (thus far).  other tools in this space that i’m looking at include:

now before you get on your high horse (data wrangler horse) and begin asking me why i didn’t include: qlikview, tableau, cognos, sas, etc.. etc..) my reasoning that i have some basic criteria that these tools must meet: a) must generate map & reduce code, b) must generate sql code, c) must work (not through 3rd party) directly with hadoop based systems, d) must have visualizations built in, e) must allow the users to construct process workflows / business logic, f) must be able to write results back to hadoop, and a few others that i won’t bore you with.  most of the other tools are considered “pure-play bi tools” – or work through partners to get these elements working.  hence not included in my list.

there is even google refine (which has a very cool set of features for refining data, but is lacking some of the more sophisticated interfaces, graphs, and visualizations that datameer has.  if you have experience in any of the above tools, i’d love to hear from you in the comments below.  please tell me the best / top 3 features you love about the tool, and the bottom 3 features you dislike about your tool.  if you’re up to it, let me know the top 3 features you wish your tool had.

i thought this entry was about managed self service bi?

it is, so let’s get back to it.  in managed ssbi, good governance, solid data modeling, and information architecture are key components to successful indulgence.  applicability increases when good business use cases have been defined.  keep in mind, that just because these tools can access / run / generate analytics in a hadoop platform, doesn’t mean you should!   some of these analytical inquires should remain as ad-hoc questions sent in sql form to a scalable database engine capable of returning sub-second response times.

it is time to bring this entry to an end, i will write more as i progress in my investigations of these tool sets.  however, i will leave you with these thoughts

  1. don’t forget to comment, i want to hear from you
  2. managed self-service bi is the proper term
  3. i.t. must still be involved
  4. good governance is a must
  5. good architecture (system, data, information) is a must
  6. relational databases still hold real value
  7. good data models help structure data in an understandable manner
  8. business use cases *must* be defined (otherwise, how will the business know if they’ve found what their looking for?  especially if they haven’t defined what they are looking for)
  9. lots of tooling doing different things
  10. we’ve been down this road before!  with eii – perhaps these new tool vendors could learn a thing or two about researching the problems and issues we had with the eii phase of the market.

of course, “what’s old is new again…” uggh, i seriously dislike this statement.

what do you think?

dan linstedt

Tags: , , , , , , , , , , , , ,

No comments yet.

Leave a Reply