Unstructured Data – Data Mining Visualizations!

if you’ve been wondering how to work with unstructured data, or how it might be visualized then maybe this set of videos will be useful for you to watch.  i’ve been researching the ontology space and visualizations of data mining results in 3d.  i’ve been working in graph theory, mathematics, ontological relationships, centralities, etc… for several years now.  i’m trying to find the best way to visualize and work with data vault model patterns, but that’s for another day.

i usually don’t promote tool sets on this blog (not for the sake of promotion), but this one is different.  it may not be cheap, it may not be available to everyone – but the videos speak volumes about where this type of technology is headed.  this is being done currently in the us government space, but it would be very cool to see this type of evolutionary approach in the commercial side of the house.

imagine the power that this application has.  unbelievable – interactive 3d graphing engine with ai, fuzzy matching, unstructured data scanning, data mining all built in.  phenomenal.

check it out:

introduction 

demo part 1

demo part 2

demo part 3

what do you think of the technology?

dan l

Tags: , , ,

5 Responses to “Unstructured Data – Data Mining Visualizations!”

  1. rklebanov 2010/07/22 at 12:48 pm #

    Hi Dan,
    Currently I am about to embark on building another data vault, this time with unstructured data as a source. I hope that my case is the “easy” type of unstructured data for I can use finite number of transformations to “carve out” my business keys out of lump of raw text.

    So, for my data modeling stage I am thinking to:
    1. Create a series of Hub for raw data with attached load_dt and rec_src and surrogate hub_key.
    2. Then a series or transformations applied to parse raw data to identify business keys to another set of Hubs/Sats with “structured” data. This new set of Hubs will have an inherited Hub keys from “unstructured” hubs for traceability
    3. From this point forward, just regular DV objects and processes.
    4. When DW is built and the Data marts are started, I would keep the bus keys from the Hubs in Dimension tables pointing back to DV data.

    I am looking for high-level recommendations on how to structure DV that have UD component in it and what DV components to keep in the DM that allow traceability back to the source.

    Your help is much appreciated!
    –Raphael

  2. dlinstedt 2010/07/22 at 1:40 pm #

    Hi Raphael,

    I’ll try to answer your questions in another blog posting soon. In the mean time, if you have additional questions, thoughts or comments you’d like to share, I’d love to hear them.

    I’m also curious to know: how does Wherescape Red work for you when building Data Vaults? What are your experiences?

    Thanks,
    Dan Linstedt

  3. rklebanov 2010/07/22 at 3:18 pm #

    Hi Dan,

    WhereScape is working quite well in building hubs and links as Normalized objects ans satellites as History objects. Actually, I have to do very little customization of the code generated by WhereScape. The company finally got serious about making Data Vault methodology fully adapted by WhereScape as creating objects types Hub, Satellites and Links. If you are interested in gory details on building Data vault using WhereScape, I can provide this. But, in general, Wherescape is well adapted even as it is now. I just recently built a data vault phase (almost 500 objects) for the customer and fully populate is in 2 weeks

    Thanks,
    –Raphael

  4. dlinstedt 2010/07/22 at 3:37 pm #

    Hi Raphael,

    Thank-you for the information. That sounds very interesting. I’d like to know more, even get a copy of Wherescape to play with, if they think it’s appropriate.

    Cheers,
    Dan L

Trackbacks/Pingbacks

  1. Tweets that mention Unstructured Data – Data Mining Visualizations! -- Topsy.com - 2010/08/04

    […] This post was mentioned on Twitter by Lex Pierik, Philip du Maine. Philip du Maine said: Nice demo on visualization of unstructured data: http://tinyurl.com/39f2fvv […]

Leave a Reply

*