One interesting aspect of big data is how it is challenging the conventional thinking regarding how the non-analytical business user should be using analytics.  An article that I read in 2006 by Chris Anderson titled “The End of Theory: The Data Deluge Makes the Scientific Method Obsolete” really got me thinking about not only the power of big data and analytics, but more importantly what the combination of big data and analytics might mean to the business user experience  The premise of the article was that the massive amounts of data were yielding insights about the business without requiring the heavy statistical modeling typically needed when using sampled data sets.  This is the quote that intrigued me:

“Google conquered the advertising world with nothing more than applied mathematics. It didn’t pretend to know anything about the culture and conventions of advertising — it just assumed that better data, with better analytical tools, would win the day. And Google was right.”

Google became the dominant player in an industry (advertising) without really knowing anything about that industry.  Google achieved this role not by understanding and perfecting advertising techniques, but by applying analytics to massive, detailed data sources to identify what works without having to worry about why it worked.  To me that’s the key “aha” moment that we as big data practitioners need to embrace – that we can leverage these vast, detailed data sets to yield significant, material, and actionable insights on the business. We don’t have to learn old school statistical techniques to understand why certain behaviors occur or why certain things happen.

Falling into the “Analytics Chasm”

The business intelligence world of the 2000’s spent considerable time, money and effort trying to move customers “beyond just reporting.”  The BI vendors added statistical and analytic capabilities to their products, all in the hope of moving business users beyond retrospective reporting into the area of predictive analytics (see chart below).

Unfortunately, the BI tools failed to help business users make the transition because the tools were totally inadequate in helping users understand why something happened these tools required business users to quantify cause-and-effect in order to build the models necessary to predict what to do next, and that was beyond their training and interest.  As a result, the users’ transition to a forward-looking view of their business fell into the “analytics chasm.”

Trying to turn the average business user into a statistical specialist failed, in the early 2000’s and it continues to fail today.  The average business user’s career aspiration is not to become a statistical expert. They are in the retail or medical or telecommunications or banking industries because they like that industry, not because they want to master statistics or manipulate large data sets. The tools today are way too hard to make that process trivial.  So what is one to do?

That’s a topic that I will cover in my next blog.

Subscribe to RSS