With people now placing their whole life stories on the web, there comes an unprecedented opportunity to better understand your customer and clients.

This would include such areas as:

Customer Interests

• What sports team do I like?
  

Behaviors

• What places do I frequently visit?
  

Directly Marketing Ideas

• What interest do I have that you can provide me with something of need/desire for?

I am referring to this opportunity as the new “Social Customer Intelligence” or for those of us in the IT space who are driven by TLA’s (or Three Letter Acronyms like: SOA, EDA, MDM,…) – “SCI”.

What Is Social Customer Intelligence [SCI]

Everyone who sells or delivers a service has customers / clients and knowing more information about them (than your competition) puts you in a position to gain market share or lead the industry.  SCI, to this author, is taking all of the information on the web about an individual and marring this up with internal information about your customer to create a “bigger picture” as to how we can best serve our clients and dominate our market.

Taking data, resolving it and then mining and understanding customer feedback develops a powerful lens into the customer experience. The problem, however, is that while many organizations are just starting to listen to the online chatter, too few are deciphering what they have gathered in a meaningful way. In fact, a recent survey by Harvard Business Review Analytic Services found that while more than half of the surveyed companies are using social media, less than one-quarter (23%) are using any form of social media analytical tools and only 5% are using some form of customer sentiment analysis.

Online client chatter is an unparalleled opportunity to identify and remove the obstacles to delivering superior customer experience. Social media analytics (or customer sentiment analysis) is a brand new tool in our IT bag to realize this unique opportunity in time (and space).

Unfortunately, most available analytic programs gravitate toward quickly available and “small” data such as ratings (stars, etc.), “followers,” “likes,” because they are rapidly attained but severely lack in the valuable information that provide for actionable analysis and provide the value executives need to drive business decisions.

Social business intelligence — or the intersection of business intelligence, social media customer feedback, and advanced “Big Data” technology — provides organizations with a data source of customer-specific desires which in turn are performance indicators and opportunities to grow your business. By capturing and aggregating a complete view of social media feedback and synthesizing it into customer insight, social business intelligence turns volumes of unstructured mentions online into a real-time performance indicator dashboard that provides clear opportunities to improve the customer experience.

A true social business intelligence platform delivers many areas of actionable insight. Consider a bank that knows only that it has a customer with an age, the amount invested in it’s “products” and that the client just stopped direct deposit.  Using data on Linkedin®, they could determine that this customer has joined a new firm and solicit them for a rollover IRA.  Or if this client liked the Boston Red Sox, they could offer a signature “BoSox” Debit Card.  This individual would be much more likely to “want” this product and the bank would be leap years ahead of their competition.   Let us explore some of the areas where SCI has applications:

Social Customer Satisfaction [SCS]

Your customers are tweeting, posting, and blogging, about whether they like your products, will buy your product again, return to your store, or recommend you to their peers. As a result, it is critical for corporations to not just resolve the existing issues, but to be proactive in averting future issues that may come down the pike. Advances in analytics — including sentiment and credibility analyses — give marketers the power to drill down into the most relevant customer feedback and identify ways to enhance the overall customer experience.

Social Competitive Intelligence [SCI]

In today’s socially connected, consumer-led world, there are few places that are more effective at helping companies gather a goldmine of information than social media. Social competitive intelligence enables organizations to understand how the competitors’ products are performing relative to yours, and other items as defined by customers online.  With social competitive intelligence, we would analyze it so they have a clear and timely understanding of competitor strengths and their latest initiatives, your product weaknesses, and other areas.

Social Marketing Intelligence [SMI]

How do marketers better understand when, where, why, and how consumers choose to engage with their brand? How do they know what social media channels are most effective? Most importantly, the wealth of information about important insight into your own products, how you can improve your products, how are they used (in a way that you did not expect or intend, etc.) and what innovations are being suggested by “savvy” consumers.

Analyzing online feedback gives organizations insight into the relevant customer impressions of their products or services and identifies and gives access to the most pertinent online influencers.  Did you know that a telecomm provider found out that highly connected individuals were 7X more likely to take their fellow callers away if they left?

Consider also that you can gain critical demographic data on customers online, engagement opportunities, and true voice-of-the-consumer insight. This new type of intelligence gives organizations the ability to measure the value of specific social media interaction, which is vital to developing online marketing strategies and informing ROI analysis.

Social Advertising Intelligence [SAI]

Every brand wants to capture new audiences, add revenue opportunities, and maximize the value of advertising while minimizing inventory. Mining and analyzing volumes of real-time, highly influential online customer feedback gives organizations the intelligence they need to learn from and transform messaging campaigns. Armed with this direct social media feedback, retailers can easily measure the efficacy of campaigns during and afterwards.

A Must Have in Today’s BNW

Alas poor Yorick, a comprehensive SBI program is a must-have in today’s Brave New World [BNW]. Just as replacing cash registers with POS systems in the late 90’s was required. Why, because the data coming from the POS systems provided insight into who was buying what products with what other products?   At Polo Ralph Lauren™, when I was the Sr. Director of Global Applications, I learned that white RL Polo™ shirts sold better next to blue dress pants.  I did not know why, but my BI solution found this trend.  The stores changed their planograms and sales increased.  By mining and analyzing customer feedback intelligently, organizations gain the depth and breadth of customer insight needed to leverage social media for a true, bottom-line competitive advantage.

Parting Shots…

Consider the banking example I used earlier.  Their traditional or existing relationship of growth or customer up-sell and cross-sell or the growth of their business was limited to:

• Reactive campaigns
• Client directly expresses needs
• Broad relationship offers
• Traditional sales cycles
• Direct mail response rates

Now consider the brave new world of SMI.  There would be a vastly accelerated relationship growth if this financial institution used SMI and the information derived from these sources to grow their business by using:

• Predictive campaigns
• Expressed and inferred needs
• Determine service level needs on a client grouping basis (What min. client SLA’s are required)

Personalized relationship offers

• Red Sox Affinity Card offers
• Card discounts on Target® purchases as we learned via tweet that the customer just entered the store in NYC
• Real time sales cycle

Rollover IRA sales call based upon change in position in real time.

• Improved marketing responses

Yes, it is a brave new world and EMC is leading the charge (not the credit kind)…

Getting At Your Information…

One of the keys to Business Intelligence is being able to get at your information.  Yes, speed is of importance and the value of information is directly proportional to how fast you can get it.   With servers, this announcement will provide a new DNA for running the infrastructure of the enterprise.

But getting at something fast, if you don’t know what it is, does not have much value – as is the case with Business Intelligence. BI as a Service, that was recently discussed by Chuck Hollis (the CIO of EMC Marketing) in his blog, noted the need to get information out into the community of users is one of the primary drivers.  It is one of many information intelligence and Big Data drivers including revolutionizing the business model and gaining insights that heretofore were just not thought of.

Big Data: From the “Old” Technology Stack to the “New”

There are many reasons or drivers why there is a need to change to a new technology stack (including Flash memory as part of the solution) for Big Data, let me try to name a few:

Yes there is a need for a completely new paradigm for the ingestion, processing, security and analysis of information and the EMC’s CTO’s office is at the forefront of this research.  I have been fortunate to stand on the shoulders of giants in this space (Jeff Nick, Dan Hushon, David Reiner P.h.D. and Nihar Nanda) and we are working toward a newer Information Management Platform with completely different DNA than its predecessors.

Big Data: The New DNA of an Information “Fabric”

Any new information platform for fabric would have to have a new DNA; one that promotes agility, business value, security and community sharing while removing the limitations imposed by existing solutions.  Consider that the strands would have to contain answers for the following dilemmas today:

Businesses Want Integrated, Timely Information for Purpose

Based upon these needs for a new DNA, the new strands have been renamed from their previously used counterparts:

Stay Tuned…

Yes, lightening is striking twice – once to change the way we store and get to information faster and the other is creating a new breed of solution with a new branch on the evolutionary path.

Feel free to post your comments and let me know your thoughts on what you perceive is the new “DNA” for information.  For it is in the dialogs of comments and exchanges of thoughts that we grow – that is learning…

66% Were Looking In All The Wrong Places

The EMC Global Data Science Study covered the United States, the United Kingdom, France, Germany, India, and China.  This study revealed and quantified a rampant scarcity across the globe of the prerequisite skills necessary for a company to capitalize on the opportunities found at the intersection of Big Data and data analytics!  The survey revealed that the explosion of digital data created by mobile sensors, surveillance, medical imaging, smart grids, mobile phones, and social media —combined with new tools for analyzing it all—has created  the opportunity to generate value and insights from the data.

“Only one-third of companies are able to effectively use new data to assist their business decision-making, gain a competitive advantage, drive productivity growth, yield innovation, and reveal customer insights. ”

Calling All Job Seekers

As such, the business demand for data scientists has quickly outpaced the supply of talent. The EMC Data Science Study respondents included nearly 500 members of the data science community. The community includes data scientists and professionals from related disciplines such as data analysts, data specialists, business intelligence analysts, information analysts, and data engineers, all of whom have IT decision-making authority.

Key Survey Takeaways

Here are 12 key takeaways from the survey in case you missed them:

• Informed Decision-making—Only 1/3 of respondents are very confident in their company’s ability to make business decisions based on new data.
• Looming Talent Shortage—65% of data science professionals believe demand for data science talent will outpace the supply over the next 5 years – with most feeling that this supply will be most effectively sourced from new college graduates.
• Barriers to Data Science Adoption—Most commonly cited barriers to data science adoption include: Lack of skills or training (32%), budget/resources (32%), the wrong organizational structure (14%), and lack of tools/technology (10%).
• Customer Insights—Only 38% of business intelligence analysts and data scientists strongly agree that their company uses data to learn more about customers.
• New Technology Fueling Growth—83% of respondents believe that new tools and emerging technology will increase the need for data scientists.
• Lack of Data Accessibility—Only 12% of business intelligence professionals and 22% of data scientists strongly believe employees have the access to run experiments on data. This undermines a company’s ability to rapidly test and validate ideas and thus its approach to innovation.
• Advanced Degrees—Data scientists are 3 times as likely as business intelligence professionals to have a Master’s or Doctoral degree.
• Augmenting Business Intelligence—Although respondents found an increasing need for data scientists in their firm, only 12% saw today’s business intelligence professionals as the most likely source to meet that demand.
• Higher-Level Skills—Data scientists require significantly greater business and technical skills than today’s business intelligence professionals. According to the Data Science Study, they are twice as likely to apply advanced algorithms to data, but also 37% more likely to make business decisions based on that data.
• Love the Work—The study discovered highly favorable attitudes toward the companies where they work. In fact, data scientists believe their IT functions are better aligned and better able to attract talent, are ahead in key technological areas like cloud computing, and not surprisingly rate their company’s data analysis and visualization abilities very favorably compared to the views of business intelligence professionals.
• Involved Across the Data Lifecycle—Data scientists are more likely than business intelligence professionals to be involved across the data lifecycle–from acquiring new data sets to making business decisions based on the data. This includes filtering and organizing data as well as representing data visually and telling a story with data.
• Tools of the Trade—Data scientists are more likely than business intelligence professionals to use scripting languages, including Python, Perl, BASH and AWK. Yet, Excel remains the tool of choice for both data scientists and business intelligence executives, followed closely by SQL.

Data Scientists Quotes

Some of the quotes that came out of the survey were indeed insightful and provided a glimpse into the future of Big Data:

“We live in a data-driven world. Increasingly, the efficient operation of organizations across sectors relies on the effective use of vast amounts of data. Making sense of big data is a combination of organizations having the tools, skills and more importantly, the mindset to see data as the new “oil” fueling a company. Unfortunately, the technology has evolved faster than the workforce skills to make sense of it and organizations across sectors must adapt to this new reality or perish.“

Andreas Weigend, Ph.D. Stanford, Head of the Social Data Lab at Stanford, former Chief Scientist Amazon.com

“Neither tools nor people alone can solve the challenges of Big Data. They must work together and that is the promise of data science. Despite advances in software tools, the number of people with experience using these tools, and with real-life exposure to large-scale data sets, is small. Data science is a young field, and its growth will be fueled as much by technology as through the mentorship of new acolytes by leading practitioners.”

Michael Driscoll, Ph.D. Boston University, Co-Founder and CTO at MetaMarkets

The Bottom Line

It is this author’s opinion that big changes are coming and there is a wealth of opportunity for those who are looking for a steady profession, just pick “Data Scientist” and you won’t have to worry about unemployment in this difficult economy!

So, I would suggest that businesses make a resolution to implement a Big Data solution this year – or be prepared for the consequences.

The Consequences of Not Resolving to Adopt Big Data

The media is a buzz about big data and rightfully so, Big Data is changing the world we live in.  In May 2011, Gartner noted: “Big data will represent a hugely disruptive force during the next five years, enabling levels of insight that are currently unachievable through any other means”.  By July 2011, they were saying “Through 2015, organizations integrating high value, diverse new information sources and types into a coherent information management infrastructure will outperform industry peers financially by more than 20%”.

The McKinsey Global Institute noted in its May 2011 report that: “analyzing large data sets—so-called big data—will become a key basis of competition, underpinning new waves of productivity growth, innovation, and consumer surplus… From the standpoint of competitiveness and the potential capture of value, all companies need to take big data seriously. In most industries, established competitors and new entrants alike will leverage data-driven strategies to innovate, compete, and capture value from deep and up-to-real-time information. Indeed, we found early examples of such use of data in every sector we examined”

The Paradigm Shift of Big Data

Yes, the very fabric of Business Intelligence and Data Warehousing is shaking as this new “paradigm”, the next generation of BI/DW, takes hold – allow me to elaborate:

With the data at hand it’s easy to see the growing influence Big Data is having, and will continue to have, on businesses going forward. Given what we predict about the definitive role Big Data will play in decision-making, advancing without being prepared would be naive. It is best to use this insight not only to be ready for the future, but more importantly prosper because of it.

For my next blog (The Business Case for Big Data Part 2), I will dig into the recently released EMC Global Data Science Study. I’ll cover in detail what the study found and give helpful insight on how people can best capitalize on the myriad of changes Big Data is bringing to business.

The data quality/data governance challenge is how to ensure that the business-critical decisions being made can be accomplished with high enough confidence that you can move from data governance to the realm of decision governance (i.e. all decisions are made with the best available information; we are acting on what and when our information says we should, etc.)

As your organization evolves to exploit big data to enable the business, there is a clear tie between the quality of decisions and the quality of the information used to derive the insight or formulate the conclusion. As such, organizations must:

• Understand their data quality issues as a feasibility watermark on any potential big data solution
• Address these issues to build a trusted view of the business information and enable high confidence in decision making
• Embrace governance (i.e., make organizational and process adjustments) to ensure that the quality of that data does not degrade overtime

With good data quality and solid data governance practices in place, the mature organizations can even evolve to a level of decision governance. Imagine policies and procedures to measure the organization’s effectiveness of executing data-based decision-making (i.e. all decisions are made with the best available information; we are acting on what and when our information says we should, etc.)

For example, a large Life Sciences company is looking beyond data governance to advance the concept of decision governance, where the organization can measure how often its stakeholders use analytics as the basis for their decisions as well as the effectiveness of those decisions.

Cloud Adds Even More Agility

How do you leverage best practices from the cloud world to ensure IT can be as agile and cost effective in the provision of data solutions as they are in applications?

For the same reasons that cloud is a compelling platform for home grown or commercial applications, it also has applicability for big data:

• IT can lower costs by consolidating, virtualizing, and reusing pooled resources with simplified infrastructure management and administration
• Big data can leverage elastic architectures that expand and contract with business needs– including the potential for cloud bursting
• Business users can provision data marts and analytic sandboxes, on demand, with self-service
• Virtual data repositories can be shared to enhance collaboration
• Data repositories can be catalogued and tracked for increased visibility and enhanced compliance

For instance, a very large financial services firm today has thousands and thousands of data repositories spread across the organization. Not only is it difficult to track what data resides where for compliance reasons, but there is very little reuse and IT struggles to respond in a timely manner when the business has a new need. They are moving towards a vision of a service catalog of available data offerings – virtualized, standardized, and templatized for rapid deployment and easy archive and reclamation.

The Bottom Line

It is this author’s opinion that every big data solution should fully consider these 5 dimensions:

• Business Intelligence and Analytics
• Agile Data Platform
• Data Quality and Data Governance
• Security
• All in a Cloud Data Management fabric

And not just  tools and technologies, but also have a plan for the potential new roles and processes as well (such as the data scientist and data quality/governance on ingestion).

So how do you get started? This author agrees with McKinsey that the best approach to big data is not trying to boil the ocean all at once. Instead, identify a few key potential big data opportunities, experiment, and then rapidly scale successes.

In order to deliver something of high value to the organization, it is critical to start the project with a laser focus on your best-fit, priority business opportunity – so focus using a prioritized scheme of what is the highest value, with the largest business benefit at a relatively low risk of failure.

Once you have established the “right” business opportunity, ask an expert to execute the investigation and requirements capturing processes across the 5 EIM dimensions. And while many of these tracks can run in parallel, there is close track-to-track collaboration (in order to generate synergies around the business opportunity) so an expert will guide you on your journey knowing the potholes.

At the end of the process, you should be presented with a phased implementation roadmap, and recommendations for transforming your EIM technologies and organizational capabilities in order to support this best-fit, priority business opportunity.

This approach ensures that what you receive is something of high business value and relevance to your “customers”, and not just a generic advisory report that does not drive specific business value.

That is what I like about working at EMC, I get the opportunity to do it all – capture, analyze, refine, prioritize and deliver, and that’s why we are successful. We don’t just talk the talk, we walk alongside with you through the journey!

Comments, thoughts, opinions are always welcomed…

How much is the landscape changing, one might ask. It was downright amazing to this author to learn that Microsoft had abandoned its big data projects and choose instead to go to open source:

“Microsoft is not only putting its weight behind Hadoop, the open source platform for crunching large amounts of data across thousands of servers. It’s abandoning the proprietary platform it built to do much the same thing. Late last week, a blog post from Redmond announced that the company would stop development on LINQ to HPC, aka Dryad, a distributed number-crunching platform developed in Microsoft’s Research Lab. Instead, the company will focus on its effort to port Hadoop to its Windows Server operating system and Windows Azure, its online service for building and deploying applications.”

Wired.com 11/17/11 “Microsoft Kills Own Big Data Project In Favor of Open Source”

Most have to recognize the investment that enterprises have already made, and encourage reuse where possible.  However, businesses should recognize that smart big data planning must consider more than the core data platform.  I believe there are 5 key dimensions that should be addressed in the context of any big data opportunity.  And, done right, IT can make choices that transform a potential problem into a powerful asset.

• BI and Analytics – How to transition from reporting on the past/present to getting insights on why things happened and eventually predicting the future (while leveraging a host of new data visualization abilities to empower users across a myriad of devices)
• Data Warehousing – How to develop a data platform purpose built to address volume, variety, complexity, and velocity  and that is agile enough to accommodate new sources quickly and with enough flexibility to address previously un-modeled and even unthought-of insights in 2^nd and 3^rd level questions
• Data Quality/Data Governance – How to ensure that the business-critical decisions being made can be made with such confidence that the organization can move from data governance to the realm of decision governance (i.e., all decisions are made with the best available information; we are acting on what and when our information says we should, etc.)
• Cloud – How to leverage best practices from the cloud world to ensure IT can be as agile and cost effective in the provision of data solutions as they are in provisioning servers and storage
• Security – When big data meets the cloud, the data could live anywhere. How does one ensure information remains secure, protected, and meets your compliance obligations?



BI & Analytics

The BI and analytics challenge is how to transition from reporting on the past/present (looking in the rear view mirror analogy) to getting insights on why things happened and eventually predicting the future (looking through the windshield). This simultaneously leverages a host of new data visualization abilities to empower users across a myriad of devices.

In the realm of business intelligence and analytics, there have been major advances with things like:

• In-database analytics that enable a whole new class of problems to be contemplated and solved without waiting for days
• The ability to incorporate unstructured data (documents, emails, other) into queries and analytics with technologies like Hadoop and MapReduce, which allow all corporate information to be considered
• Closed loop systems where analytic results can be operationalized and integrated with the LOB systems required to take action (e.g., CRM can be notified of at risk customers, etc.)
• New data visualization capabilities that can be applied to empower the user to derive conclusions and take action. They can be deployed to browsers, tablets, and even phones

For example, a large brokerage firm is integrating social media data in its customer records so that it “mines” that information to both: (1) Be more understanding of and responsive to its current customers and (2) Identify other like-minded people who may be good potential clients for similar services.  As I have noted previously, big data is very analogous to digging through tons of dirt to find that speck of gold…

In another example, Dr. Virginia A. Cardin writes

“Big Technology will enable the virtual movement of the scientist to the data rather than the data to the scientist… provides kinetic energy to the R&D cancer community as life science and medical research merge… will continue to fuel bioinformatics that will lead to the prescription to beat cancer… Value:  New subtypes of breast cancer have been defined, and the analytical methods are applicable to other cancers”

Frost & Sullivan White Paper: “Big Science à Big Data à Big Collaboration”

Agile Data Warehousing Technologies

The data warehousing challenge is how to develop a data platform purpose built to address volume, variety, complexity, and velocity and that is agile enough to accommodate new sources quickly while being flexible enough to address previously un-modeled 2^nd and 3^rd level questions.   Sounds impossible, right?  Travel to the moon seemed impossible as well, but we did that.

Agile data warehousing is no longer a theory, it’s a reality. Massively Parallel Processing (MPP) data platforms enable architectural agility.

• Data can be stored at an atomic level without the needs for aggregation tables, materialized views, and tons of indexes. This thereby accommodates the addition of a new data source with much less complexity
• Information does not have to be pre-constructed in anticipation of the users need, and the platform is more dynamic to support answers to previously un-modeled 2^nd and 3^rd level questions
• Interfaces and extensions (API’s) have been pre-built to accommodate unstructured as well as structured content
• ELT (Extract, Load, then Transform) creates powerful new paradigms that not only enhance speed but also enable the creation of new composite metrics when combined with other enterprise data

For example, a regional bank found that their existing Oracle-based data warehouse had become over-run with the endless addition of new data marts (over 1,000 data marts). They implemented an agile data warehouse environment that increased the timeliness of decision-making, while still providing a data platform that can ingest more diverse data sources more rapidly.  Since implementation, their production data warehouse has been able to grow from 4TB to over 10TB, while improving the timeliness of end user queries and IT’s ability to integrate new, more detailed data sources.

In some ways, big data is regrettably and dubiously named because it is about much more than just the volume of information.

We all know that data VOLUME is increasing:

• McKinsey cites that almost all (15 of 17) industry sectors in the US already have more data PER COMPANY than the entire US Library of Congress
• And, IDC recently completed a study that concluded information in the enterprise will grow by more than 50X in the next 10 years (before 2021)

But, smart big data is about:

VARIETY: Harnessing the power of all your information (poly-morphic variety), whether it takes the form of tabular data (databases), hierarchical data, documents, emails, video, audio, social media, or even data streams from meters or automated process controls

• VELOCITY: Meaning not just how fast data is produced, but also how fast it is absorbed and processed to meet demand
• COMPLEXITY: Regardless of different standards, domain rules, and even storage formats

And, making that information available, in a contextually relevant way to the right user on the right device.

These four big data dimensions (Volume, Variety, Velocity and Complexity) are critical not just by themselves, because they can interact to complicate things:

• Urgent demand (VELOCTY) for information puts a strain on data validation and rationalization across data of different formats (wide VARITEY and COMPLEXITY)
• Making more data available to more people and on different technologies exacerbates the challenge of unauthorized access
• Linking external data to internal data sources (VARIETY) increases the difficulty of maintaining a consistent ontology
• Adding meta-tags to illuminate the context of data can vastly increase the size (VOLUME) of the data and complicate technology issues

The point is … Focusing on volume only results in architectures which scale, but only in mechanical directions – not business directions.  It’s the combination of the volume, velocity, variety, and complexity of information assets that must be addressed in a unified, modern information architecture.

The Big Data Opportunity

Let me start by making an assertion that Big Data is not hype!

Whether you listen to McKinsey, who reports that, “We have identified five broadly applicable ways to leverage big data that offer transformational potential to create value and have implications for how organizations will have to be designed, organized, and managed.”

Or Gartner, who says, “big data is a hugely disruptive force that will enable new insights currently not achievable by other means…”

We are on the cusp of a tremendous wave of innovation, productivity, and growth … all driven by big data.  Its real and its here now!  And, those companies that harness its power will be far ahead of their peer group.

The Impact Is Significant

The promise is tremendous, but the impacts to both business and IT are significant.  In summary,

Big data can enable a lot, but business must adapt its approach and governance models to:

• Take advantage of more real-time, agile decision making
• Leverage deeper understandings of customer preferences and behaviors
• Gain greater fidelity in risk assessments and compliance enforcement

IT will need to rethink roles, tools, and architectures to:

• Offer new levels of data scientist support to the business — turning available data into productive information
• Enable an agile data platform that evolves quickly with the business
• Tailor the user experience so that relevant insights and recommendations are powerfully delivered to any device, any where

But, it is so worth doing, because the CIO (and the whole IT organization) can tangibly demonstrate how they are adding real business value — becoming THE business enabler in a very real way.

“Through 2015, organizations integrating high value, diverse new information sources and types into a coherent information management infrastructure will outperform industry peers financially by more than 20%”  Gartner July 2011 “The New Value Integrator,” Insights from the Global Chief Financial Officer Study

From Telco’s, to Insurance companies, to Media, to Healthcare, everyone is now moving in this direction – and fast!

IT’s Role In The Brave New World

So what is IT’s role in this new area one might ask?

My perspective is that big data presents the single biggest opportunity for IT to truly enable the business with new capabilities that translate directly to increased revenues, lower costs, or reduced risk. 

Because the opportunity is so relevant and meaningful to the business, and because the pre-requisites are so technology-focused, Gartner has urged businesses not to wait until they are forced to play catch-up.  Specifically, their mandates include:

• CIOs and enterprise architects should start to address the volume issues of big data today as an initial step toward meeting the broader challenges of “extreme information management” within three to five years
• CIOs are recommended to take a more entrepreneurial role in the business, potentially initiating new business ideas and actively driving new revenue streams, rather than relegating their IT organizations to a largely reactive role as a supporting function
• CEOs and business leaders should explore and embrace the new data-driven business opportunities that arise from big data, re-examine existing decision-making processes, and invest in new skills and technology capabilities to master the challenges it introduces

In my next blog, I will offer a perspective on:

• Just how much has changed in the BI/DW landscape in the last few years, and,
• Suggest some dimensions in which how IT should think about big data for the purposes of planning

William Schmarzo and I collaborated on this blog and we would really like to hear your opinions.  Please send in your thoughts, as we welcome them and are looking for a dialog, not a monologue…

As a result, businesses are aggressively launching new products, pursuing mergers and acquisitions or divestitures, and mining their corporate assets to find new ways to evolve, differentiate, and grow.  The pace of change and importance of IT to today’s business is, in turn, forcing IT to reinvent itself to both improve responsiveness and reduce costs.  We have seen huge momentum in areas such as Cloud Computing – which, in a few short years, has literally moved from an interesting concept to a widely embraced approach for scale, agility, and cost effectiveness.

All of this is good, but it is not enough.  And, I don’t think it is an overstatement, at all, to suggest that we are in the midst of one of the most significant and radical changes in how information is leveraged for business advantage that has ever occurred.  We are past the inflection point à where vast amounts of new information exist along with the tools, processes, and approaches to harness it for business improvement.

In the rear view mirror is Business Intelligence – using Data Warehouses where we took months of iterations to develop a specific drill-down report that, by the time it was developed and placed in production, was already out of date.   In the last 12-18 months there has been a radical change to forward-looking analytics where you really do not have a defined answer (in many cases not even a clue) of the trends you are looking for – you are now interested in: Statistical Analysis (“Why is this happening”), Forecasting (“What if these trends continue”), What will happen next (“Predictive Analytics”) and the holy grail or Business Process Optimization (“What’s the best thing that could happen”).

The whole game has changed from reviewing past statistics, to real-time analysis of data to find trends that if capitalized on, can rocket your business ahead of the competition. In Blog post #1, I note that a recent CIO survey notes that the game is changing to real-time and that means a whole new thinking and a whole new foundational approach.

Everyone is using the term Big Data including:  McKinsey, Gartner, Forrester, IDC, and others are writing about it.  But more importantly, leading businesses are already applying it – with impressive results.

The Brave New World

While this author does think “It’s a brave new world”, I fully acknowledge that many of the questions businesses must answer are not that new.  At a base level, companies have been asking many questions for years …

• Who are my most valuable customers?
• What are my most important products?
• Which are the most successful campaigns?

The point is this … because of all the new insights that can now be derived from data sources, tools and technologies that didn’t exist before.  The answers may become clearer, and may even change.

• Maybe your most valuable customer is not the most profitable customer, but the one that has the largest net influencer effect on a large community of customers
• Maybe you most important product is not the most profitable one, but the door-opener or leader that gets customers in the store

Those nebulous and hard to define words like valuable, important, and successful allow businesses to move beyond pure financial measures to consider the entirety of the contributions that those customer, products, and campaigns make to the business.

Consider that data sources, external to your enterprise now abound and are providing insight to your competitors.  What about data sources you did not consider like:

• Purchasing Habits & Trends
  • NPD Group
  • Neilson
• Government
  • Census / Population Data
  • Household income & statistics
• Ticker / Corporate Financials
  • Bloomberg
  • NYSE
• Personal Financials / Credit Scores
  • Experian
  • TransUnion
  • Equifax
• Other Intelligent Data Sources
  • Radian 6
  • Biz360

  

Big Data Requires A Holistic View

The problem is that, until now, most companies have been trying to answer these questions by looking in the rear-view mirror with a fractional subset of the information available across the enterprise.

We have been making decisions using:

• Only the 10% of enterprise data that is contained within the enterprise data warehouse
• Relying upon historical information only, without the benefit of projections
• With data that is often old and not easily relatable among different LOB systems
• Traditional methods that must pre-suppose the questions being asked and taking years to build and deploy

As a result, business leaders are limited – both:

• Limited in the questions they can answer, and
• Limited in the answers they can get.

As a result, too many rely on instinct and gut feel.  The promise of big data done right is new insights, delivered faster, using most if not all of the relevant, available data with forward-looking projections and recommendations.

But it requires a different, more holistic approach to your wealth of information sources.

The data quality and data governance challenge is how to ensure that the business-critical decisions being made, can be made with high confidence to the point where you can move from data governance to the realm of decision governance (i.e., all decisions are made with the best available information; we are acting on what and when our information says we should, etc.)

As your organization evolves to exploit big data to enable the business, there is a clear tie between the quality of decisions and the quality of the information used to derive the insight or formulate the conclusion.  As such, organizations must:

• Understand their data quality issues as a feasibility watermark on any potential big data solution
• Address them to build a trusted view of the business and enable high confidence decision making
• Embrace governance (i.e., make organizational and process adjustments) to ensure that the quality of that data does not degrade overtime

With good data quality and solid data governance practices in place, the mature organizations can even evolve to a level of decision governance. Imagine policies and procedures to measure the organization’s effectiveness of executing data-based decision-making (i.e., all decisions are made with the best available information; we are acting on what and when our information says we should, etc.).

For example, Life Sciences companies are being forced by the Obamacare Program to look beyond data governance to advance the concept of decision governance, where the organization can measure how often it’s stakeholders use analytics as the basis for their decisions, and the effectiveness of those decisions.

I was fortunate to join a number of industry thought leaders as a panelist on the “Big Data” discussion at Digital East. The panel was made up of a mix of business delivery and technologist and a lively debate ensued about the topic.

The panel was lead by Caribou Honig (Partner, QED Investors) who challenged the team with controversial assertions and asked for examples of success stories as well as failures. Some of the controversial assertions were:

• Is data gold or junk – most panelists agreed that you had to parse through tons of “dirt” to find a “nugget” of information
• Is data over-rated – there were some differences here on the panel with some saying it was (surprising)
• Are the large players at a disadvantage with respect to startups – there was disagreement here as I proposed that large players (with more data) would have more to gain and could monetize their information assets with others on the panel sharply disagreeing saying that politics would prevent usage

The panel closed with key points on:

• What was the #1 mistake companies are making – not looking at raw data, not understanding business value
• What are the key takeaways – don’t use OLTP databases, look at MPP and consider Moore’s Law



Looking Ahead: CTAM Summit

I will be speaking this week at the CTAM Summit in NYC on Thursday, October 5^th 2011 at the Marriott Marquis from 1:30PM – 2PM on the topic of “Making Big Data Actionable”.

The abstract for this presentation includes:  “In this presentation, attendees will learn how to understand and make analytics actionable whether it is Set Top Box analytics, Advertising analytics or Video on Demand analytics.  In today’s business climate, every leading organization finds itself in the data business. With each set-top box click, customer service call, or transaction by a user, or any other viewing activity, data is generated that can contribute to the aggregate knowledge of the consumer and therefore the business. From this data, insights that can help the business better understand its customers, determine services and subscriptions that are high-value, detect problems, improve its operations, reduce risks, or otherwise generate value for the business.

When you have “big” data, there are a number of considerations to making analytics actionable including:

• Data source rationalization 

               -On Demand Programming

              -Channel Viewing Data

              -Interactive Services (Shopping – Home, Auto, etc., Information, Web, etc.)

              -Social Media (Twitter, Facebook, etc.) 

• Metadata and ontology’s used to support the analytics
• Technologies and tools”



Big Data Effects Everyone

One of the key points is that all organizations today, being in the information age, must consider that they are in the “big data” business – whether they want to be or not. For it is in the understanding of the micro-trends (shoppers buying habits, who are my most valuable customers, what are my most important products, what are my most successful campaigns, etc.) that lead us to understand, and the therefore improve our business.

Big data is allowing for the evolution to “Continuous Intelligence”:

• Industry moving from historical and near-time reporting to real-time analytics that enable real-time actions and decisions
• Ability to predict future trends using large volumes of historical and real-time data gaining traction with customers
• Challenging economic times are driving service providers to focus on cost-cutting programs (e.g. improved operations, avoiding violations)
• Retaining customers is critical as new competition enters market and price becomes second to customer experience
• Avoidance of billing mistakes and fraud by accessing unified and complete records of individuals and organizations
• Improving access to real-time, actionable knowledge is a growing trend in the service assurance and sales areas



Big Data Is Reality

Big data is attracting a lot of attention lately as evidenced by the Google trending on the phrases such as “NoSQL”.

Even industry critics are jumping on the bandwagon, for example Gartner© has released a number of reports including these two “’Big Data’ Is Only the Beginning of Extreme Information Management” and “Does the 21st-Century “Big Data” Warehouse Mean the End of the Enterprise Data Warehouse?”.Even in report that was not directed at IT types, the following quote was pulled which directly speaks to the issue:

“Through 2015, organizations integrating high value, diverse new information sources and types into a coherent information management infrastructure will outperform industry peers financially by more than 20%” Source:  Gartner July 2011 “The New Value Integrator,” Insights from the Global Chief Financial Offers Study.

A Forrester report entitled: “Big Opportunities In Big Data, Positioning Your Firm To Capitalize In A Sea Of Information” notes that “Opportunities to improve the bottom line exist in a flood of information; however, gaining insight from data becomes challenging as it grows extremely large… Big data processing raises key questions for both business and IT — address them now and be prepared to capitalize as opportunities emerge”

Yes, it seems that big data is here to stay and being a front runner will pay off dividends to those who invest in actionable intelligence.

Wrapping Up

The growth of data sources, types and standards for interchange is leading to the big data challenge and creating critics statements like “Big data poses a major opportunity for CIOs to drive added value for the business, by deriving insights and identifying patterns from the huge amounts of data available” “ and “Organizations must consider the integration of diverse new information sources and types (structured and un-structured) into a coherent information management infrastructure” Gartner IBID  – could not have said it better myself.

As an introduction, a number of industry thought leaders are going to be debating issues surrounding big data at Digital East (www.digitaleast.com) on a “Big Data Panel” and yours truly will be one of the panelists. It looks like it will be a spirited discussion on Sept. 28 at 4:30PM in Tysons Corner, Virginia.

Digital East Conference

Digital East’s website notes “Join hundreds of Digital executives, senior marketers, entrepreneurs, web strategists, bloggers, and investors at the Second Annual Digital East for expert content on opportunities and trends created by the latest in web innovation. Hear from dozens of industry thought leaders and innovators on topics such as Social Media, Mobile, Cloud, Analytics, Big Data, Search, Online Advertising, Ecommerce, Email and much more!”  Now that’s an attractive bill-board and I sure hope the conference lives up to it’s potential.

Conferences on Big Data

As more conferences appear to address this new area of information management (NoSQL 2011, Digital East, etc.), more opinions are coming to light that traditional solutions to these problems just don’t work.

As I noted in an earlier blog, the Bi/DW solutions of the past just were not built to ingest huge volumes of data (not to mention rapidly) and then feed it back expeditiously, often in real-time, providing analytical insights that here to fore were frankly impossible previously.

It will be interesting to be with a group of professionals in this space to debate the issues around big data.

I hope that you can get a chance to attend…