Reading the Annual Report

I’m always surprised by how few people take the time to read their company’s annual report (you are excused if you work for a government agency).  In particular, the “President’s Letter to the Shareholders” is a gold mine.  It is within this section of the annual report that the President or CEO talks about all the great things he/she did for the company over the past year.  That usually takes up about 3/4^th of the letter and can largely be ignored (IMHO).  It’s the last 1/4^th of the letter that is most informative.  Usually in the last part of the Shareholders letter, the President/CEO talks about the key business initiatives for the next year.  Let’s review a couple annual reports to see what I mean.

Reading the Annual Report:  Financial Services

Below is the 2010 annual report from a financial services firm (by the way, there is no rhyme or reason behind the annual reports that I have picked.  They are just ones that have potential business impact using big data).

Figure 1: 2010 Annual Report

The above section of the Shareholder’s Letter highlights a business initiative to improve customer cross-sell effectiveness in order to reach a goal of 10.0 products per banking household (from 5.7 products per banking household today).  While 10.0 may be a BHAG (Big, Hairy, Audacious Goal), it is clear that some executive in the organization (likely in Marketing) has been chartered with increasing cross-sell effectiveness.

Here are some examples of how big data could help their cross-sell effectiveness business initiative:

• Use detailed customer financial data on the number and types of accounts held by household, combined with key account information (e.g., length of account ownership, account balance, account balance trends) as well as household demographics data, to create more granular household segments.  Run analytic models to “score” these new household segments by their likelihood of buying a specific additional financial product (e.g., households who hold these products and are in this demographic group have an X% likelihood of buying this additional product).  Develop different models for different combinations of products and household demographics.
• Use social media data from sites such as Facebook, Pinterest, and Twitter to identify trends in financial products that might be candidates for cross-product promotions (e.g., mortgage refinancing is hot, so look to bundle a mortgage refinancing with a second mortgage).  Run these trends against your household/product cross-sell models to identify direct marketing targets.

Reading the Annual Report:  Retail

Here’s another example from a retailer’s 2011 Annual Report.  There are at least two sections of the annual report where the company could leverage structured data (e.g., point-of-sale, inventory, returns and orders transactions) with unstructured (social media, web log, consumer comments) data to drive these key business initiatives.

Figure 2: Store Assortment Optimization

The first example highlights to an opportunity where big data could help drive store assortment optimization.  In particular, big data could help in the following ways:

• Integrate social media data, third-party demographics data, and product sales data to forecast optimal store assortment (at individual store level); update optimal store assortments more frequently (weekly?) based upon local events (e.g., Cinco de Mayo, San Francisco Giants home games).
• Integrate social media insights with consumer comments (call center, email, web site) with store and product sales data to calculate and track the net promoter scores and consumer sentiment for particular stores (by product category, by season); identify and act on under-performing stores, products and product categories.
• Leverage social media data to identify product and market trends (by store and by product category) that can impact pricing, in-store merchandising and store assortment planning.

Figure 3:  Private Label Effectiveness

The second example highlights driving private label effectiveness (e.g., increase private label sales from 15% of products sold to 30% over the next several years).  Here are some examples of where big data could be used to drive private label effectiveness:

• Integrate sales and inventory data with social media data to “score” product categories that are the most likely opportunities (highest probabilities of success) to introduce private label products (create scores by store, market, and product category).
• Mine social media data to identify consumers’ areas of interest that can be used for direct marketing and in-store promotions and merchandising around promoting private label products.

Reading the Annual Report:  Brokerage Firm

This third example comes from a brokerage firm’s 2010 annual report.

Figure 4:  2010 Brokerage Firm Annual Report

The annual report highlights the focus on driving their Client Promoter Score (CPS) program.  In 2010, the firm was able to achieve their highest score to-date, a record 37 percent (it would be nice to know if they had a goal and a timeline for their CPS, but maybe that’s something that can be determined via interviews).  Here are some examples where big data can be used to drive the CPS program.

• Leverage social media sites and blogs to create and continuously refine a new Client Promoter Score that enables the creation of a more timely, more comprehensive Client Promoter Score that is a better predictor of clients’ feelings and perspectives (e.g., likeliness to recommend).
• Build analytic models that incorporate social data and different financial transactions to break out and track CPS by most “valuable” customer segments; match up customer financial transaction patterns with sentiment analysis to flag potential CPS score drops by customer segment.
• Use the CPS to segment key customers; leverage Twitter and Facebook data to monitor sentiment trends across your most valuable customer segments in order to more thoroughly identify and quantify potential customer attrition and their corresponding drivers.

Summary

We talk about the business transformational power of big data; the power to tease out new customer, product, and market insights that we can use to drive higher-fidelity, more frequent business decisions.  But in order to drive business transformation, we need to “begin with an end in mind” (to steal from Stephen Covey).  We need to invest the time to understand our organization’s key business initiatives, and contemplate the “realm of the possible” with respect to the big data business drivers (e.g., more detailed structured data, new unstructured data sources, high-velocity/low-latency data access, and predictive analytics).  There is no better place to start your big data journey than by targeting the key business initiatives that can be found in your company’s annual report.

“Stump the Bloggers” with your questions on Capitalizing on Cloud Services and Transitioning to ITaaS.  Post your questions for InFocus bloggers here – and they will be answered live from EMC World May 21 (simulcast on the EMC Community Network).

This week, Wayne Pauley, Choong Keng Leung, and JP Morgenthal share their perspectives on IT Transformation and ITaaS and PaaS:

• ITaaS Assessment:  effective top down assessment process. 
• Transforming Your IT infrastructure: from Silos to consolidated infrastructures through the evolution from shared to Cloud and IT-as-a Service
• Key to Private Cloud is Removing IT Stratification: moving to an agile organization through the use of DevOps culture and other collaborative techniques
•  IaaS and PaaS: distinction between the two is no longer required

Congratulations to EMC for being recognized this week at the Technology Services World conference, the premier event in the IT Services industry. EMC was acknowledgement for commitment to service excellence in three categories:

• Innovation in Professional Services
• Professional Services Excellence in Enabling Customer Success (co-winner)
• Excellence in Use of Metrics and Business Intelligence

Frank Coleman discusses this award and the increased focus on Big Data and Business Intelligence.

Business Intelligence and Customer Data Quality:

• Bill Schmarzo explains how Hadoop, as your ODS and data staging area, can deliver new compelling business benefits: Understanding the Role Of Hadoop In Your BI Environment.
• Laddie Suk explores several core customer data issues that persist in today’s IT systems and covers Customer Data Quality- How to Measure it, Improve it and Make it Stick.

Strategic Planning

“Strategic planning is not something that can be fully programmed into a computer,” offers Dave Bagatelle in Financial Planning: Static vs. Probabilistic, contrasting straight-line assumptions and forecasting to more probabilistic advice and tools that support forward-looking advice.

The concept of IaaS as a standalone layer made sense at one time; especially in the early days of understanding cloud computing. Before we could move forward, we had to develop the methodologies, processes, technologies and capabilities for aggregating infrastructure resources and delivering them in an agile and elastic manner with the ability to measure and respond to demands. As the IaaS layer matured, it was natural that others would start to focus on the problem domain for delivery of applications that make use of the capabilities now afforded them. Hence, PaaS emerged that enabled rapid and agile delivery of large-scale distributed applications. Consider that prior to the availability of IaaS, large-scale distributed applications needed to be designed in a monolithic manner with the infrastructure requirements as a component of the overall solution. Thus, building and deploying applications that supported billions of users or petabytes of data were highly custom, required proprietary hardware and required specialized skills to build, deploy and operate.

However, the time for distinction between IaaS and PaaS is no longer required. Indeed, we’ve learned two key lessons through maturing these two environments: 1) how scale and design of the infrastructure impacts the ability to deliver highly-scalable applications through a PaaS operating on that infrastructure; and 2) how to manage the needs of multiple tenants on a single cloud that all have varying infrastructure requirements. The impact of these two lessons is that IaaS cannot live in isolation, but needs intimate knowledge (context) of the PaaS environments that will be operating on that infrastructure and must operate in support of those PaaS environments. Thus, the IaaS is not a statically defined entity, but must be agile and change over time.

Moreover, it has been realized that components that have been redundantly defined within the PaaS layer is actually an element of the infrastructure. By removing this redundancy and delivering them as part of the infrastructure, it reduces the costs and overhead associated with building and delivering applications in the enterprise. Examples of these components include data management, security, queuing, and metering. Additionally, delivering these as part of the infrastructure simplifies the ability to support multiple PaaS tenants in a consistent and supportable manner.

The collapse of these layers begin to form a new layer identified as infrastructure services.

The good news here is that this historic pattern has worked successfully in the past. At one time developing high performance graphic applications for Microsoft Windows was very difficult. It required the application developers to deliver specialized drivers along with the application. Then Microsoft provided developers with the DirectX interface. The results was simpler and faster development for application developers, greater reliability of these applications since code was not being injected into the operating system layer that conflicted and better sharing of machine resources since each application didn’t use memory and disk to overcome this problem. These are the same benefits for PaaS when we abstract these services within the infrastructure.

The difficulty moving forward will be to bring this understanding forward given the massive instruction to the IT community. The National Institutes of Standards and Technology (NIST) definition has been broadly adopted and is used as a common nomenclature for discussion and communication regarding cloud computing architecture and efforts. However, given the pace of maturity and change surrounding cloud computing, we should not be reticent to let go of early definitions and design goals in favor of adopting emerging methods for communicating design and requirements for cloud computing. Current definitions of IaaS represent compute, storage and networking, and delegate software services to PaaS and SaaS. As we move forward, the lines between these layers will continue to blur even further. Infrastructure services captures the goals and essence of the larger goal of ITaaS and services delivery much more effectively without implying association to the implementation.

Today’s blog explores some of the key customer data issues that persist in today’s IT systems… a sort of “Back to the Future.” The same techniques used 20 years ago are applicable – aided by the fact that commercial packages are available to assist in reporting and reconciliation.

I’ll approach this topic from the perspective of planning to implement a new customer relationship management (CRM) system and address the key topics, tools and techniques to prepare legacy data to be migrated to the new system.

WHAT IS DATA CLEANSING?

Data Cleansing is an organized approach to successfully migrate business data focused on achieving data integrity. Its benefits include increasing data quality. Focusing on data quality is intended to ensure that the new CRM system has clean and valid data. Though data quality at a company may be thought to be good, a formal approach should be used to address data integrity issues prior to migration into the CRM solution. Best practices include:

• Source to target mapping of existing data for CRM is usually performed for the design phase
• Prior to and during this data mapping exercise, data in your legacy repositories should be analyzed to identify potential occurrence of the following types of major data integrity issues:

1. Orphaned/Missing Data – System and/or program errors, inadequate enforcement of Relational Integrity, data that may have been considered optional in legacy systems maybe mandatory for the new CRM.
2. Invalid or Unclean data – Data entry errors, data that may have been entered in to a system prior to implementation of validation logic. Examples of these would be invalid addresses or potentially missing data.

Key Success Factor:  A cleansing effort should be undertaken to get the data-to-be-converted as clean as possible prior to extraction from legacy repositories.

How to cleanse? This depends on the specific problems:

Orphaned / Missing data

• Custom programs responsible for extracting and transforming the data in to XML (for target application load), would identify these instances
• Additionally, conversion programs will generate error reports that would allow you to verify issues were resolved at the source

Invalid or Unclean data

• A variety of COTS tools that specialize in cleansing CRM related, custom programs and/or manual  intervention are generally used to resolved this type of data integrity problems
•  Other common integrity issues include: Incorrect , Obsolete, and Duplicate data

Typical Data Cleansing Activities to Prepare for Conversion:

• Data Discovery – Identification of all data elements within the enterprise data stores
• Data Profiling – Null & valid-values Identification, outlier analysis & other advanced profiling analyses
• Data Quality – Creating metrics which future data management policies will use to demonstrate improvement.  I have often said ‘you get what you measure’… and this is true for data cleansing activities.  The good news is that today’s tools provide various metrics that can be used for tracking and improvement
• Data Matching – Determination of matching values within columns of differing types, formats and naming convent
• Data Cleansing – Replacement of invalid values, deletion of nulls and bad records

Data Collection and Preparation for Loading:

• Data preparation activities are conducted to reduce the time and effort spent converting, and to ensure that the data that is transferred to the new CRM is current and meaningful to the business
• Data preparation requires assistance from various business organizations to identify and confirm the validity of any data to be migrated, and in approving data items (i.e. products) to be removed (or mapped differently)
• Data preparation will also involve resolving any data conflicts.

Data Auditing is required to perform the following tasks:

• Exception Testing, including unwanted data, duplicate data, or conflicting data
• Exception handling based upon the results from exception testing
• Field map validation
• Data validation reports
• Data comparison reports

So what are the systems needed to support your CRM conversion?  The following architecture provides the ‘best practice’ view of solutions for complex conversions:

In the architecture above, the best practices include:

1. Using an Enterprise Application Interface (EAI) bus to develop interfaces to extract data from legacy applications. Once these application interfaces are developed once, they can be used repeatedly. This is particularly important if your legacy system may run in parallel with your new CRM system for an extended period of time.
2. Implementing a Master Data Management Model that clearly identifies systems that are primary owners of data elements as well as other systems that use the data.
3. Using Extract, Transform and Load tools (ETL) to extract legacy data, store in a temporary data base where data cleansing is performed.
4. After cleansing, legacy data is reloaded into legacy systems in order to improve the quality of source data.
5. Steps 3 and 4 can be repeated multiple times, with quality metric reports (produced by tools) used to assess and validate that quality gates have been met.
6. The same ETL tools used for cleansing data can be used in the ‘final’ extract, transform and load into the new CRM target system.

A word about tools (okay, actually several words):

Todays’ Information Data Quality tools can provide business-focused metrics that are simple and easy to disseminate to your company. For example, customer address quality has historically been a problem area. The root cause of problems is usually systems with “free form fields” where customer service representatives enter address without validation. The impact on your business is huge:  mailed invoices delayed, lost, or returned—resulting in lower cash flow.

Sample Tool Output

Below are two standard reports from a commercial tool that analyzed street addresses for 3.5 million customer records.

In this case, about 8% of the street addresses were in categories 0, 1 and 2 – which indicated that 8% of this company’s customer street address information was of poor quality.

The tool can be run again after performing a number of analyses and algorithms – including street address matching against a commercial street address database (usually accessed from government postal databases or commercial delivery service sources).

Use these tools as communications vehicles to every part of your organization… to indicate you have coupled a quality improvement strategy with a few metrics that define success.  For example, you could target reducing the poor quality street addresses to below 3%.

Finally, no Data Quality Improvement should be begun without a Governance program.  Where does your company place on the industry-accepted Data Governance Maturity Model described below?

At the lowest level of maturity, your IT team (with or without business involvement) has developed a “tactical” approach whereby the strategy and framework are defined.

In the next level of “operational” maturity, an organization (IT plus Business representatives) successfully implements scenarios & validation of the framework.

Finally, the “strategic” and highest level of maturity indicates your company focuses on maintaining original goals achieved and fosters a corporate culture to retain implemented best-practices.

Next Up:  Unlocking the Power of Big Data to Become a Next-generation Service Provider

I hope you will join me and will pass on the link to your friends and networks.  Please … subscribe, send me feedback, and check back next week for the next installment.   If nothing else, I promise the International Travel tips will be extremely useful!

Today’s International Travel Tip:   Must-Have Technology Packing Guide (Also Known as ‘Packing for a Potential Natural Disaster’)

I will not insult your intelligence about the basics, but rather provide some helpful hints that provide for your personal safety as well as business continuity.  They are the result of my experience in surviving the Chilean mega-earthquake in February, 2010 (see one of my prior blogs for more info).

1. Spare batteries for your cell phone and computer  (and keep them charged)
2. Spare power cord to charge your wireless phone
3. Small Flashlight, spare batteries
4. Did I mention spare batteries?
5. Laptop with Microsoft Communicator with voice calling
6. Laptop with Skype for voice and/or video calling (due to some company’s corporate policies that forbid Skype on corporate laptops, some travelers carry two devices – a corporate laptop and a device with Skype)
7. Copies (hard copies as well as scanned) of your passport
8. Emergency travel contacts.  Most companies have contracts with ‘travel assist’ vendors – not just travel agents, but companies that will arrange for quick extractions from countries if circumstances warrant.
9. Emergency medical assistance contacts.  Beyond the company-provided medical contacts, always obtain names and phones of at least 2 local doctors for fast emergency assistance.

One final note:  while I won’t get into the merits of an iPhone vs. a Blackberry, I will point out one key difference (that is not widely advertised) that can make your life miserable in some countries.   Blackberry smartphones have always had the option to let you manually select networks in foreign countries.  When you select ‘manual networks’ – a list of available networks is displayed and you can select the appropriate network you prefer.  This is extremely useful as you (like me) typically have a good understanding of the best local carriers in the countries where we travel.

On the other hand, an iPhone does NOT have this capability – it will automatically select a carrier in a country.  This limitation has ramifications for countries such as Brazil and Chile – where the iPhone selects a carrier for good voice capabilities – yet that carrier’s data capabilities may be (based on experience) poor or ineffective.  One of my team member spent hours on the phone with Verizon – and was told there was no way around this iPhone limitation.  Buyer beware!

Financial Planning is a growing need, especially for today’s retiring or retired baby boomers that require planning to sustain their lifestyle and for the Gen X/Y generations who are facing high education costs and less certain retirement prospects.  Planning tools are playing an increasing role in attracting and retaining these clients.  The right tools convey to clients that their adviser understands clients’ ever-changing situation, allow them to provide sound, valuable advice, and bring a level of trust and continuity that help advisers maintain relationships.

More than ever, clients want to know (and be able to track) if what their adviser said in one year holds true in the next several.  This requires the industry to move away from straight-line assumptions and forecasting to more probabilistic advice and tools that support forward-looking advice.

Probabilistic forecasting (aka the Monte Carlo method) provides two main benefits:

• Risk/Liability reduction to the firm for forecasts that do not come true
• Improved advice to clients (ex. A gifting strategy that makes sense if a portfolio grows might need to be reviewed and revised during times of market volatility; therefore, creating a strategy that factors in that volatility from the start would obviate the need to revise the strategy)

Static advice and reporting is no longer acceptable to most clients.  As account values change, clients demand to know the impacts to their goals in future years and what realistic changes are needed to stay on track.  Clients are also more skeptical than ever regarding straight line, non-volatile assumptions that do not account for real world events.  Increasingly, advisers are being required to provide advice that takes into account volatile investment types and markets.

Using straight line assumptions (e.g., US Large Cap stocks will go up 10% each year with no volatility) does not provide a realistic example of the investment world and it will almost undoubtedly provide an overly optimistic outcome as the only way an account value goes down is if spending is higher than the combined income and growth of the portfolio.

Advisers without the ability to respond to client requests for probabilistic financial planning will have a harder time retaining them since static forecasts become less reliable in a volatile environment.

Financial Planning tools that incorporate integrated data and a probability of success for individual goals allow advisers to provide strategic planning advice.  Advisers can track the probability of success when the plan is established as well as automatically on an on-going basis as actual income is realized, actual expenses are paid, and actual investments increase or decrease in value.

While assumptions still need to be made regarding the expected growth of investments, the volatility of those returns and their correlation to each other, the models will output potential long periods of negative returns in the short/mid and long term and create a framework for discussion and consideration.

Monte Carlo models rely on reasonable assumptions regarding the market return, asset class volatility and correlation and not all models are considered equal.  The most accurate models will consider mean reversion, the non-normal distribution of returns (e.g., fat-tails) and the ability to stress test a strategy using actual ‘black-swan’ (extraordinary) events.  A detailed description of these concepts and their application to financial planning models is beyond the scope of this blog.  The point is that some Monte Carlo is better than none, but actual tools and assumptions are important factors.

Advanced tools allow for the prioritization of goals and alert clients when/if material changes impact their plans.  Alerts can be set, among other things, to track the value of accounts, spending vs. budgets and allocations to specific securities or asset classes.

Best-in-breed financial planning tools allow advisers and clients to see how these occurrences impact the initial retirement goals and whether revisions are necessary.  The client can ‘play’ with several options using ‘what-if’ scenarios (ex. save more, assume to spend less in retirement, work longer).

Strategic planning is not something that can be fully programmed into a computer.  An adviser must understand the client’s situation and goals and come up with creative strategies that meet their individual needs.  Software helps this process, but one must never forget or discount the importance of human factors in the financial planning process.

This is the third year in a row that EMC has won this award. EMC’s commitment to improving our customers’ experience is what this is all about. Our core metrics are designed to look at the service experience from the customer point of view, not ours.  This has been invaluable in helping us focus our efforts on areas of the business that will have the greatest customer impact.

Our approach is a thought process, not an end state. Because our business is dynamic, we will continue to evolve with our customers and not lose sight of their experience. These are exciting times with the increased focus on Big Data and Business Intelligence; EMC is the place to be!

Figure 1: Infrastructure Transformation

This is because Infrastructure is the foundation that everything else ‘rides’ on top of. This is analogous to cruising a Mercedes Benz along a smooth flowing tar-surfaced highway versus the same car on a congested and pot-holed muddy road. You can have the best car (application or IT service) but the driving experience (IT service quality) is completely dependent on the road (Infrastructure) itself. However, this is not necessarily the easiest layer to start with. As you can imagine, making changes to the infrastructure requires significant planning, time, resources and investments.

The approach to transformation of the Infrastructure can take place over three stages. This is discussed at a high level in the following paragraphs.

Stage 1: From Silos to Consolidated Infrastructure

The logical first step to transforming the infrastructure is to consolidate and standardize the infrastructure to reduce complexity and IT costs; and improve efficiency and controls, as shown in Figure 2.

Figure 2: Stage 1 – From Silos to Consolidation

In this stage, there are three key strategies:

• Consolidate data centers: Data center consolidation has been elevated to the top of most CIOs’ agendas as they look to modify their data center facilities due to limits in power, cooling, efficiency and space. At the same time, there is a strong desire to lower facility costs and reduce operational expenditures by eliminating redundant facilities.

• Rationalize infrastructure: Most organizations spend up to around 70% of their budgets on maintaining existing assets. Rationalization strategies help reduce overlap, cut costs and add business value by aligning IT and investments that will contribute to business results.

• Virtualization: Virtualizing your data center is the first step towards achieving the flexibility and cost benefits of cloud computing. To realize the full benefits of virtualization, IT organizations must have an integrated virtual data center strategy that has integrated management of key operational processes and procedures, a flexible virtualized architecture, and support for mission critical applications.

Stage 2: From Consolidation to Shared Infrastructure

After consolidation, the next stage is to optimize and maximize the ROI and utilization of IT assets and resources. This can be accomplished via three methods as shown in Figure 3.

Figure 3: Stage 2 – From Consolidated to shared Infrastructure

• Tier Infrastructure: Defining service levels and tiering the infrastructure according to the service levels provide businesses freedom of choice – to choose and pay for the quality, service level, performance and capacity they require. Virtualization allows IT to create Compute, Storage and Network resource pools which can be aligned to service levels by abstracting resources from the appropriate infrastructure tiers.

• Optimize Resources: Increase resource utilization and efficiencies through modern technologies such as automated storage tiering, capacity management, thin provisioning, deduplication and IO virtualization which help to reduce per unit costs by doing more with less.

• Integrate and Automate Management: Management of a dynamic and fluid infrastructure requires proactive, real-time and “intelligent” management across the multi-lT layers (e.g application, compute, storage, network) and across physical, virtual and legacy platforms (e.g. mainframe, midrange, physical server, virtual server). Management tools with build-in intelligence and analytics helps to cut through the complexity of the next generation infrastructure to reduce time to resolve and improve service availability and performance.

Stage 3: From Shared Infrastructure to IT-as-a-Service

The last stage of the Infrastructure Transformation is to take advantage of cloud computing and evolve towards IT-as-a-Service ((ITaaS). In particular, IT offering Infrastructure-as-a-Service (IaaS). The IaaS service catalog may include services for compute (or server), storage, backup, disaster recovery, data center and network. As shown in figure 4.

Figure 4: Stage 3 – From Shared to Cloud and IT-as-a-Service

• Embrace Cloud Computing: Enterprises need to review existing IT portfolio and sourcing strategies and take advantage of cloud computing where it makes business sense. Cloud computing allows IT Organizations to build and own their own applications and infrastructure; or to procure these in the form of services from Cloud service providers. By understanding your core competencies and businesses, develop and implement a cloud strategy that optimizes the enterprise’s IT portfolio with a strategic-mix of internal and external vendors and service providers.

• Operate An IT-as-a-Service Model: IT organizations cannot effectively operate a cloud environment and deliver IT services if it continues to operate as the traditional infrastructure or technology provider. Using the Fast-Food Restaurant as an analogy:

• A customer orders a meal from a menu with clearly indicated pricing (service catalog)
• Customer may also go to an internet portal for ordering home delivery (self service portal)
• The order is taken and delivered within expected time duration and with consistent quality and service (orchestration, processes and roles and responsibilities)
• The customer is focused on the end-result – the meal of his choice is delivered within expected time duration and quality, he pays only for what he ordered and is served with a smile.

Summary

Hopefully this blog has provided you some high level ideas for transforming your infrastructure. More details can be obtained from the EMC Cloud website here.

JP’s IT Axiom #124: Design flaws at the top of the stack will highlight limitations at the bottom of the stack. Likewise, the design at the bottom of the stack impacts performance at the top of the stack.

There’s no escaping the fact that a poorly-designed application will put undue burden on the operating infrastructure. A “chatty” application impacts bandwidth. Improperly designed database queries will consume memory and disk capacity. Poorly-designed storage architecture will limit the amount of I/O per second (IOPS) and, thusly, limit the speed of retrieval of data to the application. IT transformation is about moving from a stratified organization to an agile organization through the use of DevOps culture and other collaborative techniques.

Read more and comment:  JP’s The Tech Evangelist Blog

The last bullet on this diagram is key: IT must become a service provider.  This is one of the most important parts of the transformation process.  Internal business units have a choice of providers, and all they have to do is pull out their credit card to use some of them.  So if their IT organization becomes a Service Provider and can not only compete with offers from external providers, but can also provide ITaaS as a trusted ‘broker’, it would be a win-win for the business.

Below is a list of some of the most critical areas we believe must be included in the assessment:

At the top are the Enterprise-level assessments that must be completed first, because to be successful, the transformation process must be completed top down.  By assessing the overall organization, the business can evaluate how it is designed from an ownership and accountability (governance) perspective, as well as how it measures success.  The traditional approach of meeting or exceeding deadlines is no longer enough.  Businesses today must be agile and responsive to changes in the market and customer demand, which requires measurement to be shifted to ‘time-to-value’ (TTV).  Purchasing a large software system and spending 3-5 years on implementation is too long a TTV.  By transforming IT to be a broker/provider, IT must make a competitive offer.  The lines of business need a new way of interacting with IT, including accepting a shift to a chargeback model (I’ll cover chargeback more in a future post).

The next section of the slide includes more organizational and governance elements that apply specifically to IT.  EMC’s Global Marketing CTO, Chuck Hollis, posted a couple of great posts along this line (In praise of the business process owner and Good help is here for your IT transformation)  The point being that we have to change the way IT is designed at the technical level (cross-domain expertise) and at the structural level (new functions such as services, sales and marketing, billing help desk, and service provider liaison).

The next section of the graphic covers the GRC and Security assessments.  Does IT have updated information flow diagrams?  Has it developed a taxonomy for what information assets can be placed with a given service provider and under what conditions?  For example, it is fairly common practice to prohibit intellectual property to flow outside the firewall.  But what if IT wants to use SP resources to run a new version of the company’s code, instead of acquiring systems that would be left idle much of the time?  This would require a new governance model, with new security controls, to ensure the IP is adequately protected.

The next section covers the services and Services levels (SLAs).  If IT is going to provide services and be competitive, it needs a self-service portal.  Does the company have a service lifecycle from creation to end-of-life?  Does the company have a service catalog?  A service orchestration engine?  These are just a few of the issues we consider in the ITaaS class that must be part of the assessment process.  SLAs are getting fiercely competitive—can IT provide SLAs that are the right cost with the right metric?  If IT is typically required to deliver Five 9s, but the business only needs and wants to pay for Three 9s, does IT know how to do that?

Next is the applications assessment, which is the most important and most difficult assessment to make.  You’ve heard of Server Huggers but they are nothing compared to App Huggers!  When it ran the applications assessment, EMC’s own IT organization found that it had more than 600 applications, which were eventually reduced to under 400.  The process for using COTS (commercial off the shelf) software also had to be adapted.  Most companies find the notion of changing business processes to be repulsive, so when a COTS package is acquired, the first thing that happens is the stakeholders with the most say start driving for customizations, which is, of course, antithetical to ITaaS.  In order for IT to be competitive with SaaS offerings, there must be an even playing field—the business must choose the provider who can best meet the core objectives of Cost, Functionality, SLA, and Trust as a package deal.  It is important to note, however, that customers often have different weighted values for these dimensions of service based on their business needs.

Last up is the technology assessment.  Some of this is quite simple and it is immediately clear if the necessary pieces are in place to stand up and operate a service-based infrastructure (IaaS), development (PaaS), and application (SaaS) environment.  Key investments start with a “virtualize everything” and “standardize the platforms” basis.  It may also include service catalogs, orchestration engines, federated access and identity, federated directory services, etc.  Some of this may be very hard to implement – for example is your network essentially flat today?  If so, how do you federate with a public provider without exposing unwanted elements to the SP network?  What happens when the SP wants to perform an update to its technology that IT has federated to key applications internally?  Have the systems become agnostic to upgrades or at least do they appear to be agnostic to the lines of business?

So as you can see, the assessment process requires an enormous amount of data gathering.  The architect must wear a lot of hats or at least be fluent enough in several of the new domains to drive the process in the right direction and eventually to the decision-making point.  Once the assessment is complete it is still necessary to evaluate the prioritization and economics of what to do first, and determine the justification and next phases of planning and design.  I will cover those topics soon in a future post.

Stage 1:  Hadoop for Data Staging and ODS

I am seeing several client engagements where the BI/DW team is looking to use Hadoop to simplify, accelerate, and enhance their existing ETL and data staging processes.  Hadoop brings at least two significant advantages to your ETL and data staging processes.  The first is the ability to ingest massive amounts of data as-is. That means that you do not need to pre-define the data schema before loading data into Hadoop. This includes both traditional transactional data (e.g., point-of-sale transactions, call detail records, general ledger transactions, call center transactions), but also unstructured internal data (like consumer comments, doctor’s notes, insurance claims descriptions, and web logs) and external social media data (from social media sites such LinkedIn, Pinterest, Facebook and Twitter).  So regardless of the structure of your incoming data, you can rapidly load it all into Hadoop, as-is, where it then becomes available for your downstream ETL, DW, and analytic processes (see Stage 1 in Figure 1 below).

Figure 1:  Hadoop as ODS/Data Stage

The second advantage that Hadoop brings to your BI/DW architecture occurs once the data is in the Hadoop environment.  Once it’s in your Hadoop ODS, you can leverage the inherently parallel nature of Hadoop to perform your traditional ETL work of cleansing, normalizing, aligning, and creating aggregates for your EDW at massive scale.

Many ETL vendors, like Pentaho, Talend, and Datameer are modifying their products to seamlessly create MapReduce parallel ETL jobs. These vendors provide drag-and-drop UIs for generating ETL MapReduce jobs, removing a great deal of complexity from the data integrator/developer.

Once the raw data is in the Hadoop environment, developers can do data transformations and enrichments that were not easy to do before including:

• Parse complex, unstructured data feeds (like consumer comments, web logs, and twitter feeds) to capture key data and metrics of importance (e.g., visitor id, session id, site id, display ad id, display ad location) that can then be integrated with the existing structured data in your EDW.  For example, imagine the business possibilities if you were able to mine the wealth of social media data on your customers to identify their interests, passions, associations, and affiliations, and then integrate those new customer insights with the existing customer data that is meticulously maintained in your CRM system.
• Create advanced composite metrics that require you to process many days, weeks, or even months of history.  I covered some of these composite metrics in a previous blog, but creating metrics around frequency, recency, and sequencing is possible in a Hadoop environment that is not easily done within your traditional ETL environment.  These composite metrics facilitate more detailed and complex analysis yielding key performance indicators that might be better indicators and/or predictors of performance.

Stage 2:  Hadoop for Feeding Your DW

After doing all of this parsing of unstructured data for new customer and business metrics and data enrichment to create new composite metrics, the output of the Hadoop ODS can then feed your standard EDW.  The Hadoop ODS has the advantage of being able to create structure out of unstructured data, which can then be integrated with your existing structured, transactional data in the EDW (see Stage 2 in Figure 2).

Figure 2: Hadoop For Feeding Your DW

And because of the massive scalability of Hadoop and the ability to ingest massive amounts of data quickly, you can significantly accelerate your traditional ETL processes and more easily fulfill your EDW SLA’s.  Not only can you shrink the latency between when the data transaction or event occurs and when it’s available in your EDW, but you can also provide more granular, detailed data in your EDW, especially if your EDW is based on an MPP architecture (which reduces the need for indices, aggregate tables and materialized views, thereby saving more even more EDW loading time and management effort).

Stage 3:  Hadoop for Feeding Your Analytic Sandbox

The third stage of Hadoop as your ODS affects the self-provisioning and rapid and frequent iterations typically required from your analytics sandbox environment. In this environment, the data scientists can grab what data they need out of the Hadoop ODS without worrying about impacting the EDW environment.  They can select whatever level of granularity they need from whichever data sources they need in order to build, test, refine, and publish their analytic results (see Stage 3 in Figure 3).

Figure 3: Hadoop For Feeding Your Analytic Sandbox

With the Hadoop ODS, the data scientist have the ability to store and access data they “might” need from an analytics perspective; data that may never find its way into the EDW.  For example, the Data Scientists might want to store large amounts of social media or web log data, or large varieties of widely available third-party data (from places like data.gov) to enhance analytic modeling.  The Data Scientists would grab this data out of the Hadoop ODS when and as they needed it, depending upon their current analytic needs.

Business Example of the Power of Hadoop as an ODS

Let’s walk through an example as to how this Hadoop ODS environment can support the types of on-demand analytics that the EDW is so poorly suited to support.

Recently there were floods in Thailand that had major impact on the supply chains of many manufacturers.  In one case, the CFO asked their BI team to come up with an estimate of the impact on the end of quarter Earnings Per Share (EPS).  To make matters worse, the BI team only had 3 weeks to complete this assessment.  This is the perfect type of analytics request for an analytic sandbox because:

• The analytics team is going to need to get access to lots of different data sources rapidly – both detailed structured data as well as some unstructured web log and media feed data.  The data requests could change rapidly (e.g., adding third-party market data, local economic, and weather data) as the analysts proceed through the analysis process.
• Much of the data being used for this analysis will never find its way into the DW, but needs to be housed in an environment that stores the data as-is and can be rapidly accessed and integrated into the analysts processes just in case they need it.
• They need to come up with an estimate – a range of probabilities – and not a precise answer.
• This is likely a one-off request, so it’s not critical to build a repeatable process for this analytic request.

In this example, the analytics team asked for not only their own supply chain data, but they also asked for the supply chain data for their top 25 suppliers (which represented about 90% of their components, and 100% of their Tier 1 components).  They also pulled in customer, inventory, orders, sales pipeline and manufacturing data from their internal systems, as well as the manufacturing bill of materials from the internal manufacturing system.  This data was then augmented with third-party data on weather forecasts, traffic and road conditions, and local support and recovery information.  All of this data could be stored in the Hadoop ODS waiting for the time when the Data Scientist team would or might need that data.

The analytics team was able to use this data to create models that answered the following questions:

• Are their forecasted weather problems or local recovery and traffic issues that will exacerbate the supply chain problem?
• Which of our products are most impacted by the shortfalls (note: this required looking into their suppliers’ supply chains to identify common subcomponent suppliers across their entire supplier network)?
• Which of our top customers are most impacted by the component shortages?
• What are the top sales opportunities in the sales pipeline based upon margin impact?
• How best do we allocate our existing products to our most important customers and deals?
• What is the likely impact on Earnings Per Share with a 95% confidence level?

The result was that the analytics team was able to not only give the CFO the estimated EPS impact (at a 95% confidence level range) but was also able to provide insights into how to optimize existing inventory allocation to minimize the EPS impact.

By the way, once this analysis was done, the analytics team packaged up the analysis and archived it.  While they were unlikely to be doing this type of analysis again anytime soon, they needed to archive the analysis for compliance and possibly legal reasons.

Summary

As you can see, using Hadoop as your ODS and data staging area can deliver new, compelling business benefits (and not to mention that it’s probably significantly cheaper and more agile then your existing ETL environment).  And I expect that we’re going to start seeing more use cases for Hadoop within our BI and DW environments.  The cost benefits, the processing power, the access to lower-latency data, and the overall simpler architecture are just too compelling to ignore.