Retailers and consumer-packaged-goods (CPG) companies have long had access to vast amounts of transaction data: every day, companies capture information about every SKU sold to every customer at every store. In addition, companies regularly use sophisticated market-research techniques to answer a variety of questions: what products should we develop and sell? How much is the customer willing to pay? Which products should we discount and when? Which marketing vehicles will allow us to reach the most customers?
Adding to the reams of data and market research already at their fingertips, consumer companies now have access to social-media information and other large data sets known as big data. In this article, we discuss the potential of big data and advanced analytics for the retail and CPG industries and examine what it takes to turn this potential into actual value.
Immense possibilities
The combination of big data and advanced analytics offers retail and CPG companies countless opportunities across the value chain. In portfolio strategy and product development, for example, companies can get a more detailed understanding of consumer needs and attitudes and more precisely identify consumer segments, improving their ability to target the highest-value opportunities. They can measure the return on investment (ROI) for marketing spend across both traditional and newer marketing vehicles (such as social media), allowing them to shift marketing dollars to the most effective channels (see sidebar, “Unraveling the Web: A new way to understand the online customer”). Through detailed hourly analysis of in-stock rates by store, retailers can reduce out-of-stocks, provide a better shopping experience for consumers, and boost sales for both themselves and their CPG partners.
Unraveling the Web: A new way to understand the online customer
Consumer-facing companies seeking to measure Web-site performance need to understand how well connected their site is to all the sites that influence purchasing decisions for their products, such as blogs and review sites. Simple “clickstream” analyses of how customers use and navigate to and from a company’s Web site neglect a substantial set of other influences that bear on today’s multitasking, site-hopping consumers who check e-mails, update their social network status, and catch up on the news—sometimes taking days to decide what to buy.
For the past several years, we have tapped a panel of about 11,000 online users, broadly representative of the vast US online population. We log all the sites that these users visit during a six-month period. When a disproportionate number of users click on any two sites within a given period, typically a week, we consider the pairs to be connected. We collect all such pairs for companies, information providers, and others in a given industry.
We present these relationships in a network map, which enables us to discern, at a granular level, a given company’s network. The number of nodes and the length of the lines reflect the strength of the network. Short lines mean a strong connection; long lines represent a weak one. Strong connectivity implies significant overlap among their visitors, meaning the influencer is likely generating traffic for the connected site, even when users do not click directly from the influencer site to the connected site.
Retailer A has two strong connections with influencer sites, while its competitors, retailers B and C, have many more.
The exhibit shows a network map of five online consumer-electronics retailers. Visitors to electronics retailers are likely to visit technology review, media, and news sites, among others. Retailer A has two strong influencer sites, while its competitors Retailer B and Retailer C have many more connections with influencer sites. This indicates that A could strengthen its relationships with influencer sites and boost traffic from them by investing in creative content partnerships, social media, and advertising. Companies can use influencer maps as they think about redistributing online advertising dollars to optimize connections with the sites that generate the most traffic. They can also marry this data with demographic information on customers to recognize those influencer sites that reach their desired customer segments.
Big data and advanced analytics allow companies to make better, faster decisions in their day-to-day business and deliver improved performance. A European CPG company, for instance, revolutionized its retailer-specific assortments and planograms: by applying advanced analytics to consumer data, it was able to determine which SKUs were selling well in which retail formats and which SKUs to swap in and out to best meet consumer preferences. It is now seeing 10 percent sales growth in a low-growth category. And the potential impact isn’t just in sales: recent research by McKinsey and Massachusetts Institute of Technology shows that companies that inject big data and analytics into their operations outperform their peers by 5 percent in productivity and 6 percent in profitability.1
Making it happen: Three ingredients
We have found that fully exploiting data and analytics requires three capabilities. First, companies must be able to choose the right data and manage multiple data sources. Second, they need the capability to build advanced models that turn the data into insights. Third and most critical, management must undertake a transformational-change program so that the insights translate into effective action. (For more, see “Putting big data and advanced analytics to work” and “Making data analytics work: Three key challenges.”)
Managing the data
As they embark on a data-and-analytics journey, companies should take a “decision back” approach that begins by crisply answering one question: which decisions do we want to improve? A retailer may, for instance, seek to make better decisions about its promotional spending. Here, the range of decisions can be quite broad: do we want to optimize the design (number of pages, number of products on each page) of our promotional leaflets and circulars? Do we want to reassess the distribution of our circulars—which newspapers they should be inserted into, which addresses they should be delivered to, and so on? Do we want to rethink the product mix in our circulars? Each decision requires different data and analyses.
Once they’ve determined the specific business decisions they want to improve, consumer companies must collect and manage the data needed to conduct insightful analysis. In our experience, this entails three deliberate actions: creatively sourcing data, defining data-governance standards, and establishing an IT infrastructure.
Creatively source data. Often, we find that a consumer company has the data it needs to unlock business improvement, but the data reside in different business groups within the company. In effect, organizational silos “hide” data from the analysts who can put it to work.
For example, a specialty retailer sought to redesign the promotional mailings it sends to the more than five million members of its loyalty-card program. While the program had been successful, revenue and profit from loyal customers was stagnating. The retailer recognized that by combining data from various functional groups—in this case, customer loyalty, marketing, and merchandising finance—it could identify subsegments among its loyal customers and figure out what types of direct mail resulted in the most incremental profit for each segment. Once the retailer understood what data it needed, it developed a focused process to extract and clean the needed data and bring it together for this new purpose. We find that such a targeted approach often leads to success and supports quick wins.
Define data-governance standards. Given the vast amount of data that retail and CPG companies collect and manage, ensuring the accuracy and reliability of data is a constant challenge. For instance, product information—package sizes, product descriptions, or even the category in which a product belongs—isn’t always up-to-date in retailers’ databases. If the data are needed for analysis, the old saying “garbage in, garbage out” still largely applies.
The increasing value of analysis-ready data mandates a mind-set shift: in particular, a shift from simply collecting the most easily available information and storing it as is, to establishing rigorous data-governance standards that call for clean, consistent data to be collected and stored in an analysis-ready state. The approach can be quite similar to the way retailers manage and maintain stores: they have processes and checks in place to ensure that store assets are properly cleaned, well designed, and periodically refurbished to maintain their useful life. In the same way, data-governance standards should define what data will be collected, the processes for checking and maintaining the data, and who in the organization is responsible for managing each data set.
Establish the IT infrastructure. Companies need the IT infrastructure to store, easily access, combine, and analyze large data sets. Setting up the IT infrastructure requires close collaboration among business managers, analysts, and IT staff to ensure that data storage is designed in a way that meets business needs and technical IT requirements.
One large retailer has taken big steps that improve data accessibility. The retailer recently announced plans to integrate its five-plus e-commerce Web sites onto one platform and move the corresponding data into a single in-house data repository. Whereas today sourcing of e-commerce data requires coordination across groups and can take several weeks, the new infrastructure will give the retailer’s business leaders instant transparency and access to e-commerce data, accelerating insight generation and business impact.
Translating data into insights
Managing the data is just the first step. Companies must next make sense of the flood of data—a task that requires sophisticated and sometimes complex analytic models. Two guiding principles can help. First, business users should be involved in the model-building process; they must understand the analytics and ensure that the model yields actionable results. Second, the modeling approach should aim for the least complex model that will deliver the needed insights.
The power of these two steps became evident in a CPG company’s efforts to assess its marketing-spend ROI. An external vendor independently built a model that suggested online paid-search ads delivered incredibly high ROI. The CPG company’s marketing executives were surprised but trusted the results, even though they didn’t understand the model. Later, the company’s internal insights group partnered with the marketing function and built its own simplified model. The results were markedly different: this model showed ROI for paid-search ads to be 15 times lower than what the vendor’s model estimated. Further investigation revealed that one of the external company’s analysts, who had no knowledge of the business, had grouped the data erroneously—a mistake that a business practitioner would have easily caught. Moreover, the complexities of the vendor’s model hid the error from view.
Turning insights into effective frontline action
Gathering the right data and developing transparent models, however, won’t yield impact unless companies can also turn data-driven insights into effective action on the front line. Companies must define new processes in a way that managers and frontline workers can readily understand and adopt. When a large CPG manufacturer implemented a new demand-forecasting system that generated automatic forecasts of each retail customer’s inventory needs, sales managers weren’t told that the new process included shipping of safety stock to every customer. Worried that their customers weren’t receiving adequate stock, sales managers regularly overwrote the automatic forecasts. It took months before they changed their behavior and trusted the new system.
We see great potential for consumer-facing organizations that adopt big data and advanced analytics as a platform for growth. Done right, a data-and-analytics initiative can yield outsize rewards for retailers and CPG companies alike.
