# Tactical and practical approach for treating outliers and missing values

Missing Data

There are multiple types of missing data e.g. Missing at Random (MAR) and Missing at Completely Random (MCAR) missing data. For details on the classification examples, users are referred to Little and Ruben1.  For satisfying both MAR and MCAR, the missing records or observation should not be related to specific information.  Example, when housewife fill up the information, the income field may be missing but for a reason.

There are multiple approaches available for missing value treatment and analysis. One of the important points to keep in mind is to review the variables and find out the rationale around missing values. There may a business reason for missing value and the reason could be helpful e.g. in understanding customer behavior.  A few years back we were building a customer churn model for a telecom client and found that a variable had around 80% missing values. Typically analyst would exclude the variables with over 30-40% missing.  When we look at the variables, we found that one of the variables was “Value of international calls”. Of course, it is not expected that all the customers would be international callers.

A few approaches on missing value treatment

• Deletion of missing observations:  This approach can be adopted with assumption of Missing at Random (MAR) or Missing Completely at Random (MCAR) otherwise the sample could be bias.
• Replacing with zero, mean or median values: This approach can also cause bias in mean or variance estimation.
• Using Multiple Imputation2,3 techniques

In the graph, it seems some of the values are outliers, but actually they are missing values. Analysts have to be careful about these values. Some time the missing values are denoted with 99999 etc. In this case for missing date of birth (DOB) and Start Date, a default date is populated hence when age and years with an organization is calculated, it has some patterns with exceptionally high values

Outlier Data

Outlier data points are the observations and values which are significant beyond the typical values of a variable. For example, income of a successful businessman or COE may have a value which is significantly higher than the typical values. The inclusion of such observations may cause bias in estimates including mean or variance values.  The impact could be more pronounced on a sample depending on whether these observations are selected in a sample.

In a statistical or predictive modeling, the outliers could be two types, first outlier values for a dependent variable and second outlier values for a predictor. Outliers for predictor variables are also called leverage points.  Residual analysis for regression and graphical analysis are some of the ways to identify outliers.

Why outliers are important? How outliers are different from influential points? How outliers can be detected? How robust regression can help?

WOE Variable transforming for tackling missing and outlier observations

One of the practical approaches adopted by many practitioners while building predictive model using Binary Logistic Regression is transforming variables to Weightage of Evidence (WOE) variables.  WOE variable transformation is used for tackling both missing and outliers. Missing or outlier classes are grouped with other classes based on Weight of Evidence (WOE) using fine and coarse classing.

Reference

1 Little, R.J.A. & Rubin, D.B. (1987). Statistical analysis with missing data. New York: Wiley.

# Approaches to build Binary Predictive Model

Summary

A predictive model is developed and used to predict an outcome. When the dependent variable (outcome) is dichotomous or has two levels (e.g.  Credit application accepted or rejected), it is called a binary predictive model.

Binary logistic regression, Decision Tree or Neural Network Statistical Techniques can be used to build a model when target or dependent variable is binary.  Predictors can be of any type – categorical (e.g. marital status), ordinal (e.g. income level) and continuous (e.g.  Spend amount). Predictor variables are also called independent or exploratory variables.

Binary predictive model has many real life applications across industries. And binary logistic regression is one of the commonly used techniques to build the predictive model.  When binary logistic regression is used for developing a predictive model, the input predictor variables are transformed to improve the model fit. This is called variable transformation. We are limiting the discussion to predictor variable transformation.

For explaining the below approaches, a customer attrition example is considered. If a customer attrite in a period the target variable takes value as 1 otherwise 0. Customer Years with an organization is taken as a predictor variable.

• Approach 1: Continuous Transformation
• Approach 2: Dummy Variable Creation
• Approach 3: WOE Variable Creation

Before advent of Business Intelligence (BI) tools such as Cognos, and Business Objects , the regular operational and analytical reports were developed using combination of tools. For example, getting summarized data using Oracle SQL and then producing reports in Microsoft Excel.  Many organizations are still pursuing this approach for variety of reasons.

The Business Intelligence (BI) tools have helped some of the organizations by

• Shifting focus from developing the reports to analyzing the reports
• Improving efficiency of report development process
• Providing flexibility in slicing and dicing the data

Some of the challenges associated with the Business Intelligence (BI) tool adoption approach for Management & Business reporting are:

• Lift and Shift approach: Typically, the reporting end users are from the business and they decide the layouts and formats of the reports. Since, they already had reports available, they demanded the almost same report be generated, leading to complicated data model and longer development cycle. If an existing report was coming of Base SAS, multiple aggregation and steps would have used to develop the final report. When the same report is required to come off a BI tool, the complications were manifold. The reason, BI tool is meant to generate all the reports with limited data duplication and standard data model.
• Known and Static Requirements: I am a critic of the Business Intelligence tools. The reason, an underlying assumption with BI tool is that the reporting requirements are static and known. This is not an appropriate or valid assumption in most of the scenarios. For example, due to customer migrating to direct channels quite a few reports may not be relevant. Also, there will be additional reporting requirements which may not be directly available out of the BI tools. And the typical BI tool deployment cycle– requirement understanding to go live- is around a year.  So, the business has its own concerns on initiating one more development cycle to incorporate the new or changes in the requirements. The outcome, BI tool and the reports remains unused.
• New Trends remain buried in obsolete groups: Typically, the variable groups are defined once at the time of BI tool deployment and remain static. The changes in customer behavior may be buried under the groups defined. For example, initial age bucket was 25-35 years, and due to economic and technological changes customers aged between 25-30 years may behave differently.  But, the new trend and insight remained buried in the group due to averaging or aggregation.

# Real time offer management in Retail scenario using Big Data

Retail customers walk & search across store aisles and pick up products. The list of products in the customers’ basket depends on customers’ actual and perceived need. The perceived need could be generated and altered by the promotions and offers.

I am sure it would have happened to you. When you went a supermarket to buy couple of products, but ended up buying a lot of additional products. Whether you realize or not the additional products selections are driven by the promotions in the store.

In another scenario, a supermarket advertised discounts on a list of items if purchased together. For example, if a customer buys 5Kg floor, 5Litre Oil and 5Kg Sugar then overall cost 20% less. The customer may not require buying sugar but s/he is buying due to the offer.

The product combinations in an advertisement and the in-store promotions could be developed based on the customers’ transaction patterns and product basket analysis.  But those may not necessarily align to individual customer needs.

A few and probably quite a few customers visit different aisles and SKUs, looks at the products but do not put in the basket.  Hypothesis is that if a customer visit an aisle and spend time looking at the product SKUs, the customer have the need. The need is expected to be stronger than the perceived need or the identified by statistical models.

Critical and key points are tracking customer movement across store aisles and identifying the product SKUs which the customer have looked at and the amount of time spend by the customers.

When the customer completes its basket and reaches the counter, based on the above information, the customer’s basket and historical spending pattern, a real time and personalized offer could be solicited to the customer. The offer so made will be far more effective and personalized to the customers.

The processing of the above the information in real time requires sophisticated algorithms and analysis capability. The big data platform and big data analytics could be looked at as a potential solution.

# Big Data Analytics – Driving value using unstructured data

Big Data has 3 important dimensions – Volume, Variety and Velocity to add Value to the organizations. In the article data Variety dimension will be explored and discussed to understand how Big Data adds value the organizations.

Data Variety can be segmented into two groups –Structured Data such as customer age, income and name, and Un-structured Data such as customer calls, emails, social media comments and tweets, and videos.

The unstructured data can be used to augment the information availability to generate more accurate insights, or it can be used on its own to generate insights. The big data analytics plays an important role in both the aspects.

Organization can supplement structured data analytics with unstructured data & analytics. Customer calls are recorded and stored only to increase the expenditure. The structured part of the customer calls – number of times customer and what time/days a customer called- are used mainly. The central or most important information – why a customer called? Was the customer satisfied with the response? – is not analyzed and actioned upon.

The big data analytics on unstructured data can help in answering the questions which otherwise would have been significantly difficult or impossible to answer. The unstructured data analytics will help in detecting common concerns of the customers, identifying opportunities in terms of providing right product and services, and servicing the customers effectively.

Steps used in Unstructured Data Analytics

Steps used in Unstructured Data Analytics

Unstructured Data is analyzed using text analytics and big data tools to identify categories and key words based on the data.  The categories and key words information can be converted into structured data for additional analysis, or can be analyzed to identify the patterns and trends. The identified patterns and trends can be explored to develop actionable insights.

An example of generating actionable insights using calls data

For a bank, the customer calls data were analyzed to identify reasons of customer closing a product. All the customer calls which were related to the product were segregated for the analysis. Based on verbatim spoken during the calls, the calls data were analyzed to identify the key words. The key words which were related to customer closing the product are grouped together. The key word categories and could were analyzed. The analysis has helped in identifying top 5 reasons of customers closing the product and helped the management in taking appropriate actions – better communication on product features and their value to the customers.

# Leveraging Big Data Analytics across Customer Life Cycle for Retail Bank

Big Data Analytics helps in developing insights which are relevant and required across customer cycle –acquisition, development and retention.  Using Big Data analytics, the social media data can be analyzed to help marketing manager focus on right social media channel for customer acquisition. Customer interaction and transaction data offer significant opportunities for the banks to understand customer needs. The channel interaction and transaction can be analyzed effectively using Big Data platform and tools. Also, analysis of customer complaint logs and emails to identify and understand common customer concerns for building strategy to address customer concerns proactively.

Retail Bank and Big Data – Examples across life cycle

# What is Analytics?

Analytics is used to validate and formulate strategies for critical aspects of a business or entity using the power of available data & information along with business understanding. Analytics is employed for understanding the current state of the business, segregating drivers of the current business performance, identifying the factors or trends which could impact the future income & profitability of the business and helping managers in formulating strategies for the business growth.

For example, if someone interested to set up a new vegetable shop, the main aim is to make it flourish and increase revenue. What are the questions which the person will want to have the answers for making informed decisions?

• What is the size of the market?
• Where should the shop be located?
• Who will be target customer group who will shop in this shop?
• What will be expected demand for each vegetable?
• How will I optimally procure vegetables?
• What will be price points?
• How to measure my performance against the targets which I have set?
• How to reassess important assumptions on an ongoing basis?

In other example, if someone is already managing a business or shop and aspiring for next stride, the person will look for right ideas and insights to build right strategies, and ahead of competitors.  What are the questions which the person will look to answer for formulating right strategies?

• What is the performance of the business and various business segments?
• What is cost efficiency ratio for various business segments?
• What is Return on Investment (ROI) for various activities?
• How to improve performance of the lagging business segments?
• Who are the competitors and what actions/strategies are they taking?
• How economic condition and competitors actions are impacting the business performance?
• What are the drivers of current business performance?
• How are these business performance drivers going to change in future?
• What are the new and emerging trends from customer, technology, economic & political and regulatory perspective?
• What are the potential opportunities for the business to be ahead of other competitors?
• How to assess feasibilities of potential opportunities with minimal investment?

The businessmen have been answering these questions for setting up shops and running the business directly and indirectly.

Due changes in business scenarios the application of analytics for business process has gone up. The businesses are using analytics as a differentiator to grow the business and profitably.

Drivers of Analytics and Insights adoption

• Competition and choices e.g. number of companies and brands selling Mobile phones
• Blurring competition boundaries e.g. retailers selling financial products
• Availability of data &information e.g.  every interaction and transactions with a bank is captured
• Technology to process information e.g. millions of transactions can be processed and analyzed quickly,
• Advancement of analytical techniques   e.g. Neural Network and Supervised Vector Machine (SVM)
• Skilled resources with knowledge of technology, analytical techniques

Some of the prominent examples of successful application of analytics are

Capital One (https://www.capitalone.com/ ) for Credit Card: Capital One has mass – customized credit card and more importantly has established an analytics closed loop(Identifying, Testing, Learning and Deploying the key insights).  They run thousands of test cases every year and learn from those experiments.

Tesco Clubcard (www.tesco.com/): Tesco Clubcard is one of the most interesting and successful story of analytics application.  With the help of DunnHumby, Tesco has converted millions of shopping transactions to insight, helping Tesco to become one of prominent supermarket in UK and in the work. TescoClubcardis one of the most successfulCustomer Loyalty Program in the world.

Netflix (http://ir.netflix.com/): Netflix is a dominant company in the DVD movie rental industry.   A collaborative filtering based recommendation engine is being used to identify movies which customers will be interested in. This is giving Netflix a huge competitive advantage.

Ecosystem for delivering Analytics and Insights

For providing relevant and actionable business insights, first the Data Driven Insights has to be a Strategic Priority for the business decision making and senior leadership shall provide required support, second delivery infrastructure – data, technology, and analytical systems- must be in place to provide Data Analytics and Insights for the business quickly and effectively, and last the availability of analytically skilled resources.

• Strategic Priorities& Focus: Mandate and focus from Senior Management to leverage Data Analytics and Insights for Decision Making
• Delivery Infrastructure: Availability of Data, Technology and Analytical Systems for carrying out data analytics and deriving and developing Insights
• Analytics Resources: One of the most important cogs in analytics delivery wheel is an analyst. Analyst should be proficient in various analytical techniques and have understanding of business processes and priorities

References

Larry E. Rosenberger, John Nash, and Ann Graham, The Deciding Factor: The Power of Analytics to Make Every Decision a Winner

Clive Humby, Terry Hunt, and Tim Phillips, Scoring Points: How Tesco is Winning Customer Loyalty

Thomas H. Davenport, and Jeanne G. Harris, Competing on Analytics: The New Science of Winning

# Analytics as a Close Loop Value Delivery Cycle

Analytics as a Close Loop Value Delivery Cycle

In these tough economic times, many organizations are facing financial hardship and must take tough and right decisions. Every investment and decision is being scrutinized. Many time business managers face challenges in terms of selecting right choices and making right decisions just based on gut feeling. Also, it is significantly important for the business managers to know what has worked and what has not. It is important to set up effective learning mechanism before start processing testing the idea(s).

An Analytics Closed Loop Value Delivery Cycle can help the business managers in leveraging data analytics and insights for Identifying potential ideas or choices, testing the list of ideas, learning from the test on what has worked and finally deploying the most successful idea for business value addition.

Analytics Closed Loop Value Delivery Cycle

Analytics as a Value Delivery Cycle

• Identify
• Test
• Learn
• Deploy

Please find detailed views with an example attached..