Sunday, December 06, 2015

Predictive Modeling Skills: Expect to be Surprised

Excerpted from Chapter 1 of my book Applied Predictive Analytics, Wiley 2014
Conventional wisdom says that predictive modelers need to have an academic background in statistics, mathematics, computer science, or engineering. A degree in one of these fields is best, but without a degree, at a minimum, one should at least have taken statistics or mathematics courses. Historically, one could not get a degree in predictive analytics, data mining, or machine learning.
This has changed, however, and dozens of universities now offer master’s degrees in predictive analytics. Additionally, there are many variants of analytics degrees, including master’s degrees in data mining, marketing analytics, business analytics, or machine learning. Some programs even include a practicum so that students can learn to apply textbook science to real-world problems.
One reason the real-world experience is so critical for predictive modeling is that the science has tremendous limitations. Most real-world problems have data problems never encountered in the textbooks. The ways in which data can go wrong are seemingly endless; building the same customer acquisition models even within the same domain requires different approaches to data preparation, missing value imputation, feature creation, and even modeling methods.
However, the principles of how one can solve data problems are not endless; the experience of building models for several years will prepare modelers to at least be able to identify when potential problems may arise.
Surveys of top-notch predictive modelers reveal a mixed story, however. While many have a science, statistics, or mathematics background, many do not. Many have backgrounds in social science or humanities. How can this be?
Consider a retail example. The retailer Target was building predictive models to identify likely purchase behavior and to incentivize future behavior with relevant offers. Andrew Pole, a Senior Manager of Media and Database Marketing described how the company went about building systems of predictive models at the Predictive Analytics World Conference in 2010. Pole described the importance of a combination of domain knowledge, knowledge of predictive modeling, and most of all, a forensic mindset in successful modeling of what he calls a “guest portrait.”
They developed a model to predict if a female customer was pregnant. They noticed patterns of purchase behavior, what he called “nesting” behavior. For example, women were purchasing cribs on average 90 days before the due date. Pole also observed that some products were purchased at regular intervals prior to a woman’s due date. The company also observed that if they were able to acquire these women as purchasers of other products during the time before the birth of their baby, Target was able to increase significantly the customer value; these women would continue to purchase from Target after the baby was born based on their purchase behavior before.
The key descriptive terms are “observed” and “noticed.” This means the models were not built as black boxes. The analysts asked, “does this make sense?” and leveraged insights gained from the patterns found in the data to produce better predictive models. It undoubtedly was iterative; as they “noticed” pat- terns, they were prompted to consider other patterns they had not explicitly considered before (and maybe had not even occurred to them before). This forensic mindset of analysts, noticing interesting patterns and making connections between those patterns and how the models could be used, is critical to successful modeling. It is rare that predictive models can be fully defined before a project and modelers can anticipate all of the most important patterns the model will find. So we shouldn’t be surprised that we will be surprised, or put another way, we should expect to be surprised.

This kind of mindset is not learned in a university program; it is part of the personality of the individual. Good predictive modelers need to have a forensic mindset and intellectual curiosity, whether or not they understand the mathematics enough to derive the equations for linear regression.
(This post first appeared in the Predictive Analytics Times)


Joseph Johnson said...

The idea of succeeding as a Data Scientist without a formal graduate degree may not be widely accepted by Fortune 500 companies, but there is a strong case that, not only is it possible, but it may be ideal to not have such a degree. John Hunstman Sr., founder of Hunstman Corporation, first brought a similar idea to my attention when he stated that he prefered MBA graduates over JD/MBA graduates. At the time, JD/MBA’s were considered a significant upgrade to a simple MBA. However, Hunstman noticed that the legal training engrained in the students a constraining fear of risk and a sense of caution that would lead to ideas being killed by fear of potential legal issues. He would rather have the MBA who was free to dream without worrying about legal implications. (see Winners Never Cheat by John Hunstman, Sr.)

In a similar manner, individuals who are steeped in Statistics may not be able to overcome the ‘sloppier’ nature of Big Data analysis. With Big Data, thousands of samples can be thrown out with affecting a model, p-values can be ignored, and algorithms can be set up to execute automatically and repeatedly.

Gaining a PhD in Machine Learning may also cripple one in their willingness to experiment, apply heuristics or attempt an approach this not painfully incremental in its variation. These approaches may appear to a PhD to be “hackish” and inelegant. However, there is no royal road to Data Science. Elegant and clean solutions often come after long hours of messy, sloppy experiments, rather than unfolding effortlessly as we find the right academic solution.

In summary, don’t let lack of a formal degree paralyze you from your pursuit of becoming an accomplished Data Scientist!

Brooke said...

What it really boils down to is that a lot of data science is all about asking questions about what you see -- and a formal degree in a data science field may or may not give you the skills to ask the right questions, look at the data the right way, or try the right experiments. As explained in the case with Target, domain knowledge and a "forensic mindset" are some of the most critical skills you can have for discovery.

Coepd said...
This comment has been removed by a blog administrator.
Muhammad Hassan said...
This comment has been removed by a blog administrator.