Where it began: John Elder and Dean Abbott at Barron Associates

I came across this photo today and couldn't resist. John Elder and I worked at the company Barron Associates, Inc. (BAI) in Charlottesville, VA in the 80s (John was employee #1). Hopefully you can identify John in the back right and me in the front right, though it will take some good pattern matching to do so!

The Founder and President of the company was Roger Barron. Both John and I were introduced to statistical learning methods at BAI and of course went on to careers in the field now known as Data Mining or Predictive Analytics (among other things). I write about my experience with BAI in the forthcoming book Journeys to Data Mining: Experiences from 15 Renowned Researchers, Ed. by Dr Mohamed Medhat Gaber, published by Springer. Authors in the book include John (thanks to John for recommending my inclusion in the book), Gregory Piatetsky-Shapiro, Mohammed J. Zaki, and of course several others.

The photo appeared as I was searching for descriptions of our field back in the 80s and was looking in particular for the Barron and Barron paper "Statistical Learning Networks: A Unifying View", where "Statistical Learning Networks" was the phrase of interesting, along with "Models", "Classifiers", and "Neural Networks". We used to refer to the field as "Pattern Recognition" and "Artificial Intelligence". It's interesting to note that pattern recognition on Wikipedia contains a list of "See Also" terms that includes the more modern terms such as data mining and predictive analytics.

I will post within a couple days on the pattern recognition terms of the day and how they are changing.

Predictive Analytics World Had the Target Story First

The New York Times Magazine article "How Companies Learn Your Secrets" by Charles Duhigg with the key descriptions of Target, pregnancy, predictive analytics (blogged on here and here) certainly generated a lot of buzz; if you are unable to see the NYTimes Magazine article, the Forbes summary is a good summary. However, few know that Eric Siegel booked Andy Pole for the October 2010 Predictive Analytics World conference as a keynote speaker. The full video of that talk is here.

In this talk, Mr. Pole discussed how Target was using Predictive Analytics including descriptions of using potential value models, coupon models, and...yes...predicting when a woman is due (if you aren't the patient type, it is at about 34:30 in the video). These models were very profitable at Target, adding an additional 30% to the number of woman suspected or known to be pregnant over those already known to be (via self-disclosure or baby registries). The fact that this went on for over a year after the Predictive Analytics World talk and before the fallout tells me that it didn't cause significant problems for Target prior to the attention brought to the subject related to the NYTimes article.

After watching the talk, what struck me most was that Target was applying a true "360 deg" customer view by linking guests through store visits, web, email, and mobile interactions. In addition, close attention was paid to linking the interactions so that coupons made sense: they didn't print coupons to those who had just purchased items they score high for couponing, and they identify which mediums don't generate responses and stop using those channels.

I suspect what Target is doing is no different than most retailers, but this talk was an interesting glimpse into how much they value the different channels and try to find ways to provide customers with what they are most interested in, and suppress what they are not interested in.

Another Wisdom of Crowds Prediction Win at eMetrics / Predictive Analytics World

This past week at Predictive Analytics World / Toronto (PAW) has been a great time for connecting with thought leaders and practitioners in the field. Sometimes there are unexpected pleasures as well, which is certainly the case this time. One of the exhibitors for the eMetrics conference, co-locating with PAW at the venue, was Unilytics, a web analytics company. At their booth there was a cylindrical container filled with crumpled dollar bills with a sign soliciting predictions of how many dollar bills were in the container (the winner getting all the dollars). After watching the announcement of the winner, who guessed $352, only $10 off from the actual $362, I thought this would be the perfect opportunity for another Wisdom of Crowds test,just like the one conducted 9 months ago and blogged on here.

Two Unilytics employees at the booth, Gary Panchoo and Keith MacDonald, were kind enough to indulge me and my request to compute the average of all the guesses. John Elder was also there, licking his wounds from finished a close second as his guess, $374 was off by $12, a mere $2 away from the winning entry! The results of the analysis are here (summary statistics created by JMP Pro 10 for the mac). In summary, the results are as follows:

Dollar Bill Guess Scores

Method	Guess Value	Error
Actual	362
Ensemble/Average (N=61)	365	3
Winning Guess (person)	352	10
John Elder	374	12
Guess without outlier (2000), 3rd place	338	24
Median, 19th place	275	87

So once again, the average of the entries (the "Crowds" answer) beat the single best entry. What is fascinating to me about this is not that the average won (though this in of itself isn't terribly surprising), but rather how it won. Summary statistics are below. Note that the Median is 275, far below the mean. Not too how skewed the distribution of guesses are (skew = 3.35). The fact that the guesses are skewed positively for a relatively small answer (362) isn't a surprise, but the amount of skew is a bit surprising to me. What these statistics tell us is that while the mean value of the guesses would have been the winner, a more robust statistic would not, meaning that the skew was critical in obtaining a good guess. Or put another way, people more often than not under-guessed by quite a bit (the median is off by 87). Or put a third way, the outlier (2000) which one might naturally want to discount because it was a crazy guess was instrumental to the average being correct. In the prior post on this from July 2011, I trimmed the guesses, removing the "crazy" ones. So when should we remove the wild guesses and when shouldn't we? (If I had removed the 2000, the "average" still would have finished 3rd). I have no answer to when the guesses are not reasonable, but wasn't inclined to remove the 2000 initially here. Full stats from JMP are below, with the histogram showing the amount of skew that exists in this data.

Distribution of Dollar Bill Guesses - Built with JMP

Summary Statistics

Statistic	Guess Value
Mean	365
Std Dev	299.80071
Std Err Mean	38.385548
Upper 95% Mean	441.78253
Lower 95% Mean	288.21747
N	61
Skewness	3.3462476
Median	275
Mode	225
2% Trimmed Mean	331.45614
Interquartile Range	185.5

Note: The mode shown is the smallest of 2 modes with a count of 3.

Quantiles

Quantile		Guess Value
100.0%	maximum	2000
99.5%		2000
97.5%		1546.8
90.0%		751.6
75.0%	quartile	406.5
50.0%	median	275
25.0%	quartile	221
10.0%		145.6
2.5%		98.2
0.5%		96
0.0%	minimum	96

Dilbert, Database marketing and spam

Ruben's comment that referred to spam reminded me of an old Dilbert comic which conveys the misconception about database marketing (e-marketing) and spam.

I know Ruben well and know he was poking fun, though I still have to correct folks who after finding out I do "data mining" actually comment that I'm responsible for spam. Answer: "No, I'm the reason you don't get as much spam!"

What I'm Working On

Sometimes folks ask me what I'm doing, so I thought I'd share a few things on my plate right now:

Courses and Conferences
1. Reading several papers for the KDD 2012 Conference Industrial / Government Track
2. Preparing for the Predictive Analytics World / Toronto "Advanced Methods Hands-on:
Predictive Modeling Techniques" workshop on April 27. I'm using the Statsoft Statistica package.
3. Starting preparation for a talk at the Salford Analytics and Data Mining Conference 2012, "A More Transparent Interpretation of Health Club Surveys" on May 24. It will highlight use of the CART software package in the analysis. This was work that motivated interviews with New York Times reporter Charles Duhigg, and ended with a mention (albeit *very* briefly) in the fascinating new book by Duhigg, "The Power of Habit: Why We Do What We Do in Life and Business".
4. Working through data exercises for the next UCSD-Extension Text Mining Course on May 11, 18, and 25th. I'm using KNIME for this course.

Consulting
Approximately 80% of my time is spent on active consulting. While I can't describe most of the work I'm doing, my current clients are in the following domains:
1. Web Analytics and email remarketing for retail via a great startup company, Smarter Remarketer headed by Angel Morales (Founder and Chief Innovation Officer), Howard Bates (CEO), and me (Founder and Chief Scientist).
2. Customer Acquisition, web/online and offline (2 clients)
3. Tax Modeling (2 clients)
4. Data mining software tool selection for a large health care provider.

Here's to productive application of predictive analytics!

Why Defining the Target Variable in Predictive Analytics is Critical

Every data mining project begins with defining what problem will be solved. I won't describe the CRISP-DM process here, but I use that general framework often when working with customers so they have an idea of the process.

Part of the problem definition is defining the target variable. I argue that this is the most critical step in the process that relates to the data, and more important than data preparation, missing value imputation, and the algorithm that is used to build models, as important as they all are.

The target variable carries with it allthe information that summarizes the outcome we would like to predict from the perspective of the algorithms we use to build the predictive models. Yet this can be misleading is many ways. I'm addressing one way we can be fooled by the target variable here, and please indulge me to lead you down the path.

Let's say we are building fraud models in our organization. Let's assume that in our organization, the process for determining fraud is first to identify possible fraud cases (by tips or predictive models), then assign the case to a manager who determines which investigator will get the case (assuming the manager believes there is value in investigating the case), then assign the case to an investigator, and if fraud is found, the case is tried in court, and ultimately a conviction is made or the party is found not guilty.

Our organization would like to prioritize which cases should be sent to investigators using predictive modeling. It is decided that we will use as a target variable all cases that were found to be fraudulent, that is, all cases that had been tried and a conviction achieved. Let's assume here that all individuals involved are good at their jobs and do not make arbitrary or poor decisions (which of course is also a problem!)

Let's also put aside for a moment the time lag involved here (a problem itself) and just consider the conviction as a target variable. What does the target variable actually convey to us? Of course our desire is that this target variable conveys fraud risk. Certainly when the conviction has occurred, we have high confidence that the case was indeed fraudulent, so the "1"s are strong and clear labels for fraud.

But, what about the "0"s? Which cases do they include?
--cases never investigated (i.e., we suspect they are not fraud, but don't know)
--cases assigned to a manager who never assigned the case (he/she didn't think they were worth investigating).
--cases assigned to an investigator but the investigation has not yet been completed, or was never completed, or was determined not contain fraud
--cases that went to court but was found "not guilty"

Remember, all of these are given the identical label: "0"

That means that any cases that look on the surface to be fraudulent, but there were insufficient resources to investigate them, are called "not fraudulent. That means cases that were investigated but the investigator was taken off the case to investigate other cases are called "not fraudulent". It means too that court cases that were thrown out of court due to a technicality unrelated to the fraud itself are called "not fraud".

In other words, the target variable defined as only the "final conviction" represents not only the risk of fraud for a case, but also the investigation and legal system. Perhaps complex cases that are high risk are thrown out because they aren't (at this particular time, with these particular investigators) worth the time. Is this what we want to predict? I would argue "no". We want our target variable to represent the risk, not the system.

This is why when I work on fraud detection problems, the definition of the target variable takes time: we have to find measures that represent risk and are informative and consistent, but don't measure the system itself. For different customers this means different trade-offs, but usually it means using a measure from earlier in the process.

So in summary, think carefully about the target variable you are defining, and don't be surprised when your predictive models predict exactly what you told them to!

Target, Pregnancy, and Predictive Analytics,Part II

This is part II of my thoughts on the New York Times article "How Companies Learn Your Secrets".

In the first post, I commented on the quote

“It’s like an arms race to hire statisticians nowadays,” said Andreas Weigend, the former chief scientist at Amazon.com. “Mathematicians are suddenly sexy.”

Comments on this can be seen in Part I here.

In this post, the next portion of the article I found fascinating can be summarized by the section that says

Habits aren’t destiny — they can be ignored, changed or replaced. But it’s also true that once the loop is established and a habit emerges, your brain stops fully participating in decision-making. So unless you deliberately fight a habit — unless you find new cues and rewards — the old pattern will unfold automatically.

Habits are what predictive models are all about. Or putting as a question, "is customer behavior predictable based on their past behavior?" The Frawley, Piatetsky-Shapiro, Mattheus definition of knowledge discovery in databases (KDD) as follows:

Knowledge discovery is the nontrivial extraction of implicit, previously unknown,and potentially useful information from data. (PDF of the paper can be found here)

This quote has often been applied to data mining and predictive analytics, and rightfully so. We believe there are patterns hidden in the data and want to characterize those patterns with predictive modeols. Predictive models usually work best when individuals don't even realize what they are doing so we can capture their behavior solely based on what they want to do rather than behavior influence by how they want to be perceived, which is exactly how the Target models were built.

So what does this have to do with the NYTimes quote? The "habits" that "unfold automatically" as described in the article was fascinating precisely because predictive models rely on habits; we wish to make the connection between past behavior and expected result as captured in the data that are consistent and repeatable (that is, habitual!). These expected results could be "is likely to respond to a mailing", "is likely purchase a product online", "is likely to commit fraud", or in the case of the article, "is likely to be pregnant". Duhigg (and presumably Pole describing it to Duhigg) characterizes this very well. The behavior Target measured was shoppers purchasing behavior when they were to give birth some weeks or months in the future, and nothing more. They had to apply broadly to thousands of "Guest IDs" for models to work effectively.

The description of what Andy Pole did for target is an excellent summary of what predictive modelers can and should do. The approach included domain knowledge, understanding of what predictive models can do, and most of all a forensic mindset. I quote again from the article:

"Target has a baby-shower registry, and Pole started there, observing how shopping habits changed as a woman approached her due date, which women on the registry had willingly disclosed. He ran test after test, analyzing the data, and before long some useful patterns emerged. Lotions, for example. Lots of people buy lotion, but one of Pole’s colleagues noticed that women on the baby registry were buying larger quantities of unscented lotion around the beginning of their second trimester. Another analyst noted that sometime in the first 20 weeks, pregnant women loaded up on supplements like calcium, magnesium and zinc. Many shoppers purchase soap and cotton balls, but when someone suddenly starts buying lots of scent-free soap and extra-big bags of cotton balls, in addition to hand sanitizers and washcloths, it signals they could be getting close to their delivery date." (emphases mine)

To me, the key descriptive terms in the quote from the article are "observed", "noticed" and "noted". This means the models were not built as black boxes; the analysts asked "does this make sense?" and leveraged insights gained from the patterns found in the data to produce better predictive models. It undoubtedly was iterative; as they "noticed" patterns, they were prompted to consider other patterns they had not explicitly considered before (and maybe had not even occurred to them before). But it was these patterns that turned out to be the difference-makers in predicting pregnancy.

So after all my preamble here, the key take-home messages from the article are:
1) understand the data,
2) understand why the models are focusing on particular input patterns,
3) ask lots of questions (why does the model like these fields best? why not these other fields?)
4) be forensic (now that's interesting or that's odd...I wonder...),
5) be prepared to iterate, (how can we predict better for those customers we don't characterize well)
6) be prepared to learn during the modeling process

We have to "notice" patterns in the data and connect them to behavior. This is one reason I like to build multiple models: different algorithms can find different kinds of patterns. Regression is a global predictor (one continuous equation for all data), whereas decision trees and kNN are local estimators.

So we shouldn't be surprised that we will be surprised, or put another way, we should expect to be surprised. The best models I've built contain surprises, and I'm glad they did!

Target, Pregnancy, and Predictive Analytics, Part I

There have been a plethora of tweets about the New York Times article "How Companies Learn Your Secrets", mostly focused on the story of how Target can predict if a customer is pregnant. The tweets I've seen on this most often have a reaction that this is somewhat creepy or invasive. I may write more on this topic at some future time (which probably means I won't!) because I don't find it creepy at all that a company would try to understand my behavior and infer the cause of that behavior. But I digress…

The parts of the article I find far more interesting include these:

“It’s like an arms race to hire statisticians nowadays,” said Andreas Weigend, the former chief scientist at Amazon.com. “Mathematicians are suddenly sexy.”

and

Habits aren’t destiny — they can be ignored, changed or replaced. But it’s also true that once the loop is established and a habit emerges, your brain stops fully participating in decision-making. So unless you deliberately fight a habit — unless you find new cues and rewards — the old pattern will unfold automatically.

Part I will address the first question, and next week I'll post the second, much longer part.

First, mathematics and predictive analytics…

The first quote is a tremendous statement and one that all of us in the field should take notice of. While college students enrollment with STEM majors continues to decline, we have fewer and fewer candidates (as a percentage) to choose from.

But I don't think this is necessarily hopeless. I just finished teaching a text mining course, and one woman in the course told me that she never liked mathematics, yet it was obvious that she not only did data mining, but she understood it and was able to use the techniques successfully. There is something different about statistics, data mining and predictive analytics. It isn't math, it's forensic. It's a like solving a puzzle rather than proving a theorem or solving for "x".

Almost every major retailer, from grocery chains to investment banks to the U.S. Postal Service, has a “predictive analytics” department devoted to understanding not just consumers’ shopping habits but also their personal habits, so as to more efficiently market to them.

Really? I appreciate the statement of how widespread predictive analytics is. But I think it overstates the case. I've personally done work for retailers and other major organizations without predictive analytics departments. Now they may have several individuals who are analysts, but they aren't organized as a department. More often, they are part of the "marketing" department with an "analyst" title. This matters because collaboration is key in building predictive models well. One thing I try to encourage with all of my customers is building a collaborate environment where ideas, insights, and lessons learned are exchanged. With most customers, this is something they already do or are eager to do. With a few it has been more challenging.

“But Target has always been one of the smartest at this,” says Eric Siegel, a consultant and the chairman of a conference called Predictive Analytics World. “We’re living through a golden age of behavioral research. It’s amazing how much we can figure out about how people think now.”

I completely agree with Eric that we live in a world now where we finally have enough data, enough accessible data, the technical ability, and the interest in understanding that data. These are indeed good times to be in predictive analytics!

We need both kinds of analysts: the mathematically astute one, and those that don't care about the match, but understand deeply how to build and use predictive models. We need to develop both kinds of analysts, but there are far more of the latter, and they can do the job.

Top 5 Posts from 2011

By far, the most visited post of 2011 was the "What Do Data Miners Need to Learn" post from June.

The top five visited posts that were first posted in 2011 are (with actual ranks for all posts):
1. What Do Data Miners Need to Learn
2. Statistical Rules of Thumb, Part III
3. Statistical Rules of Thumb, Part II
4. Number of Hidden Layer Neurons to Use
5. Statistics: The Need for Integration


The top six viewed posts in 2011 originally created prior to 2011 were:
1. Why Normalization Matters with K-Means (2009)
2. Free and Inexpensive Data Mining Software (2006)
3. Data Mining Data Sets (2008)
4. Can you Learn Data Mining in Undergraduate or Graduate School (2009)
5. Quotes from Moneyball (2007)
6. Business Analytics vs. Business Intelligence (2009)

The "Free Data Mining Tools" post is understandably relatively popular, even after 5 years. The Moneyball quotes has a particularly high bounce rate. I'm most surprised that the K-Means normalization post has remained popular for so long.

Models Behaving Badly

I just read a fascinating book review in the Wall Street Journal Physics Envy: Models Behaving Badly. The author of the book, Emanuel Derman (former head of Quantitative Analsis at Goldman Sachs) argues that the financial models involved human beings and therefore were inherently brittle: as human behavior changed, the models failed. "in physics you're playing against God, and He doesn't change His laws very often. In finance, you're playing against God's creatures."

I'll agree with Derman that whenever human beings are in the loop, data suffers. People change their minds based on information not available to the models.

I also agree that human behavioral modeling is not the same as physical modeling. We can use the latter to provide motivation and even mathematics for human behavioral modeling, but we should not take this too far. A simple example is this: purchase decisions sometimes depend not on the person's propensity to purchase alone, but also on whether or not they had an argument that morning, or if they just watched a great movie. There is an emotional component that data cannot reflect. People therefore behave in ways that on the surface are contradictory, seemingly "random", which is way response rates of 1% can be "good".

However, I bristle a bit at the the emphasis on the physics analogy. In closed systems, models can explain everything. But once one opens up the world, even physical models are imperfect because they often do not incorporate all the information available. For example, missile guidance is based on pure physics: move a surface on a wing and one can change the trajectory of the missile. There are equations of motion that describe exactly where the missile will go. There is no mystery here.

However, all operational missile guidances systems are "closed loop"; the guidance command sequence is not completely scheduled but is updated throughout the flight. Why? To compensate for unexpected effects of the guidance commands, often due to ballistic winds, thermal gradients, or other effects on the physical system. It is the closed-loop corrections that make missile guidance work. The exact same principal applies to your car's cruise control, chasing down a fly ball in baseball, or even just walking down the street.

For a predictive model to be useful long-term, it needs updating to correct for changes in the population the models are applied to, whether the models be for customer acquisition, churn, fraud detection, or any model. The "closed-loop" typical in data mining is called "model updating" and is critical for long-term modeling success.

The question then becomes this: can the models be updated quickly enough to compensate for changes in the population? If a missile can only be updated at 10Hz (10x / sec.) but uncertainties effect the trajectory significantly in milliseconds, the closed-loop actions may be insufficient to compensate. If your predictive can only be updated monthly, but your customer behavior changes significantly on a weekly basis, your models will be behind perpetually. Measuring the effectiveness of model predictions is therefore critical in determining the frequency of model updating necessary in your organization.

To be fair, until I read the book I have no quibble with the arguments. The arguments here are based solely on the book review and some ideas they prompted in my mind. I'd welcome comments from anyone who has read the book already.

The book can be found on amazon here.

UPDATE: Aaron Lai wrote an article for CFA Magazine on the same topic, also quoting Derman. I commend the article to all (note: this is a PDF file download).