Target, Pregnancy, and Predictive Analytics, Part I

There have been a plethora of tweets about the New York Times article "How Companies Learn Your Secrets", mostly focused on the story of how Target can predict if a customer is pregnant. The tweets I've seen on this most often have a reaction that this is somewhat creepy or invasive. I may write more on this topic at some future time (which probably means I won't!) because I don't find it creepy at all that a company would try to understand my behavior and infer the cause of that behavior. But I digress…

The parts of the article I find far more interesting include these:

“It’s like an arms race to hire statisticians nowadays,” said Andreas Weigend, the former chief scientist at Amazon.com. “Mathematicians are suddenly sexy.”

and

Habits aren’t destiny — they can be ignored, changed or replaced. But it’s also true that once the loop is established and a habit emerges, your brain stops fully participating in decision-making. So unless you deliberately fight a habit — unless you find new cues and rewards — the old pattern will unfold automatically.

Part I will address the first question, and next week I'll post the second, much longer part.

First, mathematics and predictive analytics…

The first quote is a tremendous statement and one that all of us in the field should take notice of. While college students enrollment with STEM majors continues to decline, we have fewer and fewer candidates (as a percentage) to choose from.

But I don't think this is necessarily hopeless. I just finished teaching a text mining course, and one woman in the course told me that she never liked mathematics, yet it was obvious that she not only did data mining, but she understood it and was able to use the techniques successfully. There is something different about statistics, data mining and predictive analytics. It isn't math, it's forensic. It's a like solving a puzzle rather than proving a theorem or solving for "x".

Almost every major retailer, from grocery chains to investment banks to the U.S. Postal Service, has a “predictive analytics” department devoted to understanding not just consumers’ shopping habits but also their personal habits, so as to more efficiently market to them.

Really? I appreciate the statement of how widespread predictive analytics is. But I think it overstates the case. I've personally done work for retailers and other major organizations without predictive analytics departments. Now they may have several individuals who are analysts, but they aren't organized as a department. More often, they are part of the "marketing" department with an "analyst" title. This matters because collaboration is key in building predictive models well. One thing I try to encourage with all of my customers is building a collaborate environment where ideas, insights, and lessons learned are exchanged. With most customers, this is something they already do or are eager to do. With a few it has been more challenging.

“But Target has always been one of the smartest at this,” says Eric Siegel, a consultant and the chairman of a conference called Predictive Analytics World. “We’re living through a golden age of behavioral research. It’s amazing how much we can figure out about how people think now.”

I completely agree with Eric that we live in a world now where we finally have enough data, enough accessible data, the technical ability, and the interest in understanding that data. These are indeed good times to be in predictive analytics!

We need both kinds of analysts: the mathematically astute one, and those that don't care about the match, but understand deeply how to build and use predictive models. We need to develop both kinds of analysts, but there are far more of the latter, and they can do the job.