Saturday, February 18, 2012

Target, Pregnancy, and Predictive Analytics, Part I

There have been a plethora of tweets about the New York Times article "How Companies Learn Your Secrets", mostly focused on the story of how Target can predict if a customer is pregnant. The tweets I've seen on this most often have a reaction that this is somewhat creepy or invasive. I may write more on this topic at some future time (which probably means I won't!) because I don't find it creepy at all that a company would try to understand my behavior and infer the cause of that behavior. But I digress…

The parts of the article I find far more interesting include these:

“It’s like an arms race to hire statisticians nowadays,” said Andreas Weigend, the former chief scientist at “Mathematicians are suddenly sexy.”


Habits aren’t destiny — they can be ignored, changed or replaced. But it’s also true that once the loop is established and a habit emerges, your brain stops fully participating in decision-making. So unless you deliberately fight a habit — unless you find new cues and rewards — the old pattern will unfold automatically.

Part I will address the first question, and next week I'll post the second, much longer part.

First, mathematics and predictive analytics…

The first quote is a tremendous statement and one that all of us in the field should take notice of. While college students enrollment with STEM majors continues to decline, we have fewer and fewer candidates (as a percentage) to choose from.

But I don't think this is necessarily hopeless. I just finished teaching a text mining course, and one woman in the course told me that she never liked mathematics, yet it was obvious that she not only did data mining, but she understood it and was able to use the techniques successfully. There is something different about statistics, data mining and predictive analytics. It isn't math, it's forensic. It's a like solving a puzzle rather than proving a theorem or solving for "x".

Almost every major retailer, from grocery chains to investment banks to the U.S. Postal Service, has a “predictive analytics” department devoted to understanding not just consumers’ shopping habits but also their personal habits, so as to more efficiently market to them.

Really? I appreciate the statement of how widespread predictive analytics is. But I think it overstates the case. I've personally done work for retailers and other major organizations without predictive analytics departments. Now they may have several individuals who are analysts, but they aren't organized as a department. More often, they are part of the "marketing" department with an "analyst" title. This matters because collaboration is key in building predictive models well. One thing I try to encourage with all of my customers is building a collaborate environment where ideas, insights, and lessons learned are exchanged. With most customers, this is something they already do or are eager to do. With a few it has been more challenging.

“But Target has always been one of the smartest at this,” says Eric Siegel, a consultant and the chairman of a conference called Predictive Analytics World. “We’re living through a golden age of behavioral research. It’s amazing how much we can figure out about how people think now.”

I completely agree with Eric that we live in a world now where we finally have enough data, enough accessible data, the technical ability, and the interest in understanding that data. These are indeed good times to be in predictive analytics!

We need both kinds of analysts: the mathematically astute one, and those that don't care about the match, but understand deeply how to build and use predictive models. We need to develop both kinds of analysts, but there are far more of the latter, and they can do the job.


Will Johnson said...

I love this statement

"There is something different about statistics, data mining and predictive analytics. t isn't math, it's forensic. It's a like solving a puzzle rather than proving a theorem or solving for "x"."

That is exactly how I have felt about data analysis and exactly the reason why I'm going to DePaul's predictive analytics graduate program. You can solve a lot of puzzles with just SQL and plugging away, but when you start applying statistics, your results taste much more sweet.

As for understanding the math behind it all, it's fascinating, but most major universities will teach you the tools to get your ANOVAs and regressions, but only briefly touch on the behind the scenes.

Meta Brown said...


These quotes about the hot market for statisticians really do create a misleading impression. It's so true that many organizations have a lot of talent already in house, or that could be easily hired if only they would establish realistic expectations, invest in training and not succumb to recruiting fantasies.

As for the creepy factor, it seems to me that in the press it oftens appears that businesses are far more perfect at predictive analytics than they really are. People might react differently if they realized how few organizations take this seriously, and how hard it is for the ones who are trying to do it right.

I posted a few observations about this story in my blog in Handling Secrets and More on Secrets Secrets .

Curtis said...

I enjoyed your post. I happen to believe that Mathematicians have always been sexy.

I agree with your statement that it is an overstatement that almost every organization has a predictive analytics department. I hope this done to improve the analytics. The analysts need to be working closely with with the subject matter experts in order to be appropriately aware of the problem to be solved. A siloed predictive analytics department would likely have better expertise but could easily be disconnected from the rest of the business.

I also resonate with your last statement. We have the data, the processing power, the science and a plethora of interesting problems to be solved. This is great news (and job security) for a good analyst. While the technology has advanced quite a long way, building a useful model still requires expertise that only an good analyst can provide.