The Power of Prescience: Achieving Lift with Predictive Analytics

I'll be participating in the DM Radio broadcast tomorrow, The Power of Prescience: Achieving Lift with Predictive Analytics Thursday, Feb 23 at 3pm ET. The best practices that we will be discussing include:

1) properly define the problem to be solved (don’t shoot in the dark); 2) identify a key target variable to predict (must be a good decision-making metric in the company); 3) determine what “good” means, success-wise (what is the baseline for success?); 4) identify the appropriate data that can aid in prediction. There’s also: 5) finding the right algorithms, but this doesn’t matter unless 1-4 are nailed.

I also plan on talking about the importance of proper perspective in building models. While we want predictive models to be good, even excellent, but in the end, we need the models to improve decision-making over what is done currently. I'm not advocating low expectations, just reasonable expectations.

The Judgement of Watson: Mathematics Wins!

Tom Davenport argues in this HBR article Why I'm Pulling for Watson - Tom Davenport - Harvard Business Review that

I want Watson to win. Why? It's elementary: my dear Watson is a triumph of human ingenuity. In other words, there is no way humans can lose this competition. Watson also illustrates that the knowledge, judgment, and insights of the smartest humans can be embedded into automated systems. I suspect that those automated systems will ultimately be used to make better decisions in many domains, and interact with humans in a much more intelligent way. If computers can persuade Alex Trebek that they're very smart—and that's what he said about Watson—they'll be able to interact effectively with almost any human with a problem to solve.

While this is true, I don't agree that Watson itself is using "judgement" or "making decisions". It appears to me that it is a very nice search engine that incorporates NLP to make these searches more relevant. It isn't giving opinions, synthesizing information to create innovative ideas, or making inferences through extrapolation, all things humans do on a regular basis. This has long been one of my complaints about the way neural networks were described: they "learn", they "think", they "make inferences". No, they are a nonlinear function that finds weights via gradient descent searches. The no more "learn" than logistic regression "learns".

A lot of the hype gets back to the old "hard AI" vs. "soft AI" debates that have been going on for decades. I appreciated very much the book by Roger Penrose on this subject, Shadows of the Mind: A Search for the Missing Science of Consciousness.

This isn't to minimize the incredible feat IBM has accomplished with Watson, or on a simpler level, the feats of decision-making that can be performed with nonlinear mathematics in neural networks or support vector machines. These are phenomenal accomplishments that are awe inspiring mathematically, and on a more practical level will assist us all in the future with improved ability to automate decision-making. Of course, these kinds of decisions are those that do not require innovation or judgement, but can be codified mathematically. Every time I check out at an automatic teller at Home Depot, deposit checks at an ATM, or even make an amazon purchase, I'm reminded of the depth of technology that makes these complex transactions simple to the user. Watson is the beginning of the next leap in this ongoing technological march forward, all created by enterprising humans who have been able to break down complex behavior into repeatable, reliable, and flexible algorithmic steps.

In the end, I agree with Mr. Davenport, "So whether the humans or Watson win, it means that humans have come out on top."

Predictive Analytics Innovation

The Predictive Analytics Summit, a relative newcomer to the Predictive Analytics conference circuit, will be held in San Diego on Feb 24-25. At the first Summit in San Francisco last Fall, I enjoyed several of the talks and the networking. This time I will be presenting a fraud detection case study.

Webinar with James Taylor -- 10 Best Practices in Operational Analytics

I'll be presenting a webinar with James Taylor this Wednesday at 10AM PST entitled "10 best practices in operational analytics".

One of the most powerful ways to apply advanced analytics is by putting them to work in operational systems. Using analytics to improve the way every transaction, every customer, every website visitor is handled is tremendously effective. The multiplicative effect means that even small analytic improvements add up to real business benefit.

In this session James Taylor, CEO of Decision Management Solutions, and Dean Abbott of Abbott Analytics will provide you with 10 best practices to make sure you can effectively build and deploy analytic models into you operational systems.

Predictive Analytics World Early-bird ends Monday

The earlybird special for Predictive Analytics World / San Francisco ends January 31, 2011 which saves you $200 on the conference rate and $100 on any workshop, including my Hands-On Predictive Analytics using SAS Enterprise Miner on March 17th.

More details on the 7 workshops can be found here.

Hope to see you there!

Do analytics books sell?

Kevin Hillstrom has a fascinating post on brief, technical ebooks (Amazon singles) sold on Amazon here: Kevin Hillstrom: MineThatData: Amazon Singles. His points: interesting content is what sells. Length doesn't matter, but these ebooks are typically less than 50 pages. Price doesn't matter.

Should I jump in? Should you?

Doing Data Mining Out of Order

I like the CRISP-DM process model for data mining, teach from it, and use it on my projects. I commend it to practitioners and managers routinely as an aid during any data mining project. However, while the process sequence is generally the one I use, I don't always; data mining often requires more creativity and "art" to re-work the data than we would like; it would be very nice if we could create a checklist and just run through the list on every project! But unfortunately data doesn't always cooperate in this way, and we therefore need to adapt to the specific data problems so that the data is better prepared.

For example, on a current financial risk project I am working, the customer is building data for predictive analytics for the first time. The customer is data savvy, but new to predictive analytics, so we've had to iterate several times on how the data is pulled and rolled up out of the database. In particular, target variable has had to be cleaned up because of historic coding anomalies.

One primary question to resolve for this project is an all-too-common debate over what is the right level of aggregation: do we use transactional data even though some customers have many transactions and some have few, or do we roll data up to the customer level to build customer risk models. (A transaction-based model will score each transaction for risk, whereas a customer-based model will score, daily, the risk associated with each customer given the new transactions that have been added.) There are advantages and disadvantages to both, but in this case, we are building a customer-centric risk model for reasons that make sense in this particular business context.

Back to the CRISP-DM process and why it is advantageous to deviate from CRISP-DM. In this project, we jumped from Business Understanding and the beginnings of Data Understanding straight to Modeling. I think in this case, I would call it "modeling" (small 'm') because we weren't building models to predict risk, but rather to understand the target variable better. We were not sure exactly how clean the data was to begin with, especially the definition of the target variable, because no one had ever looked at the data in aggregate before, only on a single customer-by-customer basis. By building models, and seeing some fields that predict the target variable "too well", we have been able to identify historic data inconsistencies and miscoding.

Now that we have the target variable better defined, I'm going back to the data understanding and data prep stages to complete those stages properly, and this is changing how the data will be prepped in addition to modifying the definition of the target variable. It's also much more enjoyable to build models than do data prep, so for me this was a "win-win" anyway!

Predictive Analytics Summit - Analytics Titles

I'm at the Predictive Analytics Summit in San Francisco. It is interesting to see the titles of Analytics people at the conference (here). They include CTO/Senior/Manager/VP of a variety of analytics variants: Predictive Analytics, Marketing Analytics, just Analytics, Data Analytics, Research & Analytics, Quant Research, etc. Others not here but that I've seen include Business Analytics and the variety of Data Mining titles.

There has been a lot of hype about data mining and predictive analytics being a great field to be in. It's interesting to me that (1) Predictive Analytics is so often part of the title now, lending credence to this term becoming a standard term companies use, and (2) the variety of ways quantitive modeling is described.

This conference is just one of many taking place in a short time period, including Predictive Analytics World, SAS M2010, IBM Information on Demand, and Teradata Partners conference, the SuperMath Conference in San Diego, and the ACM Data Mining Bootcamp in San Jose. Too many to attend all of them!

A humorous explanation of p-values

After Will's great post on sample sizes that referenced the youtube video entitled Statistics vs. Marketing, I found an equally funny and informative explanation on p-values here.

Aside from the esoteric explanations of what a p-value is, there is a point that I make often with customers that statistical significance (from p-values) is not the same thing as operational significance; just because you find a p-value of less than 0.05 doesn't mean the result is useful for anything! Enjoy.

From the Archives: A Synopsis of Programming Languages

A departure from the usual data mining and predictive analytics posts...

I was looking at old articles I clipped from the 80s, and came across my favorite programming article from the days I used to program a lot (mostly C, some FORTRAN, sh, csh, tcsh). This one from the C Advisor by Ken Arnold I found funny then, and still do now. I don't know where these are archived, so I'll just quote an excerpt here:

C advisor article by Ken Arnold from years and years ago quoting Richard Curtis

• FORTRAN was like the fifties: It's rigid and procedural, and doesn't even distinguish between cases. It's motto is "Do my thing".
• C is a real sixties language, because it just doesn't care. It doesn't type check, and it lets you get into as much trouble as you can--you own your own life. C's motto: "Do your own thing".
• Pascal is the seventies. It tries to seize control of the wild and woolly sixties, without getting too restrictive. It thus ends up pleasing no one. It's full of self-justification and self-importance--going from C to Pascal is like going from Janis Joplin to Donna Summer. It is smooth and flashy and useless for major work--truly the John Travolta of programming languages. The Pascal motto is: "Do your thing my way".
• ADA is the eighties. There is no overarching philosophy; everything is possible, but there is no ethical compass to tell you what ought to be done. (Actually, I know of two things you can't do in ADA, but I'm not telling for fear they'll be added.) It reflects the eighties notion of freedom, which is that you are free to do anything, as long as you do it the way the government wants you to--that is, in ADA. It's credo: "Do anything anyway you want".