Could these be great days for data miners?

In a recent article on cfo.com, Data Mining in the Meltdown: the Last, Best Hope? the author describes how data quality is the key to future success of businesses. But data quality by itself is not enough,

Of course, data quality matters little if a company is focusing on the wrong measures. The best companies adopt a customer-oriented definition of data quality and recognize that all items of data are not created equal...

In other words, the business objective phase (in the CRISP-DM way of viewing things) is critical. I would add that building models that are assessed in a manner commensurate with the business objective is every bit as important. If you build a series of regression models and take the one with the best R^2, you have very little idea from that metric whether or not the model will do anything productive. One must score and assess the model to reflect the business objective.

The author gets at this idea indirectly with this comment:

For every key performance indicator (KPI), for example, companies should be tracking a key risk indicator (KRI), Friend says. "You plan not just for results, but for contingencies. What happens if sales are down 20 percent?"

In other words, there may just be significant asymmetric costs to incorprate in the scoring of models. I'll be bringing this up at Predictive Analytics World this week; it is arguably one of the biggest mistakes made by modelers.

Can you learn data mining in undergraduate or graduate school?

I was recently asked by a former student from one of my data mining courses if a particular program was a good one to learn data mining (it happened to be this one, from NC State). It raises an interesting question: how much can data mining be learned from a book or a course?

Some of the best data miners I have met did not have any statistics course in their past, nor (for some) any higher level mathematics. For my part, I was a computational mathematics major undergrad, and applied math for my masters, but never took a stats course either (though I did take and TA a probability course). That stated, I always recommend in my courses that folks become familiar with basic statistics; one book I have recommended is linked in the book recommendations section--The Cartoon Guide to Statistics. Since I have never taken a college or graduate data mining course, I can't comment directly. My concern is that they are too theoretical (how the algorithms work) rather than practical (how to handle data problems, how to pose proper questions to be addressed by data mining, etc.).

I'm willing to be persuaded though, so if you have experience with good, practical data mining curricula, please let me know.

Predictive Analytics World

There is a new predictive analytics conference coming up Feb 18-19 in San Francisco called Predictive Analytics World. I'm very much looking forward to it in the hopes that it will appeal to the data mining / predictive analytics practitioner.

I'll be presenting a case study I worked on with TN Marketing using ensembles of logistic regression models. Also, I'll be on a panel discussion on Cross-Industry Challenges and Solutions in Predictive Analytics.

Hope to see some of you there!

Text Mining and Regular Expressions

I've been spending quite a lot of time in the bowels of a text mining project recently, mostly in the text/concept extraction phase. We're using the SPSS Text Mining tool for the work so far. (As a quick aside, the text mining book I've enjoyed reading the most in recent months is the Weiss, Indurkhya, Zhang, and Damerau)

The most difficult part of the project has been that all of the text is really customized lingo--a language of its own as presented in the notes sections of the documents we are reading. Therefore, we can't use the typical linguistic extraction techinques, and rather are relying heavily on regular expressions. That certainly takes me back a few years! I used to use regular expressions mostly in shell programming (Bourne, CShell, Korn Shell and later BASH).

I must say it has been very productive, though it also makes me appreciate language rules that don't exist in any consistent way with our notes. As I am able, I'll post on more specifics on this project.

Regarding books on regular expressions, I found the unix books weren't quite so good on this topic. However, the O'Reilly Mastering Regular Expressions book is quite good.

What is Predictive Analytics?

I just saw this link about the difference between BI and Predictive Analytics. This comes on the heels of a meeting I had with UCSD Extension folks, talking about predictive analytics and data mining in the context of teaching courses for professionals, and this topic came up: how is predictive analytics different from BI?

First, I'd like to applaud the author, Vladimir Stojanovski, for concluding there are differences, and for trying to get at what those differences are.

The article states that this:

To tie this all back to the question of BI vs. Predictive Analytics (PA), a metaphor I've heard used to describe the difference goes something like this: if BI is a look in the rearview mirror, predictive analytics is the view out the windshield.

In my experience, this is a common definition. Predictive Analytics and Data Mining are seen as predicting future events, whereas OLAP looks at past data.

While I'd love to jump on this bandwagon because it makes for a simple and compelling story, I cannot ride this one. And that's because both BI and PA look at historic data. PA isn't magic in coming up with predictions of the future. In fact, both BI and PA ultimately look at and use the same data (or variations of the same historic data). Both can predict the future, so long as the future is consistent with past, either in a static sense, or in a dynamic sense (by extrapolating past data into the future).

I think it is better to describe the difference in this way: BI reports on historical data based upon an analyst's perspective on which fields and statistics are interesting, whereas PA induces which fields, statistics and relationships are interesting from the data itself. I think it is the combinatorics, sifting, iterative nature of PA that gives it better predictive accuracy of the future (coupled with using business metrics to assess if the fields found truly are predictive or not).

So let's not oversell--what PA does is reason enough for it to be an integral part of any analytics or BI group.

What topics would you like to see covered at a KDD conference?

This is your chance to voice your opinion!

What topics, sessions, or tutorials would be most useful for you at a conference like KDD? Would a full industrial track be of interest, of are industries so diverse that we really need tracks to be narrowed to specific industries?

Please--practitioners only. I'm defining practitioners as those who get paid to develop models that are actually used in industry.

I'll kick it off with one idea:

Tutorials (1/2 day) geared toward the practitioner. This means that if techniques are described (such as social networking), there must be implementations of the algorithmic ideas available in competitive commercial software. As great as R and Matlab are, for example, relatively few practitioners are programmers that can take advantage of these kinds of frameworks.

I know there are tutorials at KDD every year. This year I didn't go because they were all on Sunday and I wasn't able to attend then, but would have wanted to go to the Text Mining tutorial as that is a topic that has become a significant part of my business over the past couple of years.

One last thought: I think one thing that may happen (understandably) is that topics that have been covered in years passed are not revisited. For those of us who live in the data mining world, it is far more interesting to continue to explore new ideas, especially those that build on ideas we have already explored in depth. However, as data mining increases in its use, we are bringing folks in who have not had that same benefit. For many, a tutorial on decision trees would be very useful and interesting (like the KDD 2001 tutoral--trees to my knowledge have not been revisited since except in the framework of ensembles in 2007).

Two Books of Interest

Recently, I have been reading two books which may be of interest to data miners, Statistical Rules of Thumb by Gerald Van Belle (ISBN-13: 978-0471402275) and Common Errors in Statistics (and How to Avoid Them), by by Phillip I. Good and James W. Hardin (ISBN-13: 978-0471794318). Both impart practical advice based on extensive experience and statistical rigor, yet avoid becoming hung up on academic issues.

While both are written from the point of view of traditional statisticians, they do suggest the use of some less traditional techniques, such as the bootstrap and robust regression. A wide range of topics is covered, such as sample size determination, hypothesis testing and treatment of missing values. Both books also include some material written for audiences working in specific fields, such as environmental science and epidemiology. Material in these two books will vary in applicability to data mining, given the traditional statistical focus on smaller data sets and parametric modeling.

I highly recommend both of them. Tables of contents can easily be found on-line, and an entire chapter of Statistical Rules of Thumb is available at: Chapter 2: Sample Size.

KDD 2008

It's hard to believe that KDD2008 was the first KDD I've attended in seven years. It was striking how much has changed in that time, and that was one of the primary reasons I attended this past year--to see for myself if the reports I've heard are true. Sure enough, they are.

These reports, primarily from colleagues in industry, were that KDD didn't have anything they could "take home and use". Many of these folks are analysts who are decidedly not academic, so I thought I had a sense for what they meant.

I found their reports hit the mark. Seven years ago I was able to find (1) significant numbers of industry personnel at the conference and (2) many talks that were accessible enough for non-academics to understand. This time around there were few industry practitioners I met who were not PhDs. That's not to say there weren't interesting talks. Two I didn't see in person, but read later were the Elkan paper on learning from positive and unlabelled examples and the Grossman paper on Data Clouds. Though-provoking both. The lunch talk by Trevor Hastie was very interesting in talking about regularization, but it was geared toward those who can digest his textbook (which is among the finest data mining / statistical learning texts out there).

Social networking was a key theme of the conference, and it was such a dominant force at the conference that it deserves a separate post.

Lastly, the decline in participation by the business community was nowhere more evident than in the vendors room--only a few data mining software vendors were there, which indicates to me that it isn't viewed as a place to increase sales: if I remember correctly, only Microsoft, Oracle, Statsoft, Salford Systems, and SAS were there. A quick look at the kdnuggets software survey shows who wasn't there.

So it seems that KDD has wandered from a business/academic mix to a more academic conference, which is, of course, the prerogative of the organizers. I'm still searching for a great conference for the data mining practitioner who has the level of understanding of data mining to read and absorb a book like the Witten/Frank machine learning book but desires a more practical approach to the subject.

What data mining software to buy?

This post (http://www.dmreview.com/issues/2007_46/10001040-1.html?portal=analytics) is an interesting example of the assessment of analytics software. The key paragraph is the conclusion where Mr. Raab states

Instead of a horserace between product features, this approach puts the focus where it should be: on value to your business. It recognizes that the value of a new tool depends on the other tools already available, and it forces evaluation teams to explicitly study the impact of different tools on different users. By creating a clearer picture of how each new tool will impact the way work actually gets done within the company, it leads to more realistic product assessments and ultimately to more productive selection choices.

I couldn't agree more. For the past 10 years, since the Elder and Abbott review of data mining software presented at KDD-98 (on my web site) I've tried to think of ways to summarize data mining software. The obvious way is by features, such as which algorithms a product has. The usability of a tool is another characteristic to add, as John, Philip Matkovsky and I wrote about in "An Evaluation of High-End Data Mining Tools for Fraud Detection". I've also described the different packages by the kind of interface (wizard, menu-driven, block-diagram, command line, etc.).

It's not easy to provide a summary in this multi-dimensional view of data mining tools. Sounds like an opportunity for predictive modeling!

What Makes a Data Mining Skeptic?

I just found this post expressing skepticism about data mining (I'll let go the comment about predictive analytics being the holy grail of data mining--not sure what this means).

The fascinating part for me was this paragraph:

Anyway. Lindy and I were a bit squirmy through the whole discussion. It seemed like so many hopes and dreams were being placed at the altar of the goddess Clementine... but I had to ask myself, could you REALLY get any more analysis out of it then you could get simply by asking your members what events they attend, plan to attend, ever attended, or might attend in the future, and why? Since when did we stop talking to our members about this stuff? A good internal marketing manager could give you all the answers you seek about which of your various audiences are likely to respond to which of your messages, who's going to engage with you, why and when, who's going to participate in which of your events, etcetera, and they would know these answers not through stats and charts (even if you ask for them) but through experience and listening.

It is interesting on several fronts. First, there is a strong emphasis on personal expertise and experience. But at the heart of the critique is apparently a belief that the data cannot reveal insights, or in other words, a data-driven approach doesn't give you any "analysis". Why would one believe this? (and I do not doubt the sincerity of the comment--I take it at face value).

One reason may be that this individual has never seen or experienced a predictive analytics solution. While this may be true, it also misses what I think is at the heart of the critique. There is a false dichotomy set up here between data analysis and individual expertise. Anyone who has built predictive models successfully knows that one usually must have both: expert knowledge and representative data (to build predictive models).

One reason for this is that while there are undoubtedly some individuals who can "give you all the answers you seek about which of your various audiences are likely to respond to which of your messages". But usually, this falls short for two reasons:
1) most individuals who have to deal with large quantities of data don't know as much they think they know, and related to this
2) it is difficult to impossible for anyone to sort through all the data with all of the permutations that exist.

Data mining usually doesn't tell us things that experts scratch their heads at in amazement. The usually confirm what one suspects (or one of many possible conclusions one may have suspected), but with a few unexpected twists.

So how can we persuade others that there is value in data mining? The first step is realizing there is value in the data.