6 Reasons You Hired the Wrong Data Miner

As is in any discipline, talent within data mining community varies greatly.  Generally, business people and others who hire and manage technical specialists like data miners are not themselves technical experts.  This makes it difficult to evaluate the performance of data miners, so this posting is a short list of possible deficiencies in a data miner's performance.  Hopefully, this will spare some heartache in the coming year.  Merry Christmas!


1. The data miner has little or no programming skill.

Most work environments require someone to extract and prepare the data.  The more of this process which the data miner can accomplish, the less her dependence on others.  Even in ideal situations with prepared analytical data tables, the data miner who can program can wring more from the data than her counterpart who cannot (think: data transformations, re-coding, etc.).  Likewise, when her predictive model is to be deployed in a production system, it helps if the data miner can provide code as near to finished as possible.


2. The data miner is unable to communicate effectively with non-data miners.

Life is not all statistics: Data mining results must be communicated to colleagues with little or no background in math.  If other people do not understand the analysis, they will not appreciate its significance and are unlikely to act on it.  The data miner who can express himself clearly to a variety of audiences (internal customers, management, regulators, the press, etc.) is of greater value to the organization than his counterpart who cannot.  The data miner should should receive questions eagerly.


3. The data miner never does anything new.

If the data miner always approaches new problems with the same solution, something is wrong.  She should be, at least occasionally, suggesting new techniques or ways of looking at problems.  This does not require that new ideas be fancy: Much useful work can be done with basic summary statistics.  It is the way they are applied that matters.


4. The data miner cannot explain what they've done.

Data mining is a subtle craft: there are many pitfalls and important aspects of statistics and probability are counter-intuitive.  Nonetheless, the data miner who cannot provide at least a glimpse into the specifics of what they've done and why, is not doing all he might for the organization.  Managers want to understand why so many observations are needed for analysis (after all, they pay for those observations), and the data miner should be able to provide some justification for his decisions.


5. The data miner does not establish the practical benefit of his work.

A data miner who cannot connect the numbers to reality is working in a vacuum and is not helping her manager (team, company, etc.) to assess or utilize her work product.  Likewise, there's a good chance that she is pursuing technical targets rather than practical ones.  Improving p-values, accuracy, AUC, etc. may or may not improve profit (retention, market share, etc.).


6. The data miner never challenges you.

The data miner has a unique view of the organization and its environment.  The data miner works on a landscape of data which few of his coworkers ever see, and he is less likely to be blinded by industry prejudices.  It is improbable that he will agree with his colleagues 100% of the time.  If the data miner never challenges assumptions (business practices, conclusions, etc.), then something is wrong.

Why Predictive Modelers Should be Suspicious of Statistical Tests (or why the Redskin Rule fools us)

Well, the danger is really not the statistical test per se, it the interpretation of the statistical test.

Yesterday I tweeted (@deanabb) this fun factoid: "Redskins predict Romney wins POTUS #overfit. if Redskins lose home game before election => challenger wins (17/18) http://www.usatoday.com/story/gameon/2012/11/04/nfl-redskins-rule-romney/1681023/" I frankly had never heard of this "rule" before and found it quite striking. It even has its own Wikipedia page (http://en.wikipedia.org/wiki/Redskins_Rule).

For those of us in the predictive analytics or data mining community, and those of us who use statistical tests to help out interpreting small data, 17/18 we know is a hugely significant finding. This can frequently be good: statistical tests will help us gain intuition about value of relationships in data even when they aren't obvious.

In this case, an appropriate test is a chi-square test based on the two binary variables (1) did the Redskins win on the Sunday before the general election (call it the input or predictor variable) vs. (2) did the incumbent political party win the general election for President of the United States (POTUS).

According to the Redskins Rule, the answer is "yes" in 17 of 18 cases since 1940. Could this be by chance? If we apply the chi-square test to it, it sure does look significant! (chi-square = 14.4, p < 0.001). I like the decision tree representation of this that shows how significant it is (built using the Interactive CHAID tree in IBM Modeler on Redskin Rule data I put together here):

It's great data--9 Redskin wins, 9 Redskin losses, great chi-square statistic!

OK, so it's obvious that this is just another spurious correlation in the spirit of all of those fun examples in history, such as the superbowl winning conference predicting if the stock market would go up or down in the next year at a stunning 20 or 22 correct. It even was the subject of academic papers on the subject!

The broader question (and concern) for predictive modelers is this: how do we recognize when we have uncovered spurious correlations in the data that are merely spurious? This can happen especially when we don't have deep domain knowledge and therefore wouldn't necessarily identify variables or interactions as spurious. In examples such as the election or stock market predictions, no amount of "hold out" samples, cross-validation or bootstrap sampling would uncover the problem: it is in the data itself.

We need to think about this because inductive learning techniques search through hundreds, thousands, even millions of variables and combinations of variables. The phenomenon of "over searching" is a real danger with inductive algorithms as they search and search for patterns in the input space. Jensen and Cohen have a very nice and readable paper on this topic (PDF here). For trees, they recommend using the Bonferroni adjustment which does help penalize the combinatorics associated with splits. But our problem here goes far deeper than overfitting due to combinatorics.

Of course the root problem with all of these spurious correlations is small data. Even if we have lots of data, what I'll call here the "illusion of big data", some algorithms make decisions based on smaller populations, like decision trees, rule induction and nearest neighbor (i.e., algorithms that build bottom-up). Anytime decisions are made from populations of 15, 20, 30 or even 50 examples, there is a danger that our search through hundreds of variables will turn out a spurious relationship.

What do we do about this? First, make sure you have enough data so that these small-data effects don't bite you. This is why I strongly recommend doing data audits and looking for categorical variables that contain levels with at most dozens of examples--these are potential overfilling categories.

Second, don't hold strongly any patterns discovered in your data based on solely on the data, especially if they are based on relatively small sample sizes. These must be validated with domain experts. Decision trees are notorious for allowing splits deep in the trees that are "statistically significant" but dangerous nevertheless because of small data sizes.

Third, the gist of your models have to make sense. If they don't, put on your "Freakonomics" hat and dig in to understand why the patterns were detected by the models. In our Redskin Rule, clearly this doesn't make sense causally, but sometimes the pattern picked up by the algorithm is just a surrogate for a real relationship. Nevertheless, I'm still curious to see if the Redskin Rule will prove to be correct once again. This year it predicts a Romney win because the Redskins lost and therefore the incumbent party (D) by the rule should lose. UPDATE: by way of comparison...the chances of having 17/18 or 18/18 coin flips turn up heads (or tails--we're assuming a fair coin after all!) is 7 in 100,000 or 1 in 14,000. Put another way, if we examined 14K candidate variables unrelated to POTUS trends, the chances are that one of them would line up 17/18 or 18/18 of the time. Unusual? Yes. Impossible? No!

Data Preparation: Know Your Records!

Data preparation in data mining and predictive analytics (dare I also say Data Science?) rightfully focuses on how the fields in ones data should be represented so that modeling algorithms either will work properly or at least won't be misled by the data. These data preprocessing steps may involve filling missing values, reigning in the effects of outliers, transforming fields so they better comply with algorithm assumptions, binning, and much more.

In recent weeks I've been reminded how important it is to know your records. I've heard this described in many ways, four of which are:

the unit of analysis

the level of aggregation

what a record represents

unique description of a record

For example, does each record represent a customer? If so, over their entire history or over a time period of interest? In web analytics, the time period of interest may be a single session, which if it is true, means that an individual customer may be in the modeling data multiple times as if each visit or session is an independent event.

Where this especially matters is when disparate data sources are combined. If one is joining a table of customerID/Session data with another table with each record representing a customerID, there's no problem. But if the second table represents customerID/store visit data, there will obviously be a many-to-many join resulting in a big mess.

This is probably obvious to most readers of this blog. What isn't always obvious is when our assumptions about the data result in unexpected results. What if we expect the unit of analysis to be customerID/Session but there are duplicates in the data? Or what if we had assumed customerID/Session data but it was in actuality customerID/Day data (where ones customers typically have one session per day, but could have a dozen)?

The answer is just like we need to perform a data audit to identify potential problems with fields in the data, we need to perform record audits to uncover unexpected record-level anomalies. We've all had those data sources where the DBA swears up and down that there are no dups in the data, but when we group by customerID/Session, we find 1000 dups.

So before the joins and after joins, we need to do those group by operations to find examples with unexpected numbers of matches.

In conclusion: know what your records are supposed to represent, and verify verify verify. Otherwise, your models (who have no common sense) will exploit these issues in undesirable ways!

Budgeting Time on a Modeling Project

Within the time allotted for any empirical modeling project, the analyst must decide how to allocate time for various aspects of the process.  As is the case with any finite resource, more time spent on this means less time spent on that.  I suspect that many modelers enjoy the actual modeling part of the job most.  It is easy to try "one more" algorithm: Already tried logistic regression and a neural network?  Try CART next.

Of course, more time spent on the modeling part of this means less time spent on other things.  An important consideration for optimizing model performance, then, is: Which tasks deserve more time, and which less?

Experimenting with modeling algorithms at the end of a project will no doubt produce some improvements, and it is not argued here that such efforts be dropped.  However, work done earlier in the project establishes an upper limit on model performance.  I suggest emphasizing data clean-up (especially missing value imputation) and creative design of new features (ratios of raw features, etc.) as being much more likely to make the model's job easier and produce better performance.

Consider how difficult it is for a simple 2-input model to discern "healthy" versus "unhealthy" when provided the input variables height and weight alone.  Such a model must establish a dividing line between healthy and unhealthy weights separately for each height.  When the analyst uses instead the ratio of weight to height, this becomes much simpler.  Note that the commonly used BMI (body mass index) is slightly more complicated than this, and would likely perform even better.  Crossing categorical variables is another way to simplify the problem for the model.  Though we deal with a process we call "machine learning", is is a pragmatic matter to make the job as easy as possible for the machine.

The same is true for handling missing values.  Simple global substitution using the non-missing mean or median is a start, but think about the spike that creates in the variable's distribution.  Doing this over multiple variables creates a number of strange artifacts in the multivariate distribution.  Spending the time and energy to fill in those missing values in a smarter way (possibly by building a small model) cleans up the data dramatically for the downstream modeling process.

What do we call what we do?

I've called myself a data miner for about 15 years, and the field I was a part of as Data Mining (DM). Before then, I referred to what I did as "Pattern Recognition", "Machine Learning", "Statistical Modeling", or "Statistical Learning". In recent years, I've called what I do Predictive Analytics (PA) more often and even co-titled my blog with both Data Mining and Predictive Analytics. That stated, I don't have a good noun to go along with PA. A "predictive analytist" (as if I myself were a "predictor")? A "predictive analyzer"? I often call someone who does PA a Predictive Analytics Professional. But the according to google, the trending on data mining is down. Pattern recognition? Down. Machine Learning? Flat or slightly up. Only Predictive Analytics and it's closely-related sibling, Business Analytics, are up. Even the much-touted Data Science has been relatively flat, though has been spiking Q4 the past few years.

Data Mining

Pattern Recognition

Machine Learning

Predictive Analytics

Business Analytics

The big winner? Big Data of course! It has exploded this year. Will that trend continue? It's hard to believe it will continue, but this wave has grown and it seems that every conference related to analytics or databases is touting "big data".

Big Data

 

Data Science

I have no plans of calling what I do "big data" or "data science". The former term will pass when data gets bigger than big data. The latter may or may not stick, but seems to resonate more with theoreticians and leading-edge types than with practitioners. For now, I'll continue to call myself a data miner and what I do predictive analytics or data mining.

Choose Your Target Carefully

Every so often, an article or survey will appear stressing the importance of data preparation as an early step in the process of data mining.  One often-overlooked part of data preparation is to clearly define the problem, and, in particular, the target variable.  Often, a nominal definition of the target variable is given.

As an example, a common problem in banking is to predict future balances of a loan customer.  The current balance is a matter of record and a host of explanatory variables (previous payment history, delinquency history, etc.) are available for model construction.  It is easy to move forward with such a project without considering carefully whether the raw target variable is the best choice for the model to approximate.  It may be, for instance, that it is easier to predict the logarithm of balance, due to a strongly skewed distribution.  Or, it might be that it is easier to predict the ratio of future balances to the current balance.  These two alternatives result in models whose output are easily transformed back into the original terms (by exponentiation or multiplication by the current balance, respectively).  More sophisticated targets may be designed to stabilize other aspects of the behavior being studied, and certain other loose ends may be cleaned up as well, for instance when the minimum or maximum target values are constrained.

When considering various possible targets, it helps to keep in mind that the idea is to stabilize behavior, so that as many observations as possible align in the solution space.  If retail sales include a regular variation, such as by day of the week or month of the year, then that might be a good candidate for normalization: Possibly we want to model retail sales divided by the average for that day of the week, or retail sales divided by a trailing average for that day of the week for the past 4 weeks.  Some problems lend themselves to decomposition, such as profit being modeled by predicting revenue and cost separately.  One challenge to using multiple models in series this way is that their (presumably independent) errors will compound.

Experience indicates that it is difficult in practice to tell which technique will work best in any given situation without experimenting, but performance gains are potentially quite high for making this sort of effort.

The Data is Free and Computing is Cheap, but Imagination is Dear

Recently published research, What Makes Paris Look like Paris?, attempts to classify images of street scenes according to their city of origin.  This is a fairly typical supervised machine learning project, but the source of the data is of interest.  The authors obtained a large number of Google Street View images, along with the names of the cities they came from.  Increasingly, large volumes of interesting data are being made available via the Internet, free of charge or at little cost.  Indeed, I published an article about classifying individual pixels within images as "foliage" or "not foliage", using information I obtained using on-line searches for things like "grass", "leaves", "forest" and so forth.

A bewildering array of data have been put on the Internet.  Much of this data is what you'd expect: financial quotes, government statistics, weather measurements and the like- large tables of numeric information.  However, there is a great deal of other information: 24/7 Web cam feeds which are live for years, news reports, social media spew and so on.  Additionally, much of the data for which people once charged serious bucks is now free or rather inexpensive.  Already, many firms augment the data they've paid for with free databases on the Web.  An enormous opportunity is opening up for creative data miners to consume and profit from large, often non-traditional, non-numeric data which are freely available to all, but (so far) creatively analyzed by few.

Predicting Crime

Applying inferential statistics to criminology is not new, but it appears that the market has been maturing.  See, for instance, a recent article, "Police using ‘predictive analytics’ to prevent crimes before they happen", published by Agence France-Presse on The Raw Story (Jul-29-2012).

Setting aside obvious civil liberties questions, consider the application of this technology.  My suspicion is that targeting police efforts by geographic locale and day-of-week/time-of-day using this approach will decrease the overall level of crime, but by how much is not clear. This is typical of problems faced by businesses: It is not enough to predict what we already know, nor is it enough to trot out glowing but artificial technical measures of performance.  Knowledge that real improvement has occurred requires more.  For instance, at least some effect of police effort on the street does not decrease crime, but merely moves it to new locations.

Were I mayor of a small town approached by the vendor of such a solution, I'd want to see some sort of experimental design which made apples-to-apples comparison between our best estimates of what happens with the new tool, and what happens without it.  Only once this firm measure of benefit has been obtained could one reasonably weigh it against the financial and political costs.

Where it began: John Elder and Dean Abbott at Barron Associates

I came across this photo today and couldn't resist. John Elder and I worked at the company Barron Associates, Inc. (BAI) in Charlottesville, VA in the 80s (John was employee #1). Hopefully you can identify John in the back right and me in the front right, though it will take some good pattern matching to do so!

The Founder and President of the company was Roger Barron. Both John and I were introduced to statistical learning methods at BAI and of course went on to careers in the field now known as Data Mining or Predictive Analytics (among other things). I write about my experience with BAI in the forthcoming book Journeys to Data Mining: Experiences from 15 Renowned Researchers, Ed. by Dr Mohamed Medhat Gaber, published by Springer. Authors in the book include John (thanks to John for recommending my inclusion in the book), Gregory Piatetsky-Shapiro, Mohammed J. Zaki, and of course several others.

The photo appeared as I was searching for descriptions of our field back in the 80s and was looking in particular for the Barron and Barron paper "Statistical Learning Networks: A Unifying View", where "Statistical Learning Networks" was the phrase of interesting, along with "Models", "Classifiers", and "Neural Networks". We used to refer to the field as "Pattern Recognition" and "Artificial Intelligence". It's interesting to note that pattern recognition on Wikipedia contains a list of "See Also" terms that includes the more modern terms such as data mining and predictive analytics.

I will post within a couple days on the pattern recognition terms of the day and how they are changing.

Predictive Analytics World Had the Target Story First

The New York Times Magazine article "How Companies Learn Your Secrets" by Charles Duhigg with the key descriptions of Target, pregnancy, predictive analytics (blogged on here and here) certainly generated a lot of buzz; if you are unable to see the NYTimes Magazine article, the Forbes summary is a good summary. However, few know that Eric Siegel booked Andy Pole for the October 2010 Predictive Analytics World conference as a keynote speaker. The full video of that talk is here.

In this talk, Mr. Pole discussed how Target was using Predictive Analytics including descriptions of using potential value models, coupon models, and...yes...predicting when a woman is due (if you aren't the patient type, it is at about 34:30 in the video). These models were very profitable at Target, adding an additional 30% to the number of woman suspected or known to be pregnant over those already known to be (via self-disclosure or baby registries). The fact that this went on for over a year after the Predictive Analytics World talk and before the fallout tells me that it didn't cause significant problems for Target prior to the attention brought to the subject related to the NYTimes article.

After watching the talk, what struck me most was that Target was applying a true "360 deg" customer view by linking guests through store visits, web, email, and mobile interactions. In addition, close attention was paid to linking the interactions so that coupons made sense: they didn't print coupons to those who had just purchased items they score high for couponing, and they identify which mediums don't generate responses and stop using those channels.

I suspect what Target is doing is no different than most retailers, but this talk was an interesting glimpse into how much they value the different channels and try to find ways to provide customers with what they are most interested in, and suppress what they are not interested in.