Tuesday, June 07, 2005

Beware of Being Fooled with Model Performance

Interpreting model performance is a minefield. If one wants model performance to be as good as possible, it is critical to define exactly what "good" means. How does one measure "goodness"? The easiest way to communicate performance is with a single-valued score, such as percent correct classification or R-squared. However, it is precisely this simplification of a complex idea the model is predicting to a single number that can cause one to be fooled. A simple example follows.

Let's assume that a non-profit organization wants a model built that predicts the propensity of individuals to send donations, and that this model has 80+% classification accuracy, even on a test set. Furthermore, assume that the two indicators "Recent Donation Amount" (X1) and "Average Donation Amount" (X2) are two of the top predictors in the model. The figure at the left shows what a Support Vector Machine model did with this data. Even with the good accuracy, there is something disturbing about the model that isn't clear unless one sees a picture: the model isn't finding ranges of average and recent donation amounts that are associated with donors, but rather it is finding islands of donors. The second model (on the right) provided corrective measures to smooth the model, and it much more pleasing. It is saying (roughly) that when someone donates between about $10-$50 on average (X2), they are more likely to respond. It is smooth and there are no pockets of isolated donation amounts, making this model much more believable, even though some accuracy was lost in the process.







4 comments:

Neha Singh said...

Escort Service in Gurgaon
Do you need a Call Girls in Gurgaon to give you the real pleasure of your life, well, if you revert in affirmative, in that case, you are at the right place. We are a stupendous track record of providing the best girls that would completely take the evening of yours by storm, so to avail them all you need to do is just call and we will provide the best at the right price.

Dipika Padukon said...

Call Girls in Gurgaon
Are you looking for escort service in Gurgaon, we provide the best that would completely transform your experience. Our escorts are trained meticulously to deliver best service that you are looking for.

Roshni Kumari said...

Female Escort in Gurgaon
If after a busy business tour, you are looking to ignite the evening then our Call Girls in Sohna Road are there to help you in the best way. We provide the service that would completely take away your work stress and revive you from within. So, why wait when you can get the best feeling of love at cost-effective prices. The escort service that we deliver in Sohna Road is among the best that you can get at the rates that we offer. We provide the best girls that would completely satiate you from each and every angle. So, visit us today and take your experience to a new level.

Annu Devi said...

Call Girls in Mahipalpur
If you have come from the West and you are captivated by the Asian beauty, in that case, we make sure that you get the best Call Girls in Aerocity to help you taste the Asian flavor in the best way. At Aerocity, we make sure that you are getting one heck of an experience with our girls. So, no matter whether it is official meeting or bachelor party, we are there for all your needs.