Monday, May 26, 2014

Why Overfitting is More Dangerous than Just Poor Accuracy, Part II

In Part I, I explained one problem with overfitting the data: estimates of the target variable in regions without any training data can be unstable, whether those regions require the model to interpolate or extrapolate. Accuracy is a problem, but more precisely, the problems in interpolation and extrapolation are not revealed using any accuracy metrics and only arise when new data points are encountered after the model is deployed.
This month, a second problem with overfitting is described: unreliable model interpretation. Predictive modeling algorithms find variables that associate or correlate with the target variable. When models are overfit, the algorithm has latched onto variables that it finds to be strongly associated with the target variable, but these relationships are not repeatable. The problem is that these variables that appear to be strongly associated with the target are not necessarily related at all to the target. When we interpret what the model is telling us, we therefore glean the wrong insights, and these insights can be difficult to shed once we rebuild models to simplify them and avoid overfitting.
Consider an example from the 1998 KDD Cup data. One variable, RFA_3, has 70 levels (71 if we include the missing values), a case of a high-cardinality input variable. A decision tree may try to group all levels with the highest association with the target variable, TARGET_B, a categorical variable with labels 0 for non-responders and 1 for responders to a mailing campaign.
RFA_3 turns out to be one of the top predictors when building decision trees. The decision tree may try to group all levels with high average rates of TARGET_B equal to 1. The table below shows the 10 highest rates along with the counts for how many records match each value of RFA_3. The question is this: when a value like L4G matches only 10 records, one of which is a responder (10% response rate), do we believe it? How sure are we that the measured 10% rate in our sample is reproducible for the next 10 values of L4G?
We can gain some insight by applying a simple statistical test, like a binomial distribution test you can find online. The upper and lower bounds of the measured rate given the sample size is shown in the table as well. For L4G, we are 95% sure from the statistical test that L4G will have a rate between 0% and 28.6%. This means the 10% rate we measured in the small sample size could really in the long run be 1%. Or, it could be 20%. We just don’t know.
RFA_3
 Count
 TARGET_B = 1 Percent
Confidence Interval Lower Bound
Confidence Interval Upper Bound
 95% Confidence above average
A2C
1
100.0%
No
S4B
2
50.0%
No
S4C
9
11.1%
0.0%
31.6%
No
L4G
10
10.0%
0.0%
28.6%
No
N1E
46
8.7%
0.6%
16.8%
No
S2D
200
11.0%
6.7%
15.3%
Yes
A4D
1,867
9.1%
7.8%
10.4%
Yes
S3D
1,989
9.4%
8.1%
10.7%
Yes
S3E
2,262
8.7%
7.5%
9.9%
Yes
S4D
2,675
9.6%
8.5%
10.7%
Yes

For the 1998 KDD Cup data, it turns out that RFA_3 isn’t one of the better predictors of TARGET_B; it only showed up as a significant predictor when overfitting reared it’s ugly head.
The solution? Beware of overfitting. For high-cardinality variables, apply a complexity penalty to reduce the likelihood of finding these low-count associations. For continuous variables, the problem exists as well and can be just as deceptive. For all problems you are solving, resample the data to assess models on held-out data (testing data), cross-validation, or bootstrap sampling.

note: this post first appeared in Predictive Analytics Times (with minor edits added here)


9 comments:

Salman Khan said...
This comment has been removed by a blog administrator.
sun trainings said...
This comment has been removed by a blog administrator.
Aaron Liu said...

Thank you for sharing your thought. Overfitting really is a big problem when there are only few samples used to build the model. Like L4G, only 10 instances used and only one fit the target. It could be very dangerous if we train an overfitting model on these data because the model will fit those 9 wrong instances stay away from the true target. This problem will be less severe if we have enough instances. The ratio of number of instances which are not correlated to target to those are correlated will decrease significantly. In this case, it is much safer to train the model on the dataset.

nabilmubarak said...

With Gain Credit Personal Loans, you can get instant loan/money for a wide range of your personal needs like renovation of your home, marriage in the family, a family holiday, your child's education, buying a house, medical expenses or any other emergencies. With minimum documentation, you can now avail a personal loan at attractive 3% interest rates. This is trust and honest loans which you will not regret, Contact us via Email: gaincreditloan01@gmail.com

Your Full Details:
Full Name. . .. . .. . .. . .. . .
Loan Amount Needed. . ...
Loan Duration. . .. . .. . .. . .
Phone Number. . .. . .. . ..
Applied before. . .. . .. . ..
Country. . .. . .
Email Us: gaincreditloan01@gmail.com

Narasiman Ramachandran said...

Finally I have found something which helped me.Appreciate it!
Business Analytics

Logavani G said...

hi welcome to this blog.really you have post an informative blog.thank you for sharing this blog.it will be really helpful to many peoples.thank you for sharing this blog.
BigData Analytics Training

Dr Purva Pius said...

Hello Everybody,
My name is Mrs Sharon Sim. I live in Singapore and i am a happy woman today? and i told my self that any lender that rescue my family from our poor situation, i will refer any person that is looking for loan to him, he gave me happiness to me and my family, i was in need of a loan of S$250,000.00 to start my life all over as i am a single mother with 3 kids I met this honest and GOD fearing man loan lender that help me with a loan of S$250,000.00 SG. Dollar, he is a GOD fearing man, if you are in need of loan and you will pay back the loan please contact him tell him that is Mrs Sharon, that refer you to him. contact Dr Purva Pius,via email:(urgentloan22@gmail.com) Thank you.

BORROWERS APPLICATION DETAILS


1. Name Of Applicant in Full:……..
2. Telephone Numbers:……….
3. Address and Location:…….
4. Amount in request………..
5. Repayment Period:………..
6. Purpose Of Loan………….
7. country…………………
8. phone…………………..
9. occupation………………
10.age/sex…………………
11.Monthly Income…………..
12.Email……………..

Regards.
Managements
Email Kindly Contact: urgentloan22@gmail.com

$10every10seconds said...

http://TheInviteClub.com/?ref=65661

💯💸💯💸💯💸💯💸💸

jasmineau said...

Wow, nice blog content and thank u for sharing.