Monday, April 25, 2016

Tracking Model Performance Over Time

Context

Most introductory data mining texts include substantial coverage of model testing. Various methods of assessing true model performance (holdout testing, k-fold cross validation, etc.) are usually explained, perhaps with some important variants, such as stratification of the testing samples.

Generally, all of this exposition is aimed at in-time analysis: Model development data may span multiple time periods, but the testing is more or less blind to this: all periods are treated as fair game and mixed together. This is fine for model development. Once predictive models are deployed, however, it is desirable to continue testing to track model performance over time. Models which degrade over time need to be adjusted or replaced.


Subtleties of Testing Over Time

Nearly all production model evaluation is performed with new out-of-time data. As new periods of observed outcomes become available, they are used to calculate running performance measures. As far it goes, focusing on the actual performance metric makes sense. In my experience, though, some clients become distracted by movement in the independent variables or in the predicted or actual outcome distributions, in isolation. It is important to understand the dynamic of these changes to fully understand model performance over time.

For the sake of a thought experiment, consider a very simply problem with one independent variable, and one target variable, both real numbers. Historically, the distribution of each of these variables has been confined to specific ranges. A predictive model has been constructed as a linear regression which attempts to anticipate the target variable, using only the input of the single independent variable (and a constant). Assume that errors observed in the development data have been small and otherwise unremarkable (they are distributed normally, their magnitude is relatively constant across the range of the independent variable, there is no obvious pattern to them and so forth).

Once this model is deployed, it is executed on all future cases drawn from the relevant statistical universe, and predictions are saved for further analysis. Likewise, actual outcomes are recorded as they become available. At the conclusion of each future time period, model performance within that period is examined.

Consider the simplest change to well-developed model: the distribution of the independent variable remains the same, but the actual outcomes begin to depart the regression line. Any number of changes could be taking place in the output distribution, but the predicted distribution (the regression line) cannot move since it is entirely defined by the independent variable, which in this case is stable. By definition, model performance is degrading. This circumstance is easy to diagnose: the dynamic linking the target and independent variables is changing, hence a new model is necessary to restore performance.

What happens, though, when the independent variable begins to migrate? There are two possible effects (in reality, some combination of these extremes is likely): 1. The distribution of actual outcomes will either shift to appropriately match the change ("the dots march along the regression line"), or 2. The distribution of actual outcomes does not shift to match the change. In the first case, the model continues to correctly identify the relationship between the target and the independent variable, and model performance will more-or-less endure. In the second case, reality begins to wander from the model and performance deteriorates. Notice that, in the second case, the actual outcome distribution may or may not change noticeably- either way, the model no longer correctly anticipates reality and needs to be updated.


Conclusion

The example used here was deliberately chosen to be simple, for illustrations' sake. Qualitatively, though, the same basic behaviors are exhibited by much more complex models. Models featuring multiple independent variables or employing complex transformations (neural networks, decision trees, etc.) obey the same fundamental dynamic. Given the sensitivity of nonlinear models to each of their independent variables, a migration in even one of them may provoke the changes described above. Consideration of the components of this interplay in isolation only serves to confuse: Changes over time can only be understood as part of the larger whole.

43 comments:

Anonymous said...

I think it makes perfect sense. Model training should be a on-going progress in a rapidly-changing world. Just like how humans learn, we learn new things everyday and that's how we progress. What's it gonna be like if we decide to learn all the things we need for maybe 22 years, and once we graduate from college, we refuse to learn anything new and always make decisions based on the things we learned in the first 22 years? We'll probably fail in life miserably!
However, it's a non-trivial task for models to continue learning over time. Training excessively could make the model overfit and increase its bias. I think a good approach could be that once we detect performance drop and change of variables in the real world, we could create a new model which takes the old model's parameters and learned attributes into account.

Dexus Media said...

nice article..its amazing...If you Are looking Best Digital Marketing Company in jaipur,
SEO Company in jaipur,
SEO services in jaipur,
website designer in jaipur

Coepd BA Trainings said...

We at Coepd declared Data Science Internship Programs (Self sponsored) for professionals who want to have hands on experience. We are providing this program in alliance with IT Companies in COEPD Hyderabad premises. This program is dedicated to our unwavering participants predominantly acknowledging and appreciating the fact that they are on the path of making a career in Data Science discipline. This internship is designed to ensure that in addition to gaining the requisite theoretical knowledge, the readers gain sufficient hands-on practice and practical know-how to master the nitty-gritty of the Data Science profession. More than a training institute, COEPD today stands differentiated as a mission to help you "Build your dream career" - COEPD way.

http://www.coepd.com/AnalyticsInternship.html

Coepd said...

We at COEPD provides finest Data Science and R-Language courses in Hyderabad. Your search to learn Data Science ends here at COEPD. Here, we are an established training institute who have trained more than 10,000 participants in all streams. We will help you to convert your passion to learn into an enriched learning process. We will accelerate your career in data science by mastering concepts of Data Management, Statistics, Machine Learning and Big Data.

http://www.coepd.com/AnalyticsTraining.html

sumathi s said...

It has been simply incredibly generous with you to provide openly what exactly many individuals would’ve marketed for an eBook to end up making some cash for their end, primarily given that you could have tried it in the event you wanted.
fire and safety courses in chennai

ibss said...

Great article with excellent idea!Thank you for such a valuable article
Web design company in chennai
Web development company in chennai

stella murugesan said...

I was very interested in the article , it’s quite inspiring I should admit. I like visiting your site since I always come across interesting articles like this one. Keep sharing! Regards. Read more about Advanced Analytics

pakescorts646 said...


We are Provide you well-mannered and delightful Islamabad Escorts Females Who Belong to the Upper Strata Society they are trained to pleasure a man the exact way he wants her to without any Problem Best Escorts Services in Islamabad offer you to Spend a Quality time to get mentally relaxation Call us for any kind of information about our Services.

Ruby Gracie said...

I was very interested in the article, it’s quite inspiring I should admit. I like visiting your site since I always come across interesting articles like this one. Keep sharing! Regards. Read more about Big data Services

Dharani M said...

Good information thank you sharing this information
data science training in Marathahalli

best data science courses in Marathahalli

data science institute in Marathahalli

data science certification Marathahalli

data analytics training in Marathahalli

data science training institute in Marathahalli

asha said...

Nice Post...... Thanks for sharing this post
data science training in bangalore

best data science courses in bangalore

data science institute in bangalore

data science certification bangalore

data analytics training in bangalore

data science training institute in bangalore

mounika said...

Nice post..

data science training in BTM

best data science courses in BTM

data science institute in BTM

data science certification BTM

data analytics training in BTM

data science training institute in BTM

kritika mytectra said...


myTectra the Market Leader in Machine Learning Training in Bangalore
myTectra offers Machine Learning Training in Bangalore using Class Room. myTectra offers Live Online Machine Learning Training Globally. Read More

Unknown said...

Let me help you find the best Digital Transformation Software .

jenifer irene said...

It was really an interesting blog, Thank you for providing unknown facts.
Aviation Courses in Chennai
Air Hostess Training Institute in Chennai
airport courses in Chennai
airport ground staff training courses in Chennai
medical coding course in Chennai
fashion technology courses in Chennai
Interior design courses in Chennai

NettechIndia said...

this blog provided a helpful information.I hope that you will post more updates like this.
python training in Mumbai

nivedhitha reddy said...

very nice article Leading data science training in ameerpet

Ogen Infosystem said...

Thank you so much for sharing this informative blog with us. Visit Ogen Infosystem for Website Designing Services.
Website Designing Company in Delhi

Mobile App Development Company said...

Thanks for sharing such a great blog... I am impressed with you taking time to post a nice info.
Mobile App Development Company

sachindigitalplanner said...

Rice Bags Manufacturers
Pouch Manufacturers
wall putty bag manufacturers

sachindigitalplanner said...

we have provide the best ppc service.
ppc services in gurgaon
website designing company in Gurgaon
PPC company in Noida
PPC Company in Delhi

sachindigitalplanner said...

we have provide the best fridge repair service.
fridge repair in faridabad
LG Fridge Repair in Faridabad
Videocon Fridge Repair in Faridabad
Whirlpool Fridge Repair in Faridabad
LG Refrigerator Repair In Faridabad
Washing Machine Repair Center in Noida

sachindigitalplanner said...

Bali Honeymoon Packages From Delhi
Bali Honeymoon Packages From Chennai
Hong Kong Packages From Delhi
Europe Packages from Delhi
Bali Honeymoon Packages From Bangalore
Bali Honeymoon Packages From Mumbai
Maldives Honeymoon Packages From Bangalore

nivedhitha reddy said...

Great information Top data science institute in ameerpet

hair juice accelerator said...

power testo blast - The first consideration is why you have losing a persons body weight. A lot of people diet strategy strategy and get thinner for a wide range of different aspects, which are all valid. For example to become fitter and healthier, for a unique event such as a daughter's wedding, or for your current health and fitness and wellness such as to help management suffering from diabetes issues etc.
https://newsletterforhealth.com/power-testo-blast/

keto ultra said...

cerisea medica rWhen one's extra fat get flushed out of your system, you can find yourself facing circumstances of having diarrhoea soon after meals. Some items execute differently and cause constipation. There are some that describe themselves detoxification agents. This utilizes a belief that weight-loss occurs when facing excess bodyweight.
https://newsletterforhealth.com/cerisea-medica/

test troxin said...

Beta keto About Milk products, Weight You want to look your best, but need a little help dropping bodyweight, so you want to get an awesome rapid weight-loss tactics that will help you to acquire your goal. These suggestions are all tested and tried and are the best for the goals that you have set for yourself.
https://newsletterforhealth.com/beta-keto/

test troxin said...

Keto fast enough locks with which to cover the reduce position , all the better. If you are still uncertain then contact a trichologist and seek advice before doing anything. In 1999 I created an appointment to see my physician because I had discovered a couple of places at returning of my go right on the locks line,
https://newsletterforhealth/keto-fast/

energy biotics said...

reviva brain to progressively expand progressively. It also impacts the sufferer’s claws, giving them a rough, ridged or inadequate overall look. The actual cause is still unidentified, although current concepts include of an A relatively minimal auto-immune illness, pressure or advise a got basis. If the locks loss injury usually
https://newsletterforhealth.com/reviva-brain/

testo drive 365 said...

testo edge ex
guidance, particularly if you have injuries or other limitation, see a professional!* If you think ok starting up something on your own, let me put out a simple plug for one of my favorite online workout websites, Fitness Blender.com. For approachable, achievable and FUN workouts, this site is the best. Run by a
https://newsletterforhealth.com/testo-edge/

erectify ultra said...

maxx power libido children), then learn to separate yourself momentarily from the stressor. If you think like exploding from pressure all time, take a five-minute walk outside or get a quiet room and do something you like. Learning to temporarily separate yourself from an unavoidable stressor can do wonders. sometimes (e.g. looking after yourself! Step physiological
https://newsletterforhealth.com/maxx-power-libido/

ropaxint said...

keto blast
he Mind-Body Connection in Body weight Loss The mind-body connection may sound like “magic and voodoo,” but it actually makes a lot of sense, and even health proper care science now emphasizes the need to harmonize the ideas and the whole personal whole body to boost one’s health and fitness and
https://topwellnessblog.com/keto-blast/

shimla said...

Tinnitus 911 attempting to start with dropping bodyweight goals. How can you stop pressure before it stops you? Here are some steps to help you with pressure management: Step 1: Ask yourself, what is causing your stress? Be really honest with yourself! Step 2: If the stressor is a issue you have at your workplace or at
https://newsletterforhealth.com/tinnitus-911/

creative web solution said...

Get The Best Repairing Service of office chairs shockers of your sofa or billiards and more other furniturea at best price.For more info visit our site given below :-

Chair Repairing in nawada
Shocker Repairing in dwarka
Sofa Repairing in delhi ncr
Furniture Repairing in janakpuri
Billiards Repairing in nehru place
chair repair in dwarka
sofa repair in south delhi
shocker repair in nawada
furniture repair in nawada

creative web solution said...

Are you looking for a maid for your home to care your baby,patient care taker, cook service or a japa maid for your pregnent wife we are allso providing maid to take care of your old parents.we are the best and cheapest service provider in delhi for more info visit our site and get all info.

maid service provider in South Delhi
maid service provider in Dwarka
maid service provider in Gurgaon
maid service provider in Paschim Vihar
cook service provider in Paschim Vihar
cook service provider in Dwarka
cook service provider in south Delhi
baby care service provider in Delhi NCR
baby care service provider in Gurgaon
baby care service provider in Dwarka
baby service provider in south Delhi
servant service provider in Delhi NCR
servant service provider in Paschim Vihar
servant Service provider in South Delhi
japa maid service in Paschim Vihar
japa maid service in Delhi NCR
japa maid service in Dwarka
japa maid service in south Delhi
patient care service in Paschim Vihar
patient care service in Delhi NCR
patient care service in Dwarka
Patient care service in south Delhi

creative web solution said...

We are the best Construction contractors in grater noida if you are planing for any construction in noida contact us and get best offers and all details

civil construction  contractor in Greater Noida
residential construction contractor in  Greater Noida
industrial construction conractors  in Greater Noida
building construction contractors in  Greater Noida
home renovation works in Greater Noida
painting work service in Greater Noida
wooden work service in Greater Noida
modular kitchen contractor in Greater Noida
almiraha contrcution contractor in  Greater Noida

creative web solution said...

such a nice post thanks for sharing this with us really so impressible and attractive post
are you searching for a caterers service provider in Delhi or near you then contact us and get all info and also get best offers and off on pre booking

caterers services sector 29 gurgaon
caterers services in west Delhi
event organizers rajouri garden
wedding planners in Punjabi bagh
party organizers in west Delhi
party organizers Dlf -phase-1
wedding planners Dlf phase-1
wedding planners Dlf phase-2
event organizers  Dlf phase-3
caterers services Dlf phase-4
caterers services Dlf phase-5

creative web solution said...

Alleyaaircool is the one of the best home appliances repair canter in all over Delhi we deals in repairing window ac, Split ac , fridge , microwave, washing machine, water cooler, RO and more other home appliances in cheap rates

Window AC Repair in vaishali
Split AC Repair in indirapuram
Fridge Repair in kaushambi
Microwave Repair in patparganj
Washing Machine Repair in vasundhara
Water Cooler Repair in indirapuram
RO Service AMC in vasundhara
Any Cooling System in vaishali
Window AC Repair in indirapuram

creative web solution said...

Totalsolution is the one of the best home appliances repair canter in all over Delhi we deals in repairing window ac, Split ac , fridge , microwave, washing machine, water cooler, RO and more other home appliances in cheap rates

LCD, LED Repair in Janakpuri
LCD, LED Repair in Dwarka
LCD, LED Repair in Vikaspuri
LCD, LED Repair in Uttam Nagar
LCD, LED Repair in Paschim Vihar
LCD, LED Repair in Rohini
LCD, LED Repair in Punjabi Bagh
LCD, LED Repair in Delhi. & Delhi NCR
LCD, LED Repair in Delhi. & Delhi NCR
Washing Machine repair on your doorstep
Microwave repair on your doorstep

creative web solution said...

In This Summers get the best designer umbrellas for you or for your family members we allso deals in wedding umbrellas and in advertising umbrellas For more info visit links given bellow
UMBRELLA WHOLESALERS IN DELHI
FANCY UMBRELLA DEALERS
CORPORATE UMBRELLA MANUFACTURER
BEST CUSTOMIZED UMBRELLA
FOLDING UMBRELLA DISTRIBUTORS
DESIGNER UMBRELLA
GOLF UMBRELLA DEALERS/MANUFACTURERS
TOP MENS UMBRELLA
LADIES UMBRELLA DEALERS
WEDDING UMBRELLA DEALERS
BEST QUALITY UMBRELLA
BIG UMBRELLA
Top Umbrella Manufacturers in India
Umbrella Manufacturers in Mumbai
Umbrella Manufacturers in Delhi
Garden Umbrella Dealers
Garden Umbrella Manufacturers
PROMOTIONAL UMBRELLA DEALERS IN DELHI/MUMBAI
PROMOTIONAL UMBRELLA MANUFACTURERS IN DELHI / MUMBAI
ADVERTISING UMBRELLA MANUFACTURERS

creative web solution said...

Rihan electronics is one of the best repairing service provider all over india we are giving our service in many different different cities like Noida,Gazibad,Delhi,Delhi NCR

AC Repair in NOIDA
Refrigerator Repair Gaziabad
Refrigerator repair in NOIDA
washing machine repair in Delhi
LED Light Repair in Delhi NCR
plasma TV repair in Gaziyabad
LCD TV Repair in Delhi NCR
LED TV Repair in Delhi

creative web solution said...

We are the one of the top blue art pottery manufacturers in jaipur get contact us and get all informations in detail visit our site
blue pottery jaipur
blue pottery shop in jaipur
blue pottery manufacturers in jaipur
blue pottery market in jaipur
blue pottery work shop in jaipur
blue pottery
top blue pottery in jaipur
blue pottery wholesale in jaipur

patriot power greens said...

provexum hygiene to avoid sickness. Do keep a close eye out for symptoms of sickness and look for medical care proper care if discomfort, blood loss, redness, or pain persists beyond a reasonable period of time. The success and safety of an appearance art originates with the tattooist one chooses, so be sure they are reputable and have an impeccably clean space that they are working in. tattoo designs in the past, and
https://newsletterforhealth.com/provexum/