Monday, May 22, 2017

What Programming do Predictive Modelers Need to Know?
Dean Abbott
(First published in Predictive Analytics Times, http://www.predictiveanalyticsworld.com/patimes/what-programming-do-predictive-modelers-need-to-know-0408152-2/5129/, 2014. Updated and edited here.)

In most lists of the most popular software for doing data analysis, statistics, and predictive modeling, the top software tools are Python and R—command line languages rather than GUI-based modeling packages. There are several reasons for this, perhaps most importantly that they are free, they are robust programming languages supported by a very broad user community, and they have extensive sets of algorithms.

One recent survey of software was published at r4stats.com (http://r4stats.com/articles/popularity/) and contains additional metrics not usually found in software comparisons, including scholarly articles that include software, google scholar hits, and job trends in addition to the more typical summaries by user-identified use of software such as with the Rexer Analytics surveys (http://www.rexeranalytics.com/Data-Miner-Survey-Results-2013.html) and polls on kdnuggets.com and reviews of software by technology research companies such as Gartner (http://pages.alteryx.com/GartnerMQAdvancedAnalyticsNowAvailable-T.html), Forrester (http://global.sap.com/campaign/na/usa/CRM-XU13-BIP-PATDWS/index.html?urlid=CRM-XU13-BIP-PATDWS), and Hurwitz & Associates (http://www.sas.com/content/dam/SAS/en_us/doc/analystreport/hurwitz-advanced-analytics-107212.pdf). Thanks to r4stats for providing these links in their article.

For those interested in getting a job in analytics, the article provides very useful rankings of software by the number of job postings, led by Java, SAS, Python, C/C++/C#, R, SPSS, and Matlab. They also provide a few examples of the trending in the job postings of the tools over the past 7 years—important trending to consider as well. This reveals for example a nearly identical increase in Python and R compared with a decrease in SAS over the past few years (SAS is still #2 overall in job postings though because of the huge SAS install base).

The user interface that appears to have won the day in commercial software for predictive modeling is the workflow-style interface where a user connects icons that represent functions or tasks into a flow of functions. This kind of interface has been in use for decades, and one that I was first introduce to in the software package Khoros / Cantata in the early 90s (http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.22.9854&rep=rep1&type=pdf). Clementine was an early commercial tool using this paradigm (now IBM SPSS Modeler), and now most tools, including those that have historically used drop-down “Windows-style” menus, are embracing a workflow interface. And it’s not just commercial tools that are built from the ground up using a workflow-style interface: many open source software like KNIME and RapidMiner embraced this style from their beginnings.

Even though workflow interfaces have won the day, there are several excellent software tools that still use a typical drop-down file menu interface for a variety of reasons: some legacy and some functional. I still use several of them myself.

There are several reasons I like the workflow interface. First, it is self-documenting, much like command-line interfaces. You see exactly what you did in the analysis, at least at a high level. To be fair, some of these nodes have considerable critical customization options set inside the nodes, but the function is self-evident nevertheless. Command line functions have the same issue: there are often numerous options one has to specify to call a function successfully.

Second, you can reuse the workflow easily. For example, if you want to run your validation data through the exact same data preparation steps you used in building your models, you merely connect a new data source to the workflow. Third, you can explain what you did to your manager very easily and visually without the manager needing to understand code.

Another way to think of the workflow interface is as a visual programming interface. You string together functional blocks from a list of functions (nodes) made available to you by the software. So whether you build an analysis in a visual workflow or a command line programming language, you still do the same thing: string together a sequence of commands to manipute and model the data. For example, you may want to load a csv file, replace missing values with the mean, transform your positively-skewed variables with a log transform, split you data into training and testing subsets, then build a decision tree. Each of these steps can be done with a node (visual programming) or a function (programming).

From this perspective, the biggest difference between visual programming and command line programming is that R and Python have a larger set of functionas available to you. From an algorithm standpoint, this difference is primarily manifested in obscure or new, cutting-edge algorithms. This is one important reason why most visual programming interface tools have added R and Python integration into their software, typically through a node that will run the external code within the workflow itself. The intent isn’t to replace the software, but to enhance it with functions not yet added to the software itself. This is especially the case with leading-edge algorithms that have support in R or Python already because of their ties with the academic community.

Personally, I used to create code regularly for building models, primarily in C and FORTRAN (3rd generation languages) though also in other scripting (4th generation) languages like unix shell programming (sh, csh, ksh, bash), Matlab, Mathematica, and others. But eventually I used commercial software tools because my consulting clients used them, and they contained most of what I needed to do to solve data mining problems. Since they all have limited sets of functions, I would have to sometimes make creative use of the existing functions to accomplish what I needed to do, but it didn’t stop me from being successful with them. And I didn’t have to write and re-write code for each of these clients.

Each of these tools have their own way of performing an analysis. Command line tools also have their own way of performing an analysis. Much of what separates novices and experts in a software tool is not an awareness of the particular functions or building blocks, but an understanding of how best to use the existing building blocks. This is why I recommend analysts learn a tool and learn it well, becoming an expert in the tool so that the tool is used to its fullest potential.



Examples of workflows in some of the most popular and acclaimed advanced analytics software packages are shown below. Note that the style that has dominated these top tools is the visual programming interface, and they are very similar in how the user builds these workflows.

Figure 1: KNIME Workflow from my Supercharging Predictions workshop at Predictive Analytics World


Figure 2: Statistica workflow from my Predictive Analytics World workshop, Advanced Methods Hands-On


Figure 3: IBM SPSS Modeler Stream, from https://34f2c.https.cdn.softlayer.net/8034F2C/dal05/v1/AUTH_db1cfc7b-a055-460b-9274-1fd3f11fe689/5b0fd91ef0e1a6f21f6e983ccc775a37/offering_3d4b6acc-2e09-4451-b4d0-3385f4385aa8.png





Figure 8: Orange Data Mining from http://1.bp.blogspot.com/-4h6g-5eoJpU/UeiJ1l_9mtI/AAAAAAAAAEc/sCiyxXKP48o/s1600/orangepython.png




22 comments:

Balaji said...

Thank you for taking the time to provide us with your valuable information. We strive to provide our candidates with excellent care and we take your comments to heart.As always, we appreciate your confidence and trust in us
Analytics Training In Chennai

sathya said...

I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog posts. Any way I’ll be subscribing to your feed and I hope you post again soon.


Hadoop Training in Chennai

Base SAS Training in Chennai

iWEB TECHNOLOGIES said...

Your Blog is Very Impressive. I like it......
Website Designing Faridabad
Web Design Faridabad
Web Designing Faridabad
Website Design Faridabad
Web Design Services Faridabad
Website Design Company Faridabad
Website Development Faridabad
Web Development Faridabad
Web Development Services Faridabad
Search Engine Optimization Faridabad
Seo Service Faridabad
Seo Service in Faridabad

Karthi Keyan said...

Its fantatic explaintion lot of information gather it...nice article....
seo company in Chennai

sunitha vishnu said...

It is amazing and wonderful to visit your site.Thanks for sharing this information,this is useful to me...
Android Training in Chennai
Ios Training in Chennai

Freddie King said...

mcdonaldsgutscheine | startlr | saludlimpia

Karthi Keyan said...

Interesting blog post.This blog shows that you have a great future as a content writer.waiting for more updates...
Digital Marketing Company in India

Eshika Roy said...

Thank you for posting this informative blog about cohort analysis of Turk. It has a lot of potential of smoothening things out and make the whole process faster. I can't wait to try it myself. Please keep on sharing more interesting blogs on data analytics in the upcoming posts.

Sathya said...

Here i had read the content you had posted. It is much interesting so please keep update like this.

MSBI Training in Chennai

Informatica Training in Chennai

Abi said...

those who interested to learning the sas program this very helpful..learn the sas program to achieve economical benchmark.
SAS Training in Chennai

sonia mehata said...

Hi, nice blog check out management software & employee management software are in high demand since it helps a lot in tracking the regular activities of the employees. Visit here http://employee-monitoringsoftware.blogspot.com

Blogger said...

If you need your ex-girlfriend or ex-boyfriend to come crawling back to you on their knees (even if they're dating somebody else now) you gotta watch this video
right away...

(VIDEO) Want your ex CRAWLING back to you...?

Ginger Sanches said...

This article describes in great detail possible options custom essay for solving or changing the basic principles of work.

Hope Vignesh said...

Excellent article.. I your way of presentation to deliver the information. Keep it up..

Data Analytics Training in Chennai

Data Analytics Training Institute in Chennai

Mary Brown said...

Hi, Great.. Tutorial is just awesome..It is really helpful for a newbie like me.. I am a regular follower of your blog. Really very informative post you shared here. Kindly keep blogging. If anyone wants to become a Java developer learn from Java Training in Chennai. or learn thru Java Online Training India . Nowadays Java has tons of job opportunities on various vertical industry.

mithlesh Kumar said...

Nice post! I thank you for sharing this nice blog with us. Likewise, I would like to say something that Happy Heap Marketing is also one of the Risk Analytics Company

GrayMatter Software Services said...

Useful Article on Predictive Analytics. Know more about Insurance Analytics (IA+) and also Predictive Analytics in Insurance business

Nexevo Technologies - Web Development Company Bangalore said...

Nexevo Technologies will give best web design & Development with good quality & responsiveaffordable price so get more information:

Best Website Designing Company Bangalore | Best Web Designing Company In Bangalore

kokila zenrays said...

Thanks for sharing this post. Really useful information about Data Analytics. Data Analytics Training in Bangalore

GrayMatter Software Services said...

Hi,
Thanks you for writing an article about predictive analytics.
Read more about Predictive Analytics in Insurance Business

I Digital Academy said...

It's A Nice Post.
Best SAP Training in Bangalore
Best Oracle Training in Bangalore

Tamahagane said...

I definitely appreciate your blog. Excellent work!
Data Analytics