Thursday, April 10, 2008

Data Mining: Widespread Acceptance When?

Data mining is widely accepted today among industries which have a history of "management by numbers", such as banking, pure science and market research. Data mining is easily viewed by management in such industries as a logical extension of less sophisticated quantitative analysis which already enjoys currency there. Further, information infrastructure necessary to feed the data mining process is typically already present.

It seems likely that at least some (if not many) other industries could realize a significant benefit from data mining, yet this has emerged in practice only sporadically. The question is: Why?

Under what organizational conditions will data mining spread to a broader audience?


Ralph Winters said...

Unqualified support by management that is forward looking and willing to invest in the future. Often this is overlooked as management is beholden to shareholders (for public companies), who are looking for bottom line short term results. Secondly, I think that a proper cost/benefit analysis can bring this to light, but some "soft" benefits need to be included such as better cross department collaboration, knowledge discovery,and job satisfaction benefits etc. These often can't be measured precisely but are important.

Ralph Winters

Dean Abbott said...

The companies that I've worked with in the past have had champions of data mining within the organization, regardless of the vertical market being served. I think a champion is needed because it isn't transparent exactly what data mining actually does to decision makers.

The "black box" model of selling analytics may convince one decision maker, but when that person moves on, there is often no depth of ownership of the approach, and analytics ends. I've seen this happen several times, where analytics succeeds in an organization, but when the champion moves on, the effort stops.

What is surprising to me is how often analytics make the decision maker look really good, but there is still no adoption of it. Perhaps it is as simple as this: when the technology is not well understood, a decision maker has difficulty defending its use.