Text Mining and Regular Expressions

I've been spending quite a lot of time in the bowels of a text mining project recently, mostly in the text/concept extraction phase. We're using the SPSS Text Mining tool for the work so far. (As a quick aside, the text mining book I've enjoyed reading the most in recent months is the Weiss, Indurkhya, Zhang, and Damerau)

The most difficult part of the project has been that all of the text is really customized lingo--a language of its own as presented in the notes sections of the documents we are reading. Therefore, we can't use the typical linguistic extraction techinques, and rather are relying heavily on regular expressions. That certainly takes me back a few years! I used to use regular expressions mostly in shell programming (Bourne, CShell, Korn Shell and later BASH).

I must say it has been very productive, though it also makes me appreciate language rules that don't exist in any consistent way with our notes. As I am able, I'll post on more specifics on this project.

Regarding books on regular expressions, I found the unix books weren't quite so good on this topic. However, the O'Reilly Mastering Regular Expressions book is quite good.

DM Radio and Text Mining

I'll be interviewed on the topic of text mining this coming Thursday, April 17th at 3pm EDT on DM Radio along with Barry DeVille of SAS and Jeff Catlin Lexalytics. The title of this entry links to the DM Review site.

I think you have to register to listen.

The schedule will go something like this:


3:00 PM
Hosts Eric Kavanagh and Jim Ericson frame the argument: What is text analytics, and how can it be used to find those golden needles in the haystack?

3:12 PM
Hosts interview Barry DeVille of SAS Institute: What are some good examples of customer success? What are some common mistakes?

3:24 PM
Hosts interview Jeff Catlin, CEO of Lexalytics: How does his application work? What are some examples of text mining at work?

3:36 PM
Hosts interview Dean Abbott of The Modeling Agency: We heard what the vendors said, but what does that all really mean?

3:48 PM
Roundtable discussion: All bets are off! Guests are encouraged to engage in open dialogue, and listeners can email their questions to dmradio@sourcemedia.com