The Text Frontier - Text mining, voice mining and unstructured data analysis

Things have been mighty busy here at SAS lately. Please accept our apology for the infrequent blog postings. The good news is that there have been some really interesting customer engagements going on that required our full attention. I just returned from working an exhibit with INFORMS in San Diego, where we had so many people ask about text analytics we ran out of handouts. Its the buzz on social media and the idea of mining the words and text we use in those communications that seems to catching on - Big Time! What was traditionally an optimization operations research event, has expanded into all sorts of analytics with NLP and Text Analytics being cited in a dozen different sessions.

Now - for the subject line of this blog entry........ I am delighted to announce that we are expanding our Text Analytics team. We are looking for someone to perform pragmatic product management duties along with our existing Text Miner R&D experts, the Teragram Employees, and our Text Analytic consulting and sales engineers. Text Analytics has grown to encompass much more that the SAS Text Miner product which was originally launched as an "add on" piece to our Enterprise Miner offering. Today Text Analytics includes more than predictive modeling so we are opening up new positions for experts -- thus the open position of Product Manager for SAS Text Analytics.
If you have Bachelor's degree in computer science, applied mathematics, statistics, or a related quantitative discipline and 5 years of experience in product management, consulting, or a related function in the software industry we welcome your application. Expertise in the application of text analytics methodologies is required so I encourage those of you reading this blog to help spread the word to those who run one or more of the text analytics software packages now on the market.

The instructions on how to apply are provided by our Human Relations department here
the job number for the TEXT ANALYTICS position is - 09001816.
This position is located at SAS Headquarters Cary, NC near the RDU airport.

I invite you to browse all our jobs as posted on the main sas web page which you can find by selecting CAREERS on the top horizonal menu bar, and then clicking on professional opportunities. You will see that we are also seeking experienced consultants for our Advanced Analytics Lab. So those of you familiar with our SAS software may wish to apply to those jobs also. Its exciting our these technologies and the people who run them are now IN DEMAND

Meanwhile we on the SAS team are having fun enhancing Text Miner to make the next release of version 4.2 available early December 2009. This will be the release with many Teragram capabilities weaved inside the Text Miner product. More on that topic in a future blog.

Thanks for reading and
...... Thanks for carrying the message out to your boss and your clients that Yes indeed , "Text Analytic Technologies are ready TODAY to be applied and make impacts in our world".

Last Month at the SIGIR meeting in Boston , one of the presentations given by a Teragram customer attracted notice in a twitter post.

The NY Times automated the tagging of topics for their online website by their implementation of software to automatically build their indexes. However, as the tweet points out - the machine has NOT replaced Man because the newspaper continues to rely on MANUAL entries by people who maintain and build the New York Times Index, a more traditional index.



Stephen Arnold wondered in his blog why an organization might continue to require human labor on a task machine can now perform? Could be political resistance to change? or perhaps the machine fails sometimes? Perhaps the employees without skills to be reassigned are in fact prime for the next round of employees to see a "pink slip" as budgets get cut?

Mr. Arnold's ideas are all valid possibilities and I've seen cases of each in my experiences transferring technology from the research lab into business production environments. Those who put a stake in the ground and step forward to be the first to serve as role models for how text analytics can carry their business forward - ought to pause and consider their own culture.

Since the original question was about a Teragram customer implementation, I asked Saratendu Sethi , the director of Engineering at Teragram to share what he's pbserved in his consulting engagements. Here is his response.

First of all, even if automatic categorization guarantees >99% accuracy, for a News company, it is absolutely critical to not portray any wrong information for even 1%. This can only be verified by having humans validate the categorization results. They are doing that on a subset of articles, e.g. front-page articles.

Secondly, new topics constantly emerge in the coverage of current events. Even the best text mining algorithms can’t achieve perfection in spotting emerging topics because these algorithms are usually based on processing past content. Also, the

definition of emerging topics is based on human perception

which is affected by time, location and the type of entities involved in the event. Therefore, these topics have to be manually spotted and added to documents/taxonomy while they are emerging.

Having said that, the following are four benefits that Teragram categorization achieves for New York Times:
(1) If two people are asked to suggest categories on the same document on their own, they are always going to come up with different categories. Automatic categorization enforces consistency and removes human subjectivity by automatically suggesting them categories.
(2) Automatic categorization saves time because it is easier to ask editors to select appropriate categories from an automatically generated list rather than having them to think about them. With automatic categorization, I can just spend few seconds but with manual categorization I have to use few minutes to read the content and decide the appropriate topics
(3) Entity extraction (e.g. identifying person, locations, etc), which doesn’t require much human input, is automated.
(4) Automatic categorization enables New York Times to process all their past archives. Currently New York Times re-processes all their past 25years of content with updated taxonomies every few months.

4a. The human editors are only reviewing articles for current day (~500-1000 articles/day) whereas the past archives might include 100K articles/year.

4b. If “swine flu” was only identified as a News topic in 2009, then automatic categorization allows NYT to find out what other news appeared in past.

So what do you conclude from this post? How would YOU answer the question posed in the title of this entry ? It is cost effective to apply MAN "and" Machine together -- or has the science progressed enought to replace MAN ? Is it time to choose and go with Man "or" Machine approach when deciding about becoming more efficient?

Saratendu answers with the "AND" operator -- and thats the answer I prefer too --cause i'm not comfortable letting those sci fi robots and machines take over my world.

How about you?

Where has the time gone? Here in Texas the hot sun is still roasting, while local retailers are promoting their back to school items on sale. For several weeks now, I've had a blog idea brewing from a talk entitled "WHY COUNT CRIME WHEN YOU CAN PREVENT IT?" You'll see why it caught my interest by noting the image in the top left corner of this slide.



Dr Colleen McCue shows how handwritten police notes and data taken from phone calls can be analyzed to predict future locations and potential criminal events. She'll be speaking live at M2009 but for those who you want to hear her sooner - you can view the archived presentation at your leisure. Her engaging explanation illustrates how Analytics are helping Police departments do their job of keeping neighborhoods safer.

According to Dr McCue "Automated text analytic software could be game changing in information intensive tasks (e.g., a major case will have thousands of tips – the DC Sniper case was compromised in some ways because people focused on the “white van” – the software won’t get tired, bring bias, or forget what it just read). It also has tremendous potential in culling through a lot of interview data (e.g., the detainee data), particularly when you have disparate sources that are geographically diverse but likely connected (through common operational goals, training, etc). "

Three cheers for the FBI, local police - and your local government -- all holding future potential customer success stories for text analytics. Meanwhile you don't want to miss the recent white paper Text Mining for Safety describing how the Oil and Gas industry sees Text Analytics as the answer to moving beyond simply tracking accidents (counting them) to REDUCING hazards on the job. Text Analytics is keeping us safe on the job and at home.

With SAS analytics; many of you are breaking ground as you strive to deliver more value from textual data. Data mining has matured into an accepted practice for Customer Relationship Management teams in Telco, Finance and Marketing. I'd go as far to say that its become essential to survival for most large companies across the globe. Text Mining, as you readers are well aware, is not yet as popular, with many employers assigning just one or two of you with responsibility for text mining.

It can be a burden to struggle alone in a silo without anyone to bounce ideas or brainstorm with. To make it easier for you to connect with peers who share a passion for these technologies – we set up a discussion forum on the topic of SAS Enterprise Miner and SAS Text Miner, two months ago. While SAS employees may participate on these discussions, this forum is not meant to replace the SAS Technical Support help center.


Another excellent way for you to get feedback on your work is to respond to the SGF Call for presentations 2010 (Seattle).

Honestly, one of my favorite things about SAS is
- You - ,
our innovative customer implementing software on your real world challenges.

Only a few text related topics have surfaced in the discussion forum to date, so I’m writing this blog to encourage more of you to join in and set up your profile. Please accept my invitation to post questions, experiences, and thoughts on best practices.

SAS is sending data & text mining experts (including Teragram employees) over the ocean to Europe for two different events this week.
We'll have a booth in the exhibit hall at KDD09 Sunday through Wednesday. If you are one of the lucky ones attending KDD, mark your program to attend the panel discussion to listen to Dr Wayne Thompson from SAS talk about Emerging Trends in Open Standards and Cloud Computing for Data Mining .

Even if you don't make it to the KDD conference to personally pick up the new book authored by the conference chair John Elder , you can experience our Software-on-Demand version of data mining by buying his book, "Handbook of Statistical Analysis and Data Mining Applications."


The second event where you can find us is at the SAS conference devoted to ANALYTICS called A2009 in Denmark July 1,2. The program is online. There you can read the abstract about the a Swedish Insurance firm that studied hand written notes collected by police officers and security guards during 2004-2007.

At both shows, you'll be able to see live demos of our software and pick up a hard copy of the most recent fact sheet, highlighting the enhancements that are now available with the TEXT MINER 4.1 version that was made available to customers 5 weeks ago. Those of you reading this blog that haven't yet seen it may want to read the fact sheet on our SAS 9.2 release of Text Miner on the SAS website.

What does your summer hold for you? Do you have travel plans to shows or conferences with text analytics tracks or sessions included? Please add a comment to this blog and do share!

A recent survey by IDG Research Services, highlights Business Process Automation as an IT priority.

Some of the findings include:
• More than 2/3 of respondents are automating most of their core business processes
• Another 21% are moving towards this goal
• 87% consider BPA to be a critical or important IT priority
• 87% see a connection between unified communications and process automation,
• More than one third envision communication technology being incorporated into BPA in the future

Even though I have not spoken with Joe Staples and Brad Herrington from "Interactive Intelligence", I share their observation that many in today’s economic environment, are trying to streamline operations and do more with less. As organizations seek ways to be more efficient, both in the front office and back office, we might position our technology as a tool for automating business processes leading to improved business results. Have any of you motivated your IT department to spend $$ on Textual Analytic software or recruit support for your research program with this approach?

Its rare for BPA companies to include automating manual processes surrounding words or unstructured content via TEXT Technologies. After I watch the webcast on June 25 and get the white paper - I'll let you know if any mention of Natural Language processing or Content Categorization or Sentiment Analysis is made.

Meanwhile, it's up to all of us to continue to promote awareness and implement Text Analytics into real world situations. We aren't talking about a dream of some vague emerging futuristic possibility , the time is now to include text communication in with traditional data sources of computer processing applications.

When one combines text analytics with mathematical optimization and predictive analytics, we can go well beyond merely automating business processes by improving and discovering entirely new processes leading to a sustainable future. Thanks for reading.

I wrote a piece for eWeek about the Voice of the Customer. In it, I talk about how conversational data collected in call centers is growing faster than our ability to deal with it. Those who don't want to miss insights buried in their data, can now turn to predictive modeling (data mining and text mining) to help them perform voice analytics. Armed with these emerging technologies you can decipher key messages from all the noise and really listen to what customers are saying. Those who learn quickly can respond first (before competitors do) and can deliver better service, better products resulting in happier customers!

Component No. 3: Voice mining your own business

The best way to overcome this obstacle is to secure access to analytic experts at the same time you address any voice mining software purchases. A trained analytical expert will ensure you not only "see" insights, but actually move on them and get the value out of predictive analytic workbenches.

Where have you seen these technologies implemented? do share!

If you are applying these technologies today - or are considering implementing Text Analytic into your organization in the near future - we invite you to take a few moments and take a survey here.

As Manya and others have stated , interest in this field is indeed growing, however there remain many unanswered challenges for our R&D groups to pursue. With your inputs here you can help craft the direction of the next enhancements and guide future application direction. This is an opportunity for all of you out there to share your Perceptions & Plans for text analytics.

Seth Grimes' text-analytics survey will close tommorrow - May 10. He'll write up his findings on how organizations are dealing with unstructured sources and the role text mining/analytics plays as a free report, available in early June.

The survey will take you 5-10 minutes. Thanks for responding!

PS - new members are welcome to the YAHOO group on text analytics.

read about and join us here http://tech.groups.yahoo.com/group/TextAnalytics/

Tune into the March 26 episode of DM Radio at 3 p.m. Eastern Time (U.S.) as editors from Information Management magazine (formerly DM Review) talk to several experts about the power of analytics. The live Internet radio broadcast will include SAS strategist Tammi Kay George, Forrester Analyst Boris Evelson and SPSS vice president of Customer Analytics, Colin Shearer. Together they'll answer the following questions: Why Analytics are bolstering BI across the board ; How the Cloud can make analytics very affordable ;
Why text analytics can be crucial for BI and customer relationship management

Forrester Research predicts that the BI industry will grow to nearly $13 billion by 2014, despite the current economic downturn. This is due in part because of the power and popularity of analytical software, including text analytics, data mining and predictive analytics.

While that statement comes as no surprise to readers of this blog, I find it exciting that others in the IT and BI world are finally catching on. That's why I'm recommending this short DM Radio session -- so you can invite YOUR manager to listen to this broadcast either live (with opportunity for Q&A) or as an archived recording later. Visit the Information Management site to register and find additional information.

Hot news flash - SAS just signed on as sponsors to the Boston TM summit this June. For those readers who have experience with SAS Text Miner, I invite you to contact me in the next week or two. I may be able to reserve a speaking slot on the program agenda for YOU-with SAS helping pick up your registration and travel expenses. Come see first hand what Teragram and SAS are sharing June 1st and June 2nd.

Seth Grimes has compiled an excellent set of quotes from interviews he had with leaders and software developers in this field. you can find his article posted now on the Business Intelligence Network along with Yves Schabes, Manya Mayes and Keith Collins’ views on 2009 challenges and opportunities for text analytics at SAS and Teragram.

From discussions and Q&A with attendees at the Predictive Analytics World conference - I can personally confirm manya's first projection. Namely when she said 2009 challenges will call for

"A broader set of vertical/horizontal offerings including more automated unstructured (text, voice, image) capabilities delivered for customer/product/competitive intelligence"

Although text mining was not addressed in the title or absract of any of the 30 talks on the agenda there were 3 separate discussions that did include ways text analytics fit in. My favorite was when John Elder applied text analytics to "build his haystack" out of the pile of Social Security disability claim forms. By applying text mining, 20% of the cases could be automatically approved quickly leaving more time for the manual review of the others. John's point was that while Text Mining didn't specifically find the needle in the haystack , the technology most certainly helped organize and arrange the haystack so he could allocate tasks to the "artifical intelligence" of the computer and help people do their job faster.

Yes , I say 2009 will be the year business and industries open their eyes to see untapped potential for insight now laying around as unstructured content...for example see this blog where Jason Burke says

As pharmaceutical firms have learned, you can create great solutions for sharing data, and the information still not be useful. For example, if data is sitting in an EMR as an unstructured block of free text, your ability to get to better decisions and predictable outcomes will be considerably constrained. Thankfully, we do have more mature interoperability standards today that can help us along the way. But interoperability should not just be about pushing data across systems -- it should be about facilitating medical insight across stakeholders

Jason has laid the groundwork ...and i wait with baited breath to see his next blog entry - because its the INSIGHT that excites me about analytics.

Text analytics is not promising easy answers - infact it may spur additional new questions for investigation --resulting in worthwhile innovation. Finding case studies will be a challenge for 2009 - just as the yahoo group discussion here has pointed out.

Greetings from San Francisco and the Predictive Analytics World conference!

Surrounded by hundreds of data miners, i'm keeping my hears open to hear who is integrating their unstructured data from text based sources in with their predictive models to build improved scoring engines. I'll fill you in with conference highlights in my next blog.

Monday,this week we announced our first TERAGRAM based offering that implements powerful linguistics - it is separate from our SAS text miner product and does NOT require any of our SAS/STAT or analytical technologies to run. In case you missed the news - check out the press release here

As to the title of this blog entry - I'm cheering cause we have a new medium for reaching broader audience. Interest is building as more people become aware of Text Analytics. Manya's interview on the B-eye network is now available as a podcast on I-tunes.

With my great enthusiasm for text analytics, Its sometimes frustrating when others don't see the potential value these technologies offer. Today when i read Dr G's quote and the facts that came along in the article - it came as music to my ears. My title for this blog entry was pulled from 4 feb 2009 press release. We, the SAS text analytics team, are cheering that a topic near and dear to our hearts was cited as as a key event during 2008. Out of the hundreds of solutions SAS delivered last year, the fact that Teragram's Natural Language Processing and advanced linguistic technology enhancements won top billing in the press release is a big deal!

Those who know how SAS has bundled text analytics as an add-on to SAS Enterprise Miner, the fact that organizations are implementing analytical solutions to retain current customers and win new ones -- you can conclude that seeds have been sown (Data Mining installed). For 2009 our challenge is to continue to pour water (to spread awareness) so Text Mining can indeed blossom .

Theodore Tanner blogged on predictions regarding the future of predictive analytics,semantic intelligence and vanilla BI applications made by industry analysts. I applaud Jim Kobielus who cites Text Analytics as key to staying strong (along with predictive analytics and Data Mining).

Last month when Sascom asked us for our 2009 predictions, you saw me jump in and add mine. For the past five years, we've seen Text Analytics sprout as an "emerging technology" attracting some interest by manufacturing companies for warranty solutions and occasional call centers. Many have no clue what steps to take to implement text analytics much less reap ROI. How about you ? do you think the time is ripe for text analytics to MAKE A DIFFERENCE ?

Thanks Walter, for your comments from Italy that you added to Manya's first post . You gave me the nudge i needed to introduce myself.

Since my role at SAS is in the marketing department, I'll be contributing to this blog with ideas sparked at conferences and networking groups. Your inquiry about how marketers can benefit from TM -- was music to my ears. Allow me to chime in now with my first post.

My academic background is in Operations Research and Management Science - and my passion is applying Math to improve decisions. As the technology marketer for SAS Analytics I cover Data Mining, Simulation, Optimization, Statistics, Forecasting, Visualization, Quality Control as well as Text Mining. My posts will emphasize the practical side of things as i scout about for success stories that generate business benefits across various industry scenarios.

Barry DeVille reminded me to share the web seminars available thru BetterManagement.com. In particular their "conversational marketing" piece sponsored by the AmericanMarketing Association. We invite you to listen to the free audio seminar where you'll hear from Jeff Gilleland, Global Strategist, SAS; Rachelle Spero, SVP Digital Media, Cohn & Wolfe; and Bob Pearson, VP of Communities and Conversations, Dell, as they lead a discussion on the various challenges and opportunities of marketing in the digital age. You'll hear text analytics and linguistic software mentioned as innovative methods already making impacts today.

Stay tuned for more "hits" as i share web groups, discussions, and conferences from those eager to move these "emerging technologies" out into production. Happy holiday greetings to "ya all" (I'm a texan).