Last week I saw a slide from a customer presentation that said 85 percent of that company's data is unstructured data. This isn't usual, either. But what does it mean? First and foremost, it means the data we're all collecting at unprecedented rates is no longer stored strictly in spreadsheets and tidy numeric formats. Instead, it's in field service notes, tech support memos, customer comments, e-mails, blogs, wikis, news feeds, transcripts, medical records, case histories and hundreds of other text-based formats.
It also means that it's getting harder and harder to find the information you need from the hundreds and thousands of text-based documents within an organization. Companies like Google, Microsoft and others are hoping to solve this problem with enterprise search technologies that scour unstructured data sources throughout the enterprise and feed users results in the same user-friendly format they've become accustomed to seeing in Internet search results.
But is search enough? When you're looking for answers in your company's growing collection of text-based documents, are you going to find the answers you need with a basic search algorithm? If you're looking for a single report or piece of background information, yes. But if you're looking for deeper answers, the answer is no.
Here's an example. Enterprise search can quickly reveal how many customers are complaining about your company's newest cellular calling plan for college students. But can it tell you why they're complaining? Or what other factors are affecting their complaints? No. But text mining can.
Text mining transforms text into a structured format by automatically classifying documents and finding key relationships that deliver instant insights. So, now customer letters and call center notes provide valuable messages for WHY some customers may be unsatisfied.
It might turn out, in our example above, that students in different geographic regions are complaining for different reasons. And parents' complaints might differ drastically from students' complaints. Those relationships would take much longer to ascertain with enterprise search. With text mining, the insight is automatic.
Need another example? Read how text mining helped prevent a leg amputation, from Computerworld.
What could you learn with the ability apply text mining to your company's unstructured data sources?
2 Comments
You're right, Alison.
Many use text mining to finally dig deep into stored documents to unearth buried insights.
The rewards are significant and they range from shorter medical recovery time to improved product design.
For retrieval there's search, for deeper understanding, there's text mining.
Alison, Text Mining is much more than searching and filtering to find the right document(s).
True Test Analytics is going beyond the basics of searching and finding. It should be able to make inferences on what is read in the document, make judgement calls as to whether the document is revelant to the inquiry.
We are able to read 100's of 1000's of Claim Adjuster Notes from P&C Insurance claims and determine who is liable, what kind of accident occurred, who was at fault, did the claims adjuster follow best practices, is there a growing trend within the claims that executives should know about (increase mold claims, new types of claims not yet discovered, etc ), and whether the claims adjuster missed subrogation opportunity.
There is a mountain of information within text documents and our industries have not even scratched the surface of what is possible, never mind have an idea that text can be read electronically and make smart decisions to route such info to the appropriate people and send out signals/flags/warnings that something is happening.
Heck, if our CIA, FBI and Homeland security resources utilized text mining, they may have been able to spot 9/11 before it happened.
If we can put a man/woman on the moon, we should be able to read a document and make intelligent sense of what it saying and then make a decision as to what should be done.
No Offense to SAS and other Commercial Vendors of TA, but you all have a LONG WAYS to go to show, demonstrate and prove to the CEOs of the world how they can get a HIGH ROI investing in TA. But these vendors, including SAS, need to become smarter and better at not just reading text but being able to understand what it is saying so people can make the right decisions.
I have been a champion of TA since the early 80s but it fell upon deaf ears and without success stories, TA will stay in the infancy stage for decades until a number of success stories go public and CEOs wake up and say, why cant we do that?
My $.02,
Regards,
Charles Patridge
Sr. Consultant
PDPC, Ltd
Expert in Fuzzy Logic and Data/Text Analytics