The Text Frontier - Text mining, voice mining and unstructured data analysis

When we first came out with SAS Text Miner in 2002, some customers spent weeks or months building extensive synonym lists for their applications. This is because most text mining software operates at the word level... so reducing the number of words that need to be considered is crucial. However, SAS Text Miner works differently. It constructs a Singular Value Decomposition (SVD) that essentially maps all terms into a huge multidimensional space. The closer together two terms are in this space, the more similar they are. Thus the SVD actually itself characterizes the terms based on similarity, essentially automatically creating synonyms. And these synonyms are tailored to the data, so are much more accurate than the generic synonym lists.
For example, suppose that you have a set of documents about data mining. Some might talk about "training" a model. In a generic synonym list the verb "to train" will show up in a synset with "to teach" or "to educate" --- but in the context of modeling it means something very different. By merging these terms together you have lost information --- information that is critical for knowing what the document is about.

There is one situation, however, where synonym substitution can be critical --- when you are dealing with terms that occur very rarely, for example, in just one or a few documents. In this case, the SVD doesn't have enough information to decide where to put this term in its multidimensional space. In many of these cases, the terms are misspellings --- so we have included spell checking in SAS Text Miner to automatically be able to map these terms to the terms that represent the "correct" spelling.

About those customers that have spent so much time building synonym lists? Often they find that they get the best results when they discard the synonym lists they have so caretakingly built up.



By the way, If you want to know more about how the SVD clusters like items together, take a look at this fascinating New York Times article about how SVD is being used to win the Netflix competition and its $1,000,000 grand prize...