The Text Frontier - Text mining, voice mining and unstructured data analysis

I saw the following comment on Twitter yesterday about sentiment analysis limitations and decided it would make a good topic for a blog update:

@concannon: Can anybody explain to me why automated sentiment analysis is anything more than flaky, snake-oil BS? The technology just isn't ready yet.

I’m going make a bold statement here – automated sentiment analysis using the right methodology – is actually superior to human sentiment analysis. Bear with me and read through.

The available approaches to analyzing sentiment/satisfaction vary based on the data provided. I would categorize the approaches based on the availability of three types of data:
1. Customer feedback (free-form text) with customer ranked satisfaction (discrete value), like Amazon product reviews.
2. Customer feedback (free-form text) with manually ranked satisfaction (discrete value), where human readers subjectively score the content.
3. Customer feedback only, no ranked satisfaction, as with blog posts and comments

For the first data type, machine learning algorithms do a good job of measuring overall sentiment (say, +ve/neutral/-ve). Examples of data suitable for this approach are: survey data and product review forums. The problem is that not a lot of text is gathered this way (with a purpose in mind). Even if it is, the machine learning algorithms struggle with distinguishing positive elements from negative. It's one thing to know if a customer is dissatisfied, it is another to know about what!

Given no customer ranked satisfaction, it is possible to build a statistical model using a sample of manually ranked documents, then automatically score the remaining unranked documents. Not many companies are willing to do this. It also doesn't truly represent the customer’s opinion - just the reader’s interpretation of what the customer thinks.

For the third option, customer opinion with no ranking, you can derive sentiment from the context of the text using natural language processing or NLP. This data is most common and hence so are the approaches to analyzing it. It’s not easy, but it’s the sweet spot for gain value from the massive volumes of consumer generated text.

One widely available, cheap technology assigns an overall positive or negative sentiment based on assigning positive or negative values to individual words then summing them to get an overall sentiment rating. This approach fails in situations like the following:
"It's not bad" (two negatives that actually suggest a positive)
"I'm not going to say this sucks" (sarcasm or humor)
“The keyboard is impossibly small but the display is the best I’ve seen.” (combination)

The most recent advances in sentiment analysis technology use a combination of techniques:
(1) statistics
(2) rule-based definitions and
(3) human intervention, e.g. a final review of the machine scoring.

The results are less expensive than human-only sentiment analysis, but more consistent. Why? Because the automation adds consistency, while the human verifies the result. When put in the right workflow then it clearly increases scalability by a substantial factor.

Teragram, a division of SAS, announced the Teragram Sentiment Analysis Manager at the Text Analytics Summit early June. More to come on that!