SAS® Text Analytics and Text Mining in Action: Experiences From a ‘Self-Trial’ With SAS® Contextual Analysis

0

GS_1Don’t get me wrong. I have no doubt in the capabilities of our SAS products and SAS  solutions! But I wanted to get a firsthand experience of our new solution for text analytics, SAS Contextual Analysis 14.1. And the result is very convincing!

But let’s start from the beginning.

Functions and capabilities of SAS® Contextual Analysis

If you take a look at the product description of SAS Contextual Analysis, you learn that you can use it to analyze large collections of text documents, identify sentiments, and create robust models to categorize and extract content. This allows you to automatically identify topics in your document collections and define categories and rules in natural language to assign documents to these categories.

The self-trial: Text analytics with my two SAS® Press books

To betteDPFADQFAr understand the processes and the outcome of text analytics with SAS Contextual Analysis, I used a document collection that is close to my heart and that I know in great detail: the 59 chapters of my two SAS Press books, Data Preparation for Analytics Using SAS and Data Quality for Analytics Using SAS.

Sure, the small number of 59 documents is not really a “big data problem,” and the SAS In-Memory Analytics engine can also deal with millions of documents. However, I was interested to see whether SAS Contextual Analysis can identify topics in my book chapters and which book chapters should be combined into the same cluster. And no a priori knowledge from me as an author would be used for the categorization.

Text analytics processing with SAS® Contextual Analysis

From a data mining point of view, we are dealing here with a typical unsupervised analysis. Just the data are presented to the analytic tool, and no

GS_3

Illustration of underlying topics in the documents

additional information of segment assignment is available. SAS Contextual Analysis imports the data, one file per chapter, from a folder on my hard disk and runs through the entire process of text analytics:

  • Document parsing and assigning the words to different entities (noun, verb, etc.).
  • Synonym detection and the application of stop lists to remove redundant words like “the,” “and,” “of,” “with,” “we,” etc.
  • The weighting of the terms and the identification of those terms that are important to define groups of documents.
  • Automatic detection of underlying topics in the documents.

  It works! Eight clearly separated document clusters as a result

For better illustration, I have used weights of the automatically detected topics for each of the 59 documents to cluster them with SAS® Enterprise Miner™. Eight clusters were automatically detected, which are presented in the table below.

For better visualization, the chapters of the “Data Quality Book” are shown in green and the chapters of the “Data Preparation Book” are shown in yellow.

You can easily see how the chapters grouped to clusters based on content. Some clusters only contain chapters from one book:

  • Cluster 1 contains those chapters from the Data Quality Book that deal with the topic of missing values.
  • Cluster 7 contains the simulation studies that are described in chapters 15-23 of the Data Quality Book.
Clustering with SAS Text Analytics

SAS Text Analytics automatically detected 8 clusters in the 2 books

 

 

 

 

 

 

 

Some clusters contain chapters from both books:

  • Cluster 8 contains chapters from the Data Preparation Book that deal with analytics data mart structures. And Appendix E in the Data Quality Book is a summary of the content of these chapters. This is an impressive example of documents only grouped based on their content. And chapter content that is considered to be “close” or “similar” is truly detected as such.

The different number of documents per cluster also show that no fixed clustering scheme is used here, but that the document content defines how the groups are set up and how they are populated.

  • Cluster 4 only contains a single chapter. This chapter is an introduction to a collection of case studies and obviously does not compare with other chapters in the books.

Moving on to new business cases

These results convinced me even more that SAS Contextual Analysis allows you to gain insight into your document collections. You learn what your customers think and write about your company or organization. You see the topics that are contained in your documents and how you can automatically group them without having to read every single document.

I also presented a paper (in German) on this case study on the Austrian SAS User Conference “SAS Club” in November 2015.

Share

About Author

Gerhard Svolba

Principal Solutions Architect

Dr. Gerhard Svolba ist Analytic Solutions Architect und Data Scientist bei SAS Institute in Österreich. Er ist in eine Vielzahl von analytischen und Data Science Projekten quer über fachliche Domains wie Demand Forecasting, analytisches CRM, Risikomodellierung und Produktionsqualität involviert. Seine Projekterfahrung reicht von der fachlichen und technischen Konzeption über die Datenaufbereitung und die analytische Modellierung in unterschiedlichen Branchen. Er ist der Autor der SAS Press Bücher Data Preparation for Analytics Using SAS, Data Quality for Analytics Using SAS and “Applying Data Science: Business Case Studies Using SAS”. Als nebenberuflich Lehrender unterrichtet er Data Science Methoden an der Medizinischen Universität Wien, der Universität Wien und an Fachhochschulen. Sie finden auch Beitrage auf: Github und Twitter. ENGLISH: Dr. Gerhard Svolba ist Analytic Solutions Architect und Data Scientist bei SAS Institute in Österreich. Er ist in eine Vielzahl von analytischen und Data Science Projekten quer über fachliche Domains wie Demand Forecasting, analytisches CRM, Risikomodellierung und Produktionsqualität involviert. Seine Projekterfahrung reicht von der fachlichen und technischen Konzeption über die Datenaufbereitung und die analytische Modellierung in unterschiedlichen Branchen. Er ist der Autor der SAS Press Bücher Data Preparation for Analytics Using SAS®, Data Quality for Analytics Using SAS® and “Applying Data Science: Business Case Studies Using SAS”. Als nebenberuflich Lehrender unterrichtet er Data Science Methoden an der Medizinischen Universität Wien, der Universität Wien und an Fachhochschulen. Sie finden auch Beitrage auf: Github und Twitter.

Leave A Reply

Back to Top