Bayesian optimization in SAS
Learn how Bayesian optimization works through a simple demo.
Author
Dynamic programming with SAS FCMP
Dynamic programming is a powerful technique to implement algorithms, and is often used to solve complex computational problems. This article shows how to use PROC FCMP to implement the "edit distance" algorithm.
How to tokenize documents into sentences
SAS Visual Analytics includes text parsing actions that can help tokenize sentences, and SAS Visual Text Analytics provides even better, more sophisticated methods. This article contains code samples and cites papers for more details.
How to sample textual data with SAS
See how to sample unstructured (text) data using SAS Viya and CAS actions. This post includes complete code to cluster the text documents via k-means, and treats the cluster memberships as strata for analysis.
How to calculate Word Mover's Distance with SAS
Word Mover's Distance (WMD) is a distance metric used to measure the dissimilarity between two documents, and its application in text analytics was introduced by a research group from Washington University in 2015. The group's paper, From Word Embeddings To Document Distances, was published on the 32nd International Conference on Machine
How to get N-grams and TF-IDF count from Chinese documents
SAS Visual Text Analytics provides dictionary-based and non-domain-specific tokenization functionality for Chinese documents, however sometimes you still want to get N-gram tokens. This can be especially helpful when the documents are domain-specific and most of the tokens are not included into the SAS-provided Chinese dictionary. What is an N-gram? An
Clustering of papers using Community Detection
Community detection has been used in multiple fields, such as social networks, biological networks, tele-communication networks and fraud/terrorist detection etc. Traditionally, it is performed on an entity link graph in which the vertices represent the entities and the edges indicate links between pairs of entities, and its target is to
How to extract domain-specific sentiment lexicons
In 2011, Loughran and McDonald applied a general sentiment word list to accounting and finance topics, and this led to a high rate of misclassification. They found that about three-fourths of the negative words in the Harvard IV TagNeg dictionary of negative words are typically not negative in a financial
Analyzing Trump v. Clinton text data at Reddit
Recently a colleague told me Google had published new, interesting data sets at BigQuery. I found a lot of Reddit data as well, so I quickly tried running BigQuery with these text data to see what I could produce. After getting some pretty interesting results, I wanted to see if
Word scatter plot with SAS
In my last post, I showed you how to generate a word cloud of pdf collections. Word clouds show you which terms are mentioned by your documents and the frequency with which they occur in the documents. However, word clouds cannot lay out words from a semantic or linguistic perspective.
Fun with SAS Text Analytics: A qualitative analysis of IALP papers
Last week, I attended the IALP 2016 conference (20th International Conference on Asian Language Processing) in Taiwan. After the conference, each presenter received a u-disk with all accepted papers in PDF format. So when I got back to Beijing, I began going through the papers to extend my learning. Usually, when