Mining the VP debate results according to Twitter -- with SAS

Like millions of Americans, I watched the vice presidential debate on television last night. I also watched it on Twitter -- which is to say that while watching the debate, I watched the real-time responses of hundreds of Twitter users.

Twitter users, as you may know, have up to 140 characters to tell the world of Twitter followers what they are doing right now. (Do you care what I'm doing right now?) As a Twitter user, you can sacrifice part of your precious 140 to lend structure to your data through the use of "hash" (as in #) terms. Today, a day after the debate, Twitter users are "voting" for the perceived winner of the debate by tagging their entries with "#vpwinner".

Today I wrote a SAS program that gathers the most recent of these responses, live from Twitter search, and "analyzes" them to tally the votes. Here is the basic approach:

Use the SAS XML Mapper to define an XML mapping of the RSS feed from Twitter to a SAS data set structure.

Use the FILENAME URL method to retrieve the RSS feed from the Twitter search site.

Use the XML libname engine to define the RSS file as a data source.

Repeat the previous two steps to gather at least the last 80 or so "tweets".

Use regular expressions as a crude technique to decide, for each tweet, who the "vote" falls to.

Create a simple bar chart with the result.

I'm in no danger of replacing Gallup polls as an authority on the political pulse of the nation, but the example shows that there is data out there, and SAS can get to it and analyze it.

Here is the output from my quick-and-dirty program, followed by the program itself.

filename twsearch temp;

/** this is the XML map that will convert the RSS search feed
into a SAS data set **/
data _null_;
infile datalines truncover;
file twsearch;
input line $1000.;
put line;
datalines4;
<?xml version="1.0" encoding="windows-1252"?>





<SXLEMAP name="SXLEMAP" version="1.2">


<TABLE-PATH syntax="XPath">/feed/entry</TABLE-PATH>
<COLUMN name="id">
<PATH syntax="XPath">/feed/entry/id</PATH>
<TYPE>character</TYPE>
<DATATYPE>string</DATATYPE>
<LENGTH>37</LENGTH>
</COLUMN>
<COLUMN name="published">
<PATH syntax="XPath">/feed/entry/published</PATH>
<TYPE>numeric</TYPE>
<DATATYPE>datetime</DATATYPE>
<FORMAT width="19">IS8601DT</FORMAT>
<INFORMAT width="19">IS8601DT</INFORMAT>
</COLUMN>
<COLUMN name="link">
<PATH syntax="XPath">/feed/entry/link</PATH>
<TYPE>character</TYPE>
<DATATYPE>string</DATATYPE>
<LENGTH>32</LENGTH>
</COLUMN>
<COLUMN name="title">
<PATH syntax="XPath">/feed/entry/title</PATH>
<TYPE>character</TYPE>
<DATATYPE>string</DATATYPE>
<LENGTH>140</LENGTH>
</COLUMN>
<COLUMN name="content">
<PATH syntax="XPath">/feed/entry/content</PATH>
<TYPE>character</TYPE>
<DATATYPE>string</DATATYPE>
<LENGTH>261</LENGTH>
</COLUMN>
<COLUMN name="updated">
<PATH syntax="XPath">/feed/entry/updated</PATH>
<TYPE>numeric</TYPE>
<DATATYPE>datetime</DATATYPE>
<FORMAT width="19">IS8601DT</FORMAT>
<INFORMAT width="19">IS8601DT</INFORMAT>
</COLUMN>
<COLUMN name="author">
<PATH syntax="XPath">/feed/entry/author</PATH>
<TYPE>character</TYPE>
<DATATYPE>string</DATATYPE>
<LENGTH>32</LENGTH>
</COLUMN>

</SXLEMAP>
;;;;

/** this is the data set that will hold the "tweet" content **/
data work.feed;
length winner $ 10;
run;

/** this macro makes it simple to get several "pages" worth of tweets **/
%macro getpage(num);
%let feed="http://search.twitter.com/search.atom?lang=en&q=%23vpwinner&page=&num";
filename twit URL &feed
/** if you need to specify a proxy server to get to the internet **/
/** proxy="http://myproxy.com" **/
;

/** use the XML library engine **/
libname tf XML xmlfileref=twit xmlmap=twsearch;

data work.feed;
set work.feed tf.entry;
run;
%mend;

%getpage(1);
%getpage(2);
%getpage(3);
%getpage(4);
%getpage(5);
%getpage(6);
%getpage(7);

/** the regular expressions in this data step are crude, but they work **/
/** if the tweet contains #vpwinner followed by "Biden", he gets a tick **/
/** if the tweet contains #vpwinner followed by "Palin", she gets a tick **/
data work.feed;
set work.feed;
if prxmatch('/(#vpwinner)([\s\-\:]+)(biden)/',lowcase(title)) >0 then winner="Biden";
if prxmatch('/(#vpwinner)([\s\-\:]+)(palin)/',lowcase(title)) >0 then winner="Palin";
if prxmatch('/(#vpwinner)([\s\-\:]+)(tie)/',lowcase(title)) >0 then winner="Tie";
run;

title "VP Debate winner according to Twitter users";
ods graphics / width=400 height=250;
proc sgplot data=work.feed;
hbar winner;
xaxis label="#vpwinner 'votes'";
run; quit;