How to tokenize documents into sentences

SAS Text Analytics analyze documents at document-level by default, but sometimes sentence-level analysis gains further insights into the data. Two years ago, SAS Text Analytics team did some research on sentence-level text analysis and shared their discoveries in a SGF paper Getting More from the Singular Value Decomposition (SVD): Enhance Your Models with Document, Sentence, and Term Representations. Recently my team started working on a concept extraction project. We need to extract all sentences containing one or two query words, so that linguists don't need to read the whole documents in order to write concept extraction rules. This improves their work efficiency on rules development and rule tuning significantly.

Sentence boundary detection

Sentence boundary detection is a challenge in Natural Language Processing -- it's more complicated than you might expect. For example, most sentences in English end with a period, but sometimes a period is used to denote an abbreviation or used as a part of ellipsis. My colleagues Biljana and Teresa wrote an article about the complexities of how a period may be used. if you are interested in this topic, please check out their article Text analytics through linguists' eyes: When is a period not a full stop?

Sentence boundary rules are different for different languages, and when you work with multilingual data you might want to write one set of code to manipulate all data in varied languages. For example, a period in German is used to denote ending of an ordinal number token; in Chinese, the sentence-final period is different from English period; and Thai does not use period to denote the end of a sentence.

Here are several sentence boundary examples:

Sentences | Language | Text
1 | English | Rolls-Royce Motor Cars Inc. said it expects its U.S. sales to remain steady at about 1,200 cars in 1990.
2 | English | I paid $23.45 for this book.
3 | English | We earn more and more money, but we feel less and less happier. So…what happened to us?
4 | Chinese | 北京确实人多车多，但是根源在哪里？
5 | Chinese | 在于首都集中了太多全国性资源。
6 | German | Was sind die Konsequenzen der Abstimmung vom 12. Juni?

How to tokenize documents into sentences with SAS?

There are several methods to build a sentence tokenizer with SAS Text Analytics. Here I only list three methods:

Method 1: Use CAS action applyConcept and SAS Viya

Method 2: Use CAS action tpParse and SAS Viya

Method 3: Use SAS Data Step Code and SAS 9

Among the above three methods, I recommend the first method, because it can extract sentences and keep the raw texts intact. With the second method, uppercase letters are changed into lowercase letters after parsing with SAS, and some unseen characters will be replaced with white spaces. The third method is based on traditional SAS 9 technology (not SAS Viya), so it might not scale to large data as well.

In my article, I show the SAS code of only the first two methods. For details of the SAS code for the last method, please check out the paper Getting More from the Singular Value Decomposition (SVD): Enhance Your Models with Document, Sentence, and Term Representations.

Use CAS action applyConcept and SAS Viya

SAS Visual Text Analytics (on SAS Viya) includes a rich set of CAS actions for categorization, concept extraction and sentiment analysis. The applyConcept action performs concept extraction using a concept extraction model that you compile and validate.

%macro sentenceTokenizer1(
dsIn=,
docVar=,
textVar=,
language=,
dsOut=
);
/* Rule for determining sentence boundaries */
data sascas1.concept_rule;
length rule $ 200;
ruleId=1;
rule='ENABLE:SentBoundaries';
output;

ruleId=2;
rule='PREDICATE_RULE:SentBoundaries(first,last):(SENT, (SENTSTART_1, "_first{_w}"), (SENTEND_1, "_last{_w}"))';
output;
run;

proc cas;
textRuleDevelop.validateConcept /
table={name="concept_rule"}
config='rule'
ruleId='ruleId'
language="&language"
casOut={name='outValidation',replace=TRUE}
;
run;
quit;

/* Compile concept rule; */
proc cas;
textRuleDevelop.compileConcept /
table={name="concept_rule"}
config="rule"
enablePredefined=false
language="&language"
casOut={name="outli", replace=TRUE}
;
run;
quit;

/* Get Sentences */
proc cas;
textRuleScore.applyConcept /
table={name="&dsIn"}
docId="&docVar"
text="&textVar"
language="&language"
model={name="outli"}
matchType="best"
casOut={name="outpos_eli", replace=TRUE}
factOut={name="&dsOut", replace=TRUE, where="_fact_argument_=''"}
;
run;
quit;

proc cas;
table.dropTable name="concept_rule" quiet=true; run;
table.dropTable name="outli" quiet=true; run;
table.dropTable name="outpos_eli" quiet=true; run;
quit;
%mend sentenceTokenizer1;

Use CAS action tpParse and SAS Viya

If you don't have SAS Visual Text Analytics, then SAS Visual Analytics includes CAS actions for more basic parsing and categorization, including an NLP technique called tpParse.

%macro sentenceTokenizer2(
dsIn=,
docVar=,
textVar=,
language=,
dsOut=
);
/* Parse the data set */
proc cas;
textparse.tpParse /
docId="&docVar"
documents={name="&dsIn"}
text="&textVar"
language="&language"
cellWeight="NONE"
stemming=false
tagging=false
noungroups=false
entities="none"
selectAttribute={opType="IGNORE",tagList={}}
selectPos={opType="IGNORE",tagList={}}
offset={name="offset",replace=TRUE}
;
run;

/* Get Sentences */
proc cas;
table.partition /
table={name="offset"
groupby={{name="_document_"}, {name="_sentence_"}}
orderby={{name="_start_"}}
}
casout={name="offset" replace=true};
run;

datastep.runCode /
code= "
data &dsOut;
set offset;
by _document_ _sentence_ _start_;
length _text_ varchar(20000);
if first._sentence_ then do;
_text_='';
_lag_end_ = -1;
end;
if _start_=_lag_end_+1 then
_text_=cats(_text_, _term_);
else
_text_=trim(_text_)||repeat(' ',_start_-_lag_end_-2)||_term_;
_lag_end_=_end_;
if last._sentence_ then output;
retain _text_ _lag_end_;
keep _document_ _sentence_ _text_;
run;
";
run;
quit;

proc cas;
table.dropTable name="offset" quiet=true; run;
quit;
%mend sentenceTokenizer2;

Here are three examples for using each of these tokenizer methods:

/*-------------------------------------*/
/* Start CAS Server. */
/*-------------------------------------*/
cas casauto host="host.example.com" port=5570;
libname sascas1 cas;

/*-------------------------------------*/
/* Example 1: Chinese texts */
/*-------------------------------------*/
data sascas1.text_zh;
infile cards dlm='|' missover;
input _document_ text :$200.;
cards;
1|北京确实人多车多，但是根源在哪里？在于首都集中了太多全国性资源。
;
run;

%sentenceTokenizer1(
dsIn=text_zh,
docVar=_document_,
textVar=text,
language=chinese,
dsOut=sentences_zh1
);

%sentenceTokenizer2(
dsIn=text_zh,
docVar=_document_,
textVar=text,
language=chinese,
dsOut=sentences_zh2
);

/*-------------------------------------*/
/* Example 2: English texts */
/*-------------------------------------*/
data sascas1.text_en;
infile cards dlm='|' missover;
input _document_ text :$500.;
cards;
1|Rolls-Royce Motor Cars Inc. said it expects its U.S. sales to remain steady at about 1,200 cars in 1990.
2|I paid $23.45 for this book.
3|We earn more and more money, but we feel less and less happier. So…what happened to us?
;
run;

%sentenceTokenizer1(
dsIn=text_en,
docVar=_document_,
textVar=text,
language=english,
dsOut=sentences_en1
);

%sentenceTokenizer2(
dsIn=text_en,
docVar=_document_,
textVar=text,
language=english,
dsOut=sentences_en2
);

/*-------------------------------------*/
/* Example 3: German texts */
/*-------------------------------------*/
data sascas1.text_de;
infile cards dlm='|' missover;
input _document_ text :$600.;
cards;
1|Was sind die Konsequenzen der Abstimmung vom 12. Juni?
;
run;

%sentenceTokenizer1(
dsIn=text_de,
docVar=_document_,
textVar=text,
language=german,
dsOut=sentences_de1
);

%sentenceTokenizer2(
dsIn=text_de,
docVar=_document_,
textVar=text,
language=german,
dsOut=sentences_de2
);

The sentences extracted of the three examples as Table 2 shows below.

Example

Doc

Text

Sentence (Method 1)

Sentence (Method 2)

English

1

Rolls-Royce Motor Cars Inc. said it expects its U.S. sales to remain steady at about 1,200 cars in 1990.

Rolls-Royce Motor Cars Inc. said it expects its U.S. sales to remain steady at about 1,200 cars in 1990.

rolls-royce motor cars inc. said it expects its u.s. sales to remain steady at about 1,200 cars in 1990.

2 | I paid $23.45 for this book. | I paid $23.45 for this book. | i paid $23.45 for this book.
3 | We earn more and more money, but we feel less and less happier. So…what happened to us? | We earn more and more money, but we feel less and less happier. | we earn more and more money, but we feel less and less happier.
So…what happened? | so…what happened?

Chinese

1

北京确实人多车多，但是根源在哪里？在于首都集中了太多全国性资源。

北京确实人多车多，但是根源在哪里？

北京确实人多车多，但是根源在哪里？

在于首都集中了太多全国性资源。 | 在于首都集中了太多全国性资源。
German | 1 | Was sind die Konsequenzen der Abstimmung vom 12. Juni? | Was sind die Konsequenzen der Abstimmung vom 12. Juni? | was sind die konsequenzen der abstimmung vom 12. juni?

From the above table, you can see that there is no difference between two methods with Chinese textual data, but many differences between two methods with English or German textual data. So which method you should use? It depends on the SAS products that you have available. Method 1 depends on compileConcept, validateConcept, and applyConcept actions, and requires SAS Visual Text Analytics. Method 2 depends on the tpParse action in SAS Visual Analytics. If you have both products available, then consider your use case. If you are working on text analytics that are case insensitive, such as topic detection or text clustering, you may choose method 2. Otherwise, if the text analytics are case sensitive such as named entity recognition, you must choose method 1. (And of course, if you don't have SAS Viya, you can use method 3 with SAS 9 and guidance from the cited paper.)

If you have SAS Viya, I suggest trying the above sentence tokenization method with your data and then run text mining actions on the sentence-level data to see what insights you will get.

How to tokenize documents into sentences was published on SAS Users.