This week's SAS tip is from Michele Burlew and her book SAS Macro Programming Made Easy, Second Edition. Michele is the author of several extremely helpful SAS books. Visit her author page to learn more about her work and for additional free content.

The following excerpt is from SAS Press author Michele Burlew's book "SAS Macro Programming Made Easy, Second Edition" Copyright © 2006, SAS Institute Inc., Cary, North Carolina, USA. ALL RIGHTS RESERVED. (please note that results may vary depending on your version of SAS software).

Understanding Tokens

The fundamental building blocks of a SAS program are the tokens that the word scanner creates from your SAS language statements. Each word, literal string, number, and special symbol in the statements in your program is a token.

The word scanner determines that a token ends when either a blank is found following a token or when another token begins. The maximum length of a token under SAS 9 is 32,767 characters.

Two special symbol tokens, when followed by either a letter or underscore, signal the word scanner to turn processing over to the macro processor. These two characters, the ampersand (&) and the percent sign (%), are called macro triggers.

Table 2.2 describes the four types of tokens that SAS recognizes.

 The importance of understanding tokenization is evident if you have ever dealt with unmatched quotation marks in your SAS language statements. Matched quotation marks delimit a literal token. When you omit a closing quotation mark, SAS continues to add text to your literal token beyond what you intended. The SAS programming statements added to the literal token never get tokenized by the word scanner. Eventually, the literal token terminates when either another quotation mark is encountered or the literal token reaches its maximum length (32K characters under SAS 9). At that point, your program cannot compile correctly and processing stops.