Attention SAS administrators! When running SAS batch jobs on schedule (or manually), they usually produce date-stamped SAS logs which are essential for automated system maintenance and troubleshooting. Similar log files have been created by various SAS infrastructure services (Metadata server, Mid-tier servers, etc.) However, as time goes on, the relevance of such logs diminishes while clutter stockpiles. In some cases, this may even lead to disk space problems.I also recommend: SAS administrators tip: Keeping track of SAS users
There are multiple ways to solve this problem, either by deleting older log files or by stashing them away for auditing purposes (zipping and archiving). One solution would be using Unix/Linux or Windows scripts run on schedule. The other is much "SAS-sier."
Let SAS clean up its "mess"
We are going to write a SAS code that you can run manually or on schedule, which for a specified directory (folder) deletes all .log files that are older than 30 days.
First, we need to capture the contents of that directory, then select those file names with extension .log, and finally, subset that file selection to a sub-list where Date Modified is less than Today's Date minus 30 days.
Perhaps the easiest way to get the contents of a directory is by using the X statement (submitting DOS’ DIR command from within SAS with a pipe (>) option, e.g.
x 'dir > dirlist.txt';
or using pipe option in the filename statement:
filename DIRLIST pipe 'dir "C:\Documents and Settings"';
However, SAS administrators know that in many organizations, due to cyber-security concerns IT department policies do not allow enabling the X statement by setting SAS XCMD system option to NOXCMD (XCMD system option for Unix). This is usually done system-wide for the whole SAS Enterprise client-server installation via SAS configuration. In this case, no operating system command can be executed from within SAS. Try running any X statement in your environment; if it is disabled you will get the following ERROR in the SAS log:
ERROR: Shell escape is not valid in this SAS session.
To avoid that potential roadblock, we’ll use a different technique of capturing the contents of a directory along with file date stamps. This coding technique described below does not rely on the XCMD system option and therefore is preferable.
Macro to delete old log files in a directory/folder
The following SAS macro cleans up a Unix directory or a Windows folder removing old .log files. I must admit that this statement is a little misleading. The macro is much more powerful. Not only it can delete old .log files, it can remove ANY file types specified by their extension.
%macro mr_clean(dirpath=,dayskeep=30,ext=.log); data _null_; length memname $256; deldate = today() - &dayskeep; rc = filename('indir',"&dirpath"); did = dopen('indir'); if did then do i=1 to dnum(did); memname = dread(did,i); if reverse(trim(memname)) ^=: reverse("&ext") then continue; rc = filename('inmem',"&dirpath/"!!memname); fid = fopen('inmem'); if fid then do; moddate = input(finfo(fid,'Last Modified'),date9.); /* see WARNING below */ rc = fclose(fid); if . < moddate <= deldate then rc = fdelete('inmem'); end; end; rc = dclose(did); rc = filename('inmem'); rc = filename('indir'); run; %mend mr_clean;
This macro has 3 parameters:
- dirpath - directory path (required);
- dayskeep - days to keep (optional, default 30);
- ext - file extension (optional, default .log).
This macro works in both Windows and Linux/Unix environments. Please note that dirpath and ext parameter values are case-sensitive.
WARNING: In most cases, finfo(fid,'Last Modified') returns a date/time string in the DDMMMYYYY:HH:MM:SS format as described in the Usage Note 40934. In these cases applying DATE9 informat produces valid SAS date. In some other cases using ANYDTDTM informat may be appropriate. However, as reported by reader Rajeev Meena there are OS installations that return date/time strings in some odd formats, such as "Mon Jun 21 11:11:00 2018". I am not aware of any SAS informat that can directly convert such a string into SAS date. In such cases, some string parsing might be in order to get a convertible string (see Rajeev's suggested solution in his comment below). I suggest checking your operating system for returned date/time string format by running the following code:
data _null_; rc = filename('infile','full_path_name_of_any_existing_file'); fid = fopen('infile'); s = finfo(fid,'Last Modified'); rc = fclose(fid); put s=; run;
By looking at the s value in the SAS log you can decide what informat to use, DATE9, ANYDTDTM or you need to rearrange your date/time string before converting it to a SAS date.
Here are examples of the macro invocation:
1. Using defaults
%let dir_to_clean = C:\PROJECTS\Automatically deleting old SAS logs\Logs; %mr_clean(dirpath=&dir_to_clean)
With this macro call, all files with extension .log (default) which are older than 30 days (default) will be deleted from the specified directory.
2. Using default extension
%let dir_to_clean = C:\PROJECTS\Automatically deleting old SAS logs\Logs; %mr_clean(dirpath=&dir_to_clean,dayskeep=20)
With this macro call, all files with extension .log (default) which are older than 20 days will be deleted from the specified directory.
3. Using explicit parameters
%let dir_to_clean = C:\PROJECTS\Automatically deleting old SAS logs\Logs; %mr_clean(dirpath=&dir_to_clean,dayskeep=10,ext=.xls)
With this macro call, all files with extension .xls (Excel files) which are older than 10 days will be deleted from the specified directory.
Old file deletion SAS macro code explanation
The above SAS macro logic and actions are done within a single data _NULL_ step. First, we calculate the date from which file deletion starts (going back) deldate = today() - &dayskeep. Then we assign fileref indir to the specified directory &dirpath:
rc = filename('indir',"&dirpath");
Then we open that directory:
did = dopen('indir');
and if it opened successfully (did>0) we loop through its members which can be either files or directories:
do i=1 to dnum(did);
In that loop, first we grab the directory member name:
memname = dread(did,i);
and look for our candidates for deletion, i.e., determine if that name (memname) ends with "&ext". In order to do that we reverse both character strings and compare their first characters. If they don’t match (^=: operator) then we are not going to touch that member - the continue statement skips to the end of the loop. If they do match it means that the member name does end with "&ext" and it’s a candidate for deletion. We assign fileref inmem to that member:
rc = filename('inmem',"&dirpath/"!!memname);
Note that forward slash (/) Unix/Linux path separator in the above statement is also a valid path separator in Windows. Windows will convert it to back slash (\) for display purposes, but it interprets forward slash as a valid path separator along with back slash.
Then we open that file using fopen function:
fid = fopen('inmem');
If inmem is a directory, the opening will fail (fid=0) and we will skip the following do-group that is responsible for the file deletion. If it is file and is opened successfully (fid>0) then we go through the deletion do-group where we first grab the file Last Modified date as moddate, close the file, and if moddate <= deldate we delete that file:
rc = fdelete('inmem');
Then we close the directory and un-assign filerefs for the members and directory itself.
Deleting old files across multiple directories/folders
Macro %mr_clean is flexible enough to address various SAS administrators needs. You can use this macro to delete old files of various types across multiple directories/folders. First, let’s create a driver table as follows:
data delete_instructions; length days 8 extn $9 path $256; infile datalines truncover; input days 1-2 extn $ 4-12 path $ 14-270; datalines; 30 .log C:\PROJECTS\Automatically deleting old files\Logs1 20 .log C:\PROJECTS\Automatically deleting old files\Logs2 25 .txt C:\PROJECTS\Automatically deleting old files\Texts 35 .xls C:\PROJECTS\Automatically deleting old files\Excel 30 .sas7bdat C:\PROJECTS\Automatically deleting old files\SAS_Backups ;
This driver table specifies how many days to keep files of certain extensions in each directory. In this example, perhaps the most beneficial deletion applies to the SAS_Backups folder since it contains SAS data tables (extension .sas7bdat). Data files typically have much larger size than SAS log files, and therefore their deletion frees up much more of the valuable disk space.
Then we can use this driver table to loop through its observations and dynamically build macro invocations using CALL EXECUTE:
data _null_; set delete_instructions; s = cats('%nrstr(%mr_clean(dirpath=',path,',dayskeep=',days,',ext=',extn,'))'); call execute(s); run;
Alternatively, we can use DOSUBL() function to dynamically execute our macro at every iteration of the driver table:
data _null_; set delete_instructions; s = cats('%mr_clean(dirpath=',path,',dayskeep=',days,',ext=',extn,')'); rc = dosubl(s); run;
Put it on autopilot
When it comes to cleaning your old files (logs, backups, etc.), the best practice for SAS administrators is to schedule your cleaning job to automatically run on a regular basis. Then you can forget about this chore around your "SAS house" as %mr_clean macro will do it quietly for you without the noise and fuss of a Roomba.
Your turn, SAS administrators
Would you use this approach in your SAS environment? Any suggestions for improvement? How do you deal with old log files? Other old files? Please share below.
I also recommend: SAS administrators tip: Keeping track of SAS users