List the contents of your ZIP files using SAS

16

SAS programmers often resort to using the X command to list the contents of file directories and to process the contents of ZIP files. In centralized SAS environments, the X command is unavailable to most programmers. NOXCMD is the default setting for these environments (disallowing shell commands), and SAS admins are reluctant to change it.

Update 28Nov2016: I updated this article to remove the text about gz (gzip) file support. Currently, the FILENAME ZIP method works only with ZIP files -- on Windows and Unix.

In this article, I'll share a SAS program that can retrieve the contents of a file directory (all of the file names), and then also report on the contents of every ZIP file within that directory -- without using any shell commands. The program uses two lesser-known tricks to retrieve the information:

  1. The FILENAME statement can be applied to a directory, and then the DOPEN, DNUM, DREAD, and DCLOSE functions can be used to retrieve information about that directory. (Check SAS Note 45805 for a better example of just this - click the Full Code tab.)
  2. The FILENAME ZIP method (added in SAS 9.4) can retrieve the names of the files within a compressed archive (ZIP files). For more information, see all of my previous articles about the FILENAME ZIP access method.

I wrote the program as a SAS macro so that it should be easy to reuse. And I tried to be liberal with the comments, providing a view into my thinking and maybe some opportunities for improvement.

%macro listzipcontents (targdir=, outlist=);
  filename targdir "&targdir";
 
  /* Gather all ZIP files in a given folder                */
  /* Searches just one folder, not subfolders              */
  /* for a fancier example see                             */
  /* http://support.sas.com/kb/45/805.html (Full Code tab) */
  data _zipfiles;
    length fid 8;
    fid=dopen('targdir');
 
    if fid=0 then
      stop;
    memcount=dnum(fid);
 
    /* Save just the names ending in ZIP*/
    do i=1 to memcount;
      memname=dread(fid,i);
      /* combo of reverse and =: to match ending string */
      /* Looking for *.zip files */
      if (reverse(lowcase(trim(memname))) =: 'piz.') then
        output;
    end;
 
    rc=dclose(fid);
  run;
 
  filename targdir clear;
 
  /* get the memnames into macro vars */ 
  proc sql noprint;
    select memname into: zname1- from _zipfiles;
    %let zipcount=&sqlobs;
  quit;
 
  /* for all ZIP files, gather the members */
  %do i = 1 %to &zipcount;
    %put &targdir/&&zname&i;
    filename targzip ZIP "&targdir/&&zname&i";
 
    data _contents&i.(keep=zip memname);
      length zip $200 memname $200;
      zip="&targdir/&&zname&i";
      fid=dopen("targzip");
 
      if fid=0 then
        stop;
      memcount=dnum(fid);
 
      do i=1 to memcount;
        memname=dread(fid,i);
 
        /* save only full file names, not directory names */
        if (first(reverse(trim(memname))) ^='/') then
          output;
      end;
 
      rc=dclose(fid);
    run;
 
    filename targzip clear;
  %end;
 
  /* Combine the member names into a single data set        */
  /* the colon notation matches all files with "_contents" prefix */
  data &outlist.;
    set _contents:;
  run;
 
  /* cleanup temp files */
  proc datasets lib=work nodetails nolist;
    delete _contents:;
    delete _zipfiles;
  run;
 
%mend;

Use the macro like this:

%listzipcontents(targdir=c:\temp, 
 outlist=work.allfiles);

Here's an example of the output.
zip file contents within the target directory

Experience has taught me that savvy SAS programmers will scrutinize my example code and offer improvements. For example, they might notice my creative use of the REVERSE function and "=:" operator to simulate and "ends with" comparison function -- and then suggest something better. If I don't receive at least a few suggestions for improvements, I'll know that no one has read the post. I hope I'm not disappointed!

Share

About Author

Chris Hemedinger

Senior Manager, SAS Online Communities

+Chris Hemedinger is the manager of SAS Online Communities. Since 1993, Chris has worked for SAS as an author, a software developer, an R&D manager and a consultant. Inexplicably, Chris is still coasting on the limited fame he earned as an author of SAS For Dummies
He also hosts the SAS Tech Talk webcasts each year from SAS Global Forum, connecting viewers with smart people from SAS R&D and the impressive work that they do.

Related Posts

16 Comments

  1. just to add a comment, consider the scan function - allowing negative values for 2nd parameter, easily extracts the file type when separated by a dot
    Instead of
    /* combo of reverse and =: to match ending string */
    /* Looking for *.zip and *.gz files */
    if (reverse(lowcase(trim(memname))) =: 'piz.') OR
    (reverse(lowcase(trim(memname))) =: 'zg.') then

    consider what follows - it almost doesn't need the clarifying comment
    /* Looking for *.zip and *.gz files */
    if lowcase( scan( memname,-1,'.') in( 'zip', 'gz') then
    That -1 directs scanning for the last "word" delimited by third parameter ('.')

    • To extract the extension from a filename use the SCAN() function. But watch out for filenames without any periods or file names that start with period (hidden files on Unix).
      if index(filename,'.') > 1 then extenstion=lowcase(scan(filename,-1,'.'));

  2. Hi, Chris
    I would like to thank you first for sharing this blog, it's really wonderful. As you say, I have another more straightforward way to find the .gz and .zip files as shown below:

    IF UPCASE(SUBSTR(memname, LENGTH(memname) - 2 )) = ".GZ"
    OR UPCASE(SUBSTR(memname, LENGTH(memname) - 3)) = ".ZIP";

  3. Thank you, thank you , thank you. we are moving to SAS Grid and I really needed the heads up that Admins might set NOXCMD. Anything else I should know?

    This is something I do all the time and so I am likely to need this alternative code.

    • Chris Hemedinger
      Chris Hemedinger on

      Glad that you found this helpful. Usually people moving to SAS Grid are actually experiencing a number of changes: moving off of PC SAS, perhaps moving SAS from Windows to Linux (resulting in file path differences), having data and files on a remote server instead of local. This comprehensive paper can help to prepare.

  4. This works great with one possible exception. I've found that when my ZIP file is larger than 4GB it doesn't find any members. I get a note if using the DEBUG option in the FILENAME statement stating: "ERROR: The system could not find the environment option that was entered."

    Does anyone else find this or have a work around for reading large ZIP files?

    • Chris Hemedinger
      Chris Hemedinger on

      In the initial release of 9.4 there was a limitation when your FILENAME ZIP archive file (the compressed ZIP file) exceeds 4GB -- the resulting file could not be read. That has been fixed in 9.4 Maint 2.

      • Ahhhhhhh...I'm running M1. Time to contact my IT group and get the updated release! Thanks, this has been bugging me for months now.

  5. Cory Vandenberg on

    Chris,

    Thanks for this post. I've been looking for a way to do this on gzipped datasets for a long time. The only thing I've found prior was this, but it requires writing out the compressed dataset from SAS. You can't do the gzip command from unix command line, which is common practice.

    http://support.sas.com/kb/25/214.html

    I've tried your macro above, and while it does list the compressed files in the directory, unfortunately, it does not list the contents in the _Contents1 dataset. There are 0 obs. I have seen your Filename ZIP blog post as well. Hopefully, I can get this working. I would love to leave my very large datasets compressed (350GB+ uncompressed, <70GB compressed).

    I'm using 9.4 M4. Any thoughts?

    Regards,
    Cory Vandenberg

    • Chris Hemedinger
      Chris Hemedinger on

      I assume you read this post about using FILENAME ZIP to read data from a zipped file. You might be running into trouble with limited space (unzipping a large file will require a large amount of temp space) or some other limitation with the FILENAME ZIP method. You might need to contact SAS Technical Support for some guidance if you can't get it working.

  6. Pingback: Using FILENAME ZIP and FINFO to list the details in your ZIP files - The SAS Dummy

Leave A Reply

Back to Top