Reading and writing GZIP files with SAS

13

Remember when 100MB was large?

SAS 9.4 Maintenance 5 includes new support for reading and writing GZIP files directly. GZIP files, usually found with a .gz file extension, are a different format than ZIP files. Although both are forms of compressed files, a GZIP file is usually a compressed copy of a single file, whereas a ZIP file is an "archive" -- a collection of files in a compressed virtual folder. GZIP tools are built into Unix/Linux platforms and are commonly used to save space when storing large text-based files that you're not ready to part with: log files, csv files, and more. The algorithm used to compress GZIP files performs especially well with text files, although you can technically GZIP any file that you want.

I've written extensively about using FILENAME ZIP to read and write ZIP archives with SAS. The latest version of SAS adds support for GZIP by extending the FILENAME ZIP method. When working with GZIP files, simply add the GZIP keyword to the FILENAME statement. Example:

filename my_gz ZIP "path-to-file/compressedfile.txt.gz" GZIP;

Here's an example that creates a compressed version of a log file:

filename source "C:\Logs\SEGuide_log.10168.txt";
filename tozip ZIP "C:\Logs\SEGuide_log.10168.txt.gz" GZIP;
 
data _null_;   
   infile source;
   file tozip ;
   input;
   put _infile_ ;
run;

In my test here, the result represents a significant size difference, with the compressed file occupying just 14% of the space.


To "re-inflate" the compressed file, we can perform the opposite operation. (I added the ENCODING option here because I know my log file was UTF-8 encoded.)

filename target "C:\LogsExpanded\SEGuide_log.10168.txt" encoding='utf-8';
filename fromzip ZIP "C:\Logs\SEGuide_log.10168.txt.gz" GZIP;
 
data _null_;   
   infile fromzip;
   file target ;
   input;
   put _infile_ ;
run;

You don't have to explicitly expand a compressed text file in order to read it with SAS. You can use the GZIP method to read and parse a .gz file directly, similar to the zcat command that you might be familiar with from the Unix shell:

filename fromzip ZIP "C:\Logs\SEGuide_log.10168.txt.gz" GZIP;
data logdata;   
   infile fromzip; /* read directly from compressed file */
   input  date : yymmdd10. time : anydttme. ;
   format date date9. time timeampm.;
run;

If your file is in a binary format such as a SAS data set (sas7bdat) or Excel (XLS or XLSX), you probably will need to expand the file completely before reading it as data. These files are read using special drivers that don't process the bytes sequentially, so you need the entire file available on disk.

Note: Because each GZIP file represents just one compressed file, the MEMBER= option doesn't apply. When dealing with ZIP file archives that contain multiple files, you could use the MEMBER= option on FILENAME ZIP to address a specific file that you want. My recent example about FINFO and file details relies heavily on that approach. However, the GZIP option and MEMBER= options are mutually exclusive. In that way, it's much simpler...just like its Unix shell equivalent.


* ZIP drive image By © Raimond Spekking / CC BY-SA 4.0 (via Wikimedia Commons), CC BY-SA 4.0, Link  

Share

About Author

Chris Hemedinger

Senior Manager, SAS Online Communities

+Chris Hemedinger is the manager of SAS Online Communities. Since 1993, Chris has worked for SAS as an author, a software developer, an R&D manager and a consultant. Inexplicably, Chris is still coasting on the limited fame he earned as an author of SAS For Dummies.  He also hosts the SAS Tech Talk webcasts each year from SAS Global Forum, connecting viewers with smart people from SAS R&D and the impressive work that they do.

Related Posts

13 Comments

  1. Chris,

    Are other zip programs supported? Such as bz2 (.bz), xz (.xz), compress (.Z)? I could see how this could be useful in ODA since we have users that want to copy a large number of files from their home directory but SAS Studio only allows you to do one at a time. If they zip them first, that would let them grab their data in bulk...back up their EM projects, etc. hmmm, bet that would make a good blog post :-).

    • Chris Hemedinger
      Chris Hemedinger on

      Hey Galen, standard ZIP file archives are supported with FILENAME ZIP (read and write) and also ODS PACKAGE (write only) -- you can use either of those methods to create a single compressed archive from a collection of files. I've got plenty of blog posts on all of those methods.

    • Chris Hemedinger
      Chris Hemedinger on

      Portability (works the same on every system, regardless of operating system), and not every SAS environment is set up to allow X command. Most SAS Enterprise Guide and SAS Studio users don't have access to the operating system shell for commands.

  2. Julia Cohen (jcinma) on

    How can I adapt the "filename source" line up above to write a temporary sas7bdat file in my Work library to a permanent sas7bdat.gz file? Thanks!

    • Chris Hemedinger
      Chris Hemedinger on

      Here's an example that shows a "round trip" -- from data file, to GZ file, then expanded back to a sas7bdat.

      /* get a data set into a library */
      libname lib "c:\temp";
      data lib.cars;
        set sashelp.cars;
      run;
       
      filename raw "%sysfunc(pathname(lib))/cars.sas7bdat";
      filename zipped ZIP "c:\temp\cars.sas7bdat.gz" GZIP;
       
      /* Compress the data set into a GZ file */
      data _null_;
         infile raw 
             lrecl=256 recfm=F length=length eof=eof unbuf;
         file   zipped lrecl=256 recfm=N;
         input;
         put _infile_ $varying256. length;
         return;
       eof:
         stop;
      run;
       
      /* Delete the original uncompressed file */
      data _null_;
       rc=fdelete('raw');
      run;
       
      /* put the file back in the directory, expanded */
      data _null_;
         infile zipped
             lrecl=256 recfm=F length=length eof=eof unbuf;
         file raw lrecl=256 recfm=N;
         input;
         put _infile_ $varying256. length;
         return;
       eof:
         stop;
      run;
  3. Dear Chris, thank you very much for your code. It works fine. However, I failed when I replace the sas7bdat.gz file with the file I need to work on.
    The url link of the data: http://scholar.rhsmith.umd.edu/sites/default/files/sbrown/files/pins_vdj_ann.sas7bdat.gz?m=1467366849
    My code:
    libname pinlib "E:\data\PIN_Stephen_Brown";
    filename target "%sysfunc(pathname(pinlib))/test.sas7bdat";
    filename fromzip ZIP "E:\data\PIN_Stephen_Brown\test.sas7bdat.gz";
    data _null_;
    infile fromzip
    lrecl=256 recfm=F length=length eof=eof unbuf;
    file target lrecl=256 recfm=N;
    input;
    put _infile_ $varying256. length;
    return;
    eof:
    stop;
    run;
    It reports the error message as :
    ERROR: The file "E:\data\PIN_Stephen_Brown\test.sas7bdat.gz" exists and is not a zip
    file. The output file must be a zip file.
    I appreciate your help very much if you could help me with this case.

    • Chris Hemedinger
      Chris Hemedinger on

      Here's a SAS program that breaks the problem down to the necessary steps. First step: downloads the file. Next step, assigns a FILENAME ZIP (GZIP) fileref to the downloaded file. Then a DATA step to copy its contents to a WORK data set.

      /* This gets the data in GZ format */
      filename webdata "%sysfunc(getoption(WORK))/pins_vdj_ann.sas7bdat.gz";
      proc http
       url="http://scholar.rhsmith.umd.edu/sites/default/files/sbrown/files/pins_vdj_ann.sas7bdat.gz?m=1467366849"
       out=webdata
       method="GET";
      run;
      filename webdata clear;
       
      /* The expands the GZ data to a WORK data set */
      filename zipdata ZIP "%sysfunc(getoption(WORK))/pins_vdj_ann.sas7bdat.gz" GZIP;
      filename unzip "%sysfunc(getoption(WORK))/pins_vdj_ann.sas7bdat";
       
      data _null_;
         infile zipdata
             lrecl=256 recfm=F length=length eof=eof unbuf;
         file unzip lrecl=256 recfm=N;
         input;
         put _infile_ $varying256. length;
         return;
       eof:
         stop;
      run;
       
      proc print data=work.pins_vdj_ann (obs=5);
      run;
  4. Chris,
    I have a variation of the gz zip that I am trying to resolve. My gz file contains many .tar files containing xml that I need to process individually. Using the solution shown above I get everything decompressed as a single file instead of the individual files structure that I need. Is there a solution for this?

    • Chris Hemedinger
      Chris Hemedinger on

      My understanding is that a gz file can contain just one entry -- it's a single file, compressed. But that could be a tar file, which is a "tarball" collection of several files together. So uncompressing these is a two step process. Use the FILENAME ZIP with GZIP to get the tarball (.tar file). Then (and I haven't tried this) you might be able to use FILENAME ZIP (not GZIP) to get to the individual tarball members. If not, you might need to use FILENAME PIPE (if you can) to untar the members.

Leave A Reply

Back to Top