A few months ago I released the Copy Files task for use with SAS Enterprise Guide. The task allows you to transfer any files between your PC and a SAS Workspace session, much like an FTP process. It doesn't rely on FTP though; it uses a combination of SAS code, Windows APIs, and SAS Integration Technologies to get the job done.
It's proven to be a very popular task, because it can be useful in so many situations. It even earned a mention in a SAS Global Forum paper this year (and no, it wasn't a paper that I wrote).
Today I'm going to point out the things that the task doesn't do so well. Or at least, that it didn't do well until I made some updates. My changes were based on two "complaints" from several SAS users.
Read on for the details. But if you don't care and you just want the latest version of the task, you can download it from here.
Complaint #1: Wildcards that are a little too "wild"
The task allows you to use wildcard characters in your file specifications so that you can match multiple files to transfer. A problem occurs though, when your file specification looks like this:
/usr/local/data/*.xls
Can you guess the problem? What if I told you that the task stores your file specification in a SAS macro variable? Yep, it's that "/*" sequence in the value that trips things up, because SAS interprets it as the start of a comment. Left unchecked, this sabotages the remainder of the SAS code that is included in the process.
The SAS macro experts are already shouting out the answer to fix this: use %STR to wrap the slash and "hide" the token from the SAS parser. That's a great idea! Except that the task relies on the SAS "internal" value for this value --and not the displayed value -- when it comes time to process. These values are different when %STR wraps a special character like the forward slash. The macro facility changes out this character with a hexadecimal character called a delta character.
To illustrate, I used another popular custom task -- the SAS Macro Variable Viewer -- to show the inner value of a SAS macro variable:
Notice the funky arrow characters. Is that what you were expecting?
Now the task detects the presence of a forward slash (and some other special characters) and will automatically add the %STR so you don't have to. (But you can still use %STR if you want to.) And it correctly detects the delta characters, if present, to convert them back to their correct form before trying to use the value.
Complaint #2: Fixing line-ending characters but breaking other stuff
Users of FTP might be familiar with binary versus ASCII mode for file transfers. Because UNIX line-endings are different than Windows line-endings for text files, transferring a file in ASCII mode helps to ensure proper line-ending behavior for the target host.
The Copy Files task transfers ALL files using a binary mode. Why? Because in today's global workplace even text-based files often don't adhere to the limited English-centric ASCII standard. Attempting a text-based file transfer could result in encoding mismatches, so it's much safer to transfer content as "binary blobs".
But you still want your text files to have the proper line endings for the target host. To answer that, the Copy Files task offers a "Fix line-ending characters" option that does the following:
- Scans the file to determine whether it's a text file. (This relies on the file content and not on special file extensions such as .TXT or .CSV.).
- Rewrites the file and replaces the line-ending characters as needed for the target file system (Windows or UNIX).
The problem was that in rewriting the file (using Windows-based StreamReader and StreamWriter functions), the Copy Files task was changing the file encoding to UTF-8. That encoding works fine on Windows and most users didn't even notice. But some users sent me output from file dump tools and comparisons that showed the byte-order mark characters that were added to the file. (SAS users: I knew I could count on you!)
To address this, I changed the "fix line endings" process to use lower level I/O functions that simply scan through the text files as a binary stream, byte-for-byte, and change the line endings as needed. Trying to decide on proper encoding is risky business, so I decided to leave the character encoding untouched.
In addition to my own testing, a couple of users out there have confirmed that my changes fix the issues -- at least for now. Thanks for that! If you want to try the latest, get it now from here:
>> Download the Copy Files task
Update 15Aug2014: SAS users often ask me for an updated version to work with SAS Enterprise Guide 6.1. The 5.1 version of the task has always worked fine with the 6.1 version of the application. But the instructions were not clear on that, so I've built a "6.1" version of the task and added it to the package. If the 5.1 version has been working for you, there is no need to change. If you want to try the 6.1 version (which will work only with SAS Enterprise Guide 6.1), then you now have the option.
Related articles
Copying files in SAS Enterprise Guide
Inspecting SAS macro variables in SAS Enterprise Guide
12 Comments
Pingback: Export and download any file from SAS Enterprise Guide - The SAS Dummy
Chris -- great tool for what I've been able to use it for so far! I'm fairly stuck right now though and cannot see why. Any insight would be great!
Problem: I'm setting a destination path using two macro variables created in a previous linked program:
/.../20&cyr._&cmnth./
The previous program creates cyr and cmnth macro variables like this (hard coded date for easier reading here):
call symput('cyr', substr("01Jan2013", 8, 2));
When I run the program, the Copy Files add-in produces the following warning and does not copy as intended:
WARNING: Apparent symbolic reference CYR not resolved.
Again, thanks for any help here!
Brett,
I think your specific example has a mistake in the SUBSTR function. You want:
/* start at position 6, not 8 */ call symput('cyr', substr("01Jan2013", 6, 2));A complete "self contained" expression might be:
/../20%sysfunc(substr(&sysdate,6,2))_%sysfunc(month(%sysfunc(today())),z2.)
Which (today) yields:
This post of yours does pretty much exactly what I'm trying to do (in reverse). I've tried hard-coding the variable:
%let download_to = /../2013_08/;
I can run %put _global_; after this and see that the macro variable is set how I want it to be set. When the Copy Files program runs, I still get the same warning that it cannot resolve. The EG process I'm running uses prompts that create separate macro variables that then run successfully where needed...it's only in Copy Files that they seem to not resolve.
Brett, is it possible that you are running in a SAS Grid environment? Possibly the macro variables are not defined in the primary SAS workspace, and you would need to SYSLPUT them to the local context?
It looks like that did it. I had checked "run on grid if available" accidentally at some point. Thanks for the help!
I am experiencing problems with the latest version of the Copy Files task. I've just been migrated to Windows 7 environment from Windows XP where this tak ran without any issues. I am getting this error: ERROR: Index (zero based) must be greater than or equal to zero and less than the size of the argument list. All macro variables and paths resolve without any issues. Any ideas on how I can remedy this?
Olga,
Some things are different on Windows 7 -- the location of the Custom tasks folder, for example. Can you verify that you have the proper task DLL in %appdata%\SAS\EnterpriseGuide\\Custom? And that the DLL is Unblocked, per this note?
If you continue to have problems, post back or send me a note at chris.hemedinger[at]sas.com. We might need to turn on some additional logging to see what's happening.
Followup: It turns out that Olga's problem was due to the fact that "My Documents" folder moved to just "Documents" in Windows 7. Once that was corrected in her macro variable, the task worked again.
Hi Chris,
This is really a nice utility and I wonder if I can modify this to transfer files between other ftp servers.
Can you please share your solution file for same? I will give a try!!
Alok, thanks for the kind words. This tool is specialized to use SAS Integration Technologies and the IOM FileService to copy files between a local PC and remote SAS server. It's like FTP, but it does not use the FTP protocol. In SAS you can use FILENAME FTP to get/put files on an FTP server. With that mechanism, you could use a series of FILENAME FTP statements with DATA steps to accomplish what you want, I think. See this example -- which uses PROC HTTP -- to accomplish something similar.
I found the solution to copy multiple files. Instead of using /a/b/c/*.xls use /a/b/c/?*.xls means any one character followed by any number of characters followed by .xls which would serve our purpose.