One of the problems that trips up experienced SAS users when they begin to use SAS Enterprise Guide is a result of simple geography.
The SAS Enterprise Guide application runs here, on your desktop. The SAS Workspace session (which accesses data and cranks through your analysis) runs over there, on a remote machine. If you're accustomed to "PC SAS" running all on your local box, you might be forced to rethink a few of your processes. For example, do you have Excel files on your PC to import using PROC IMPORT? Does your SAS program create output that you have to save on your PC? How will you get these files to where they need to be?
There are some strategies that help, including mapped drives, UNC paths, and network shares via a mechanism like Samba. These mechanisms allow your local SAS Enterprise Guide and your remote SAS to "see" a shared location on the network. Also, SAS Enterprise Guide has a few focused tasks that can "move" files for you under the guise of "import" and "export" steps.
But there isn't a general method to copy any file you want from your PC to the SAS Workspace, or from the SAS Workspace to your PC. Until now.
I give you: the Copy Files task for SAS Enterprise Guide.
What "Copy Files" does
The Copy Files task allows you to capture file transfer steps within your SAS Enterprise Guide process flow. You can upload files from your local PC to the remote machine where your SAS Workspace is running. And you can download files from your remote SAS Workspace to a folder on your local PC. It is similar in concept to an FTP process, except that this transfer operation uses your already-existing connection to a SAS Workspace and does not require a separate application.
Some readers might be familiar with SAS/CONNECT Data Transfer Services (PROC UPLOAD and PROC DOWNLOAD), which allows you to transfer files between two SAS sessions. Again, this is similar in concept, but since there is only one SAS session in play here, we need to use a different mechanism.
How to copy files in bulk
The Copy Files task supports two features that allow you to copy multiple files with a single step. First, you can use wildcard characters to match on multiple file names. An asterisk (*) matches all characters, in any number, before the next non-wildcard character. A question mark (?) matches any single character that occupies that position in the file name.
Second, you can use SAS macro variables and expressions to specify any part of the source file names or destination folder. Use a SAS program to determine the files you need to copy earlier in the process, assign that value to a macro variable, and then have the task "dynamically" select the correct files when it's time to copy them.
Support for Task Templates
The Copy Files task also supports a standard SAS Enterprise Guide feature: Task Templates. You can create an instance of the task that is useful in one process, then save those task settings to your personalized set of Task Templates that you can use in another process or even in another project. To save your settings as a template, right-click on the Copy Files task within your process flow and select Create Task Template. To use the new template, select it as a menu item from Tasks->Task Templates.
Limitations to note
This task does not generate a SAS program that you can reuse in a batch SAS process or SAS stored process. The task works by using SAS Workspace APIs to transfer data across your network connection, so all of the work happens "behind the scenes". However, the task does create a detailed log output that shows what files were copied, how many bytes were transferred, how long it took, and whether there were any errors.
How to get the task
You can download the Copy Files task from this link (Zip file). The download package includes three versions that depend on your version of SAS Enterprise Guide: one for v4.3, one for v5.1, and one for v4.1 (with a reduced feature set). There is also a documentation file (PDF) that provides instruction for how to install and use the task.
29 Comments
Early Christmas, thank you for this great tool Chris
Very nice! I'd love to see this enhanced with point-and-click dialog boxes to select the files (rather than manually typing them into the field), if not for the "remote server" side then at least for the "local" PC side.
This is a great enhancement, solving a big "usability" problem. Thanks!
"and one for v4.1 (with a reduced feature set).",what does it mean?Can the version for v4.1 copy multiple files?
Thanks Chris!
Yes, it can - it supports wildcard notation. But the user interface is a bit more rudimentary, and it doesn't support Task Templates (available in EG 4.2 and later).
For "Upload from local PC to SAS session", it's OK.But it does not support wildcard notation when download from SAS session to local PC. Is it ok with yours? Or it is just a reduced feature set.
Jacob,
Are you referring to the 4.1 version? It *should* support wildcard notation for source files, whether coming from your local PC or the remote SAS file system. If you're not seeing that, please post back with the content of the log output and I'll try to see what's going on.
Chris,
Yes,I am referring to the 4.1 version.
There existed txt,csv,sas7bdat files under test folder.
I tryed "*.txt" and "*.csv",none successed.My colleagues meet the same issues.
I think it might be the problem of the server where SAS EG4.1.0.0 installed,but not sure.
Do you have any suggestions?
By the way,I'm from Shanghai,and my English is not good,and expression may be not clear or syntax error.If you think my expression is not clear,please let me know.
Thanks very much.
Jacob,
I understand the problem...but I cannot tell from the error what the cause is. It's possible that the path you are specifying is different from the SAS file system. That is, the path might need to be expressed differently.
To test, can you run a SAS program from EG with this statement?
If that works, then SAS sees the path as valid and accessible. If not, then it's possible you need to supply a more complete path in order for SAS to resolve it.
Chris,
The path is OK,I use ' libname test "/sascnapp/data/dm/psacn147/test/"; '
or ' libname Temp "/sascnapp/data/dm/psacn147/MIS361/201212";'
or ' libname test "/sascnapp/data/dm/psacn147/&StaMth."; '(such as %let &StaMth=201301),
ect.,everyday,and SET datasets into the lib.It does work.
Is it that it did not resolve numbers??Oh,I'll test it tomorrow.
Thanks.
Okay, remember these limitations:
- the wildcard notation is only on the file name, not the folder names. The task won't resolve wildcards in folder names.
- the Copy Files task doesn't search into subfolders and process the directories "recursively". It operates only on the one named folder path.
Chris,
This is not the problem of path.It may be the server.Anyway,I find another way to DOWNLOAD files.
Thanks very much for your advice.
Chris,
Make a little change,
/sascnapp/data/dm/psacn147/test/MIS*.sas7bdat
then it works! How Strange!
Excellent! Glad to hear that it's working for you.
This is an elegant tool!
Thank You Chris.
Chris:
I'm having the same trouble as a few folks here but lack their .net expertise. To sum up :
(1) I get the following error when trying to load custom tasks : "Unable to load file...Error: Could not load file or assembly 'SAS.Tasks.ProjectReviewer' or one of its dependencies. Operation is not supported.
(2) Upon further investigation I believe the issue is addressed in this post by Microsoft :
http://blogs.msdn.com/b/shawnfa/archive/2009/06/08/more-implicit-uses-of-cas-policy-loadfromremotesources.aspx
(3) Essentially I need to bypass the issue is by setting :
loadFromRemoteSources enabled="true"
The problem is I don't know where to do that. I tried modifying the SAS config file for the executable but then SAS would not load. It could be that I set it in the wrong place.
Do you think you can provide some insight?
Roberto, check the "Special steps for running task examples" section on this page with some of my examples. Ensure that the DLL file is "Unblocked" on your system, per these instructions. If that doesn't solve the issue, post back and I'll look into it some more.
Thanks!!! That worked!! Have to buy your book now :D!
This is EXCELLENT! We just switched to SAS on a server from PC. This was a major problem until I stumbled upon The Dummy Blog and found this handy add-in. Thanks for sharing it.
Chris,
Thanks for this! I found it yesterday and am using it in conjunction with a web service. Fantastic!
Tom, great to hear it! "Copy files task" + "web service" isn't a mashup that I imagined, but I find that SAS users are often more creative than I am at solving problems.
Hi Chris,
We installed the DLL in the specific users directory originally (C:\Documents and Settings\\Application Data\SAS\EnterpriseGuide\5.1), then changed our minds. When we removed it from the users directory and placed it in the app directory (C:\Program Files\SASHome\SASEnterpriseGuide\5.1\Custom), the DLL doesn't function. Is there a registry change that has to be made or is there a setting within EG itself that can be altered to override the original location of the DLL?
Overall, fabulous task!
-Tom
Tom, by chance, did you re-download the file from support.sas.com when you "moved" it?
Check the "Special steps for running task examples" section on this page with some of my examples. Ensure that the DLL file is "Unblocked" on your system, per these instructions. If that doesn't solve the issue, post back and I'll look into it some more.
Hi Chris,
I have problems installing this add-on on my EG 4.3. It says that ''Unable to load file SASFileTransferTasks43.dll. Error: Add-In is invalid file type."
The dll-file is located the same place as two of your other very nice add-ins, which works perfectly fine. It doesn't make any difference that i unblock the files. Hope you have some suggestions because I really need the Add-In feature :-)
Rune, the file name *should* be SAS.Tasks.CopyFiles43.dll. Maybe you transcribed the message incorrectly in the comment, but just want to make sure.
You can capture a few more diagnostics by turning on logging using the instructions in this SAS Note. If you do that, you'll find a text file with internal logging activity in the %appdata%\sas\enterpriseguide\4.3\logs folder. If you capture the log while you see the "add-in" error message, there should be some additional entries in the log around SAS.Tasks.CopyFiles43.dll and the SasFileTransferTask. If you post those back (or e-mail to chris.hemedinger [at] sas.com) I'll take a look.
Hi Chris,
You are ofcause right that I transcripted the message incorrectly, the file name is "SAS.Tasks.CopyFiles43.dll".
I tried turning on logging and got the following result:
[Main] INFO SAS.Shared.AddIns.Management.AddInRegistry [(null)] - Loading custom task assembly: \\De2900\STRAT\Diverse\SAS\EG\SAS.MacroViewer\SAS.MacroViewer.dll
[Main] INFO SAS.Shared.AddIns.Management.AddInRegistry [(null)] - Successfully loaded custom task assembly: SAS.MacroViewer, Version=4.3.0.0, Culture=neutral, PublicKeyToken=null
[Main] WARN SAS.Shared.AddIns.Management.AddInRegistry [(null)] - Type OptionsTask failed inspection as a custom task: Exception has been thrown by the target of an invocation.
[Main] WARN SAS.Shared.AddIns.Management.AddInRegistry [(null)] - Type MacroTask failed inspection as a custom task: Exception has been thrown by the target of an invocation.
[Main] ERROR SAS.Shared.AddIns.Management.ManagerForm [(null)] - GetAddInRegistryInfo() -
System.Exception: Add-In is an invalid file type.
at SAS.Shared.AddIns.Management.AddInRegistry.LoadElementFromAssembly(String targetAssembly)
at SAS.Shared.AddIns.Management.ManagerForm.GetAddInRegistryInfo(String FilePath)
Any suggestions? I have the same problem with the Macro variable viewer, but not with the SAS Catalog Explorer.
Rune, it looks like you are trying the load the DLL from a UNC path (\\de2900). By default, The .NET Framework disables loading assemblies (DLLs) from network paths, through a feature called "Code Access Security". You'll need to put the DLL on a local drive (not network drive) in order for EG to pick it up. Alternatively, you can enable the Code Access Security by adding your network location as a Full Trust location. To do this, you launch ".NET Framework 2.0 Configuration" (tool on your Windows machine), then select Configure Code Access Security, then either Increase Assembly Trust (to add just the one assembly) or Adjust Zone Security (to make your Local Intranet have Full Trust). Search for "Microsoft .NET code access security" for more background and instructions for the tool.
Hi Chris,
We are using SAS 9.2 with EG 4.2 version do you also have a copy utility for that.4.1 and 4.3 is not working.
Also We are planning to migrate on EG5.1 Citirx but in PoC I could not locate SAS EG on my local machine.Could you please help?
Prashant,
Sorry, I don't have a 4.2 version of the Copy Files utility at this time. If you're planning to move to 5.1 anyway, it sounds like your need for a 4.2 version is short-lived (I hope).
Regarding migrating to Citrix: if you run via Citrix or Terminal Server, your EG session is on a remote Windows environment that you log into. It won't be on your local machine. Thus, a tool like Copy Files (if that's what you're trying to use) has to be installed on the Citrix environment as well. You'll need to ask your IT support staff to help make that happen.
Thank you! This is great! I look forward to integrating all of those repetitive file transfers right into my Process Flow!
4 Trackbacks
[...] Copy Files task (for FTP-like file transfers). It supports wildcard notation and macro expressions, providing lots of flexibility. [...]
[...] usefulness, might work against the policies of those who maintain a SAS environment. For example, the Copy Files task makes it very easy to move files between your SAS server and your local PC. Some organizations [...]
[...] Copy Files to/from SAS [...]
[...] few months ago I released the Copy Files task for use with SAS Enterprise Guide. The task allows you to transfer any files between your PC and a [...]