SAS users, by definition, do not embrace the mysterious.
That's one of the main reasons that they use SAS: to demystify some data or process. And so, when you (as a SAS user) have gone to the trouble of designing a process flow in SAS Enterprise Guide, you like to be aware of some basic metrics, such as "how long will it take to run?"
It's difficult to predict how long a SAS process will take to run, as it depends more upon the data than on the actual program instructions. But one thing that we're very good at is telling you how long it took to run the last time that you ran it. In SAS Enterprise Guide, you can find this information at the task level by right-clicking on the task (or program node) and selecting Properties. On the General tab, you'll see the "Last execution time".
If you want to gather this information at a process flow or project level, you can repeat these steps for each item in the flow, make a note of the "execution time", then add up the numbers (expressed in hours, minutes, and seconds) to create a grand total. This tedious assignment makes for a perfect torture device for a summer intern who, in this economy, should be grateful to have a job at all.
Or, to make the job less tedious, you could use the Project Reviewer task. This is a custom task (available for download here) that shows a summary view of your process flows and allows you to create a report from the information. The task works with SAS Enterprise Guide 4.3 and 5.1.
Features of the Project Reviewer task include:
1. A selection list with all of the process flows within your project.
2. A list of each "runnable" task; that is, program, task, query, export step, etc. Each task has an "ordinal" (its sequence in the process flow), a name, a descriptive type, the user ID of the last person to modify it, the running time for its most recent iteration, the date/time modified, the date/time created, and whether the task generated errors ("red X" in the flow).
You can sort the items in the list by clicking on the column header for the value you want to sort. Click on the column header again to reverse the sort sequence.
3. A summary of the task count, and total clock time that "running" the flow represents.
4. A Create Report button, which generates a SAS program to produce a simple report of all of the project contents, summarized by each process flow. If you have multiple SAS environments, you can use the Report server list to select which SAS server to use when processing the report.
Here's a sample of the report output:
The reporting process also generates a SAS program and a data set (which are added to your project), so that you can easily adapt these for custom reports.
Let me know if you find this task to be useful and whether you have any improvements to suggest.
Some final notes/links:
- Download location for the Project Reviewer task (ZIP file). The ZIP file includes the task DLL and a README.PDF document with instructions for how to install and use the task. Note that there is a different DLL for each minor version of SAS Enterprise Guide -- be sure to copy/install the correct one for your version.
- Related tip: Using the SYSECHO statement to track progress of SAS programs in SAS Enterprise Guide.
- SAS Enterprise Guide hotfix locations: 4.305 with 9.2, 4.305 with 9.3, and 5.1 - if you intend to use this task along with SAS Enterprise Guide automation (for example, in a scheduled project), you'll need the latest hotfixes for proper operation
51 Comments
I don't know how you figure out all these cool tricks.
Inside information. Frowned upon for securities trading, but definitely an advantage in software development.
Are there any other add-ins for EG5.1?
Bob, most of the add-ins that work with 4.3 also work with 5.1. For example, check out the Macro Variable Viewer and the SAS System Options viewer tasks.
Sorry Chris - I'm running 5.1 and this task did not appear automatically when added to the Custom folder (whch I had to create in the designated location). I tried to add it via the Add-in Manager and got message "Unable to load file...Error: Could not load file or assembly 'SAS.Tasks.ProjectReviewer' or one of its dependencies. Operation is not supported. (Exception from HRESULT..) I do not have Admin rights and cannot make changes to the registry - is this the problem?
Hi Bob, I'm not sure what's going on here, but you might be able to get some diagnostics by enabling additional logging. Follow the instructions in this SAS Note SN-17730 (applying the operations to the 5.1 folders). You'll end up with a text log in %appdata%\SAS\EnterpriseGuide\5.1\Logs, and it will show some activity where the DLL was examined for inclusion on the Tools->Add-in menu. You might also try a different custom task (see response to your other comment) to make sure the mechanism is working for you in general.
Thanks Chris - seems to be a local .net problem:
An attempt was made to load an assembly from a network location which would have caused the assembly to be sandboxed in previous versions of the .NET Framework. This release of the .NET Framework does not enable CAS policy by default, so this load may be dangerous. If this load is not intended to sandbox the assembly, please enable the loadFromRemoteSources switch. See http://go.microsoft.com/fwlink/?LinkId=155569 for more information.
Bob, good to hear you've got a handle on it. I can see this happening if you put the DLL on a mapped drive or UNC path, or perhaps if your %appdata% location maps to something other than your local machine (perhaps a roaming profile). But you can probably workaround it by creating a "Custom" folder in your SAS Enterprise Guide app directory (with an admin's help or if you have permissions) and then placing the DLL in there.
You're so awesome! I do not believe I've read a single thing like this before. So good to discover somebody with a few genuine thoughts on this topic. Really.. thanks for starting this up. This site is one thing that is required on the internet, someone with some originality!
Willie Mae,
Thank you for your over-the-top response. Despite the fact that your comment contains no specific reference to the content of this post, I shouldn't assume that this is the work of a spam-bot, should I? Instead, I choose to believe that your comment is sincere and, of course, completely justified in its enthusiasm.
I hope that you don't mind that I removed the trackback link -- I didn't want my large readership to overload your website with traffic.
Chris,
Is there any features in SAS 9.2 for Unix that allows for similar report creation? I have used scripts to pull cpu time and real time counts out of the log file, but this would be easier.
Thanks -
Bo,
Of course, if your process was driven through SAS Enterprise Guide, this task would work for that as well, even with the SAS processing on UNIX. Aside from that, you might look at the SAS Logging Facility -- which maybe you already use. You would still need to script the extraction of the timings to create a report, I think.
Pingback: My favorite SAS tips and tools from 2012 - The SAS Dummy
Pingback: Upcoming SAS Talks: Custom tasks for SAS Enterprise Guide - The SAS Dummy
Pingback: Custom tasks for SAS Enterprise Guide: Q&A - The SAS Dummy
Hi Chris, you've got a great site! I would like to user your Project Reviewer custom task but ran into trouble. We are running SAS on a UNIX server, with EG running on Windows clients. I tried using the Add In Manager in SAS EG 4.3 to bring in the DLL, but got an error. Does our SAS Admin first need to create a "custom" folder in a location that only he can access and we users cannot? Any other instructions relevant to UNIX?
The error was: "Unable to load file "...." Error: Could not load file or assembly 'SAS_Tasks_ProjectReviewer.dll' or one of its dependencies. The located assembly's manifest destination does not match the assembly reference. (Except from HRESULT: 0x80131040)
Thanks so much !
--Mark
Hi Mark,
Check the steps in this post about installing custom tasks and see if it helps.
This looks useful... Thanks
When I run the report, I see that the data is entered through the datalines statement
Where does that data originate?
Can I get it programmatically?
Maybe through system/macro variables?
That data is stored internally in the EG project. The task does the work of pulling it out and creating the DATALINES records. Unfortunately, there isn't a SAS program method to pull out this same information.
Hi Chris,
I've been looking for the informations we can see in the properties of a program within SASEGuide, like Last Execution Time, and I do not find it in any property of the SAS EG Scripting objects. Can you help me find it ? to be able to enhance a project reviewer...
Thank you very much
Pascal, there isn't an exposed property for that. You can perhaps glean from the Log.Text, but that's not as neat as a property would be.
Thank you Chris,
could I put it in the shopping list for Santa ?
PS: I hope it is stored somewhere in the project...
have a nice day
Pascal, I'll put in a word to the Big Man, but you should have a backup gift request. How about a nice football?
Yes, the information is in the project, but not exposed through any public/documented property. That might inspire you to go spelunking; use caution when straying from the documented APIs.
Great tool, thank you very much for it!
Hi Chris,
I've the SAS 9.4 with EG 6.1 which is on x64 and the selecte process flow is blank.
Here for more detail:
Exception Details:
----------------------------------------
Exception type: System.IO.FileNotFoundException
Message: Impossible de charger le fichier ou l'assembly 'SAS.EG.ProjectElements, Version=4.3.0.0, Culture=neutral, PublicKeyToken=be58efc3b934219b' ou une de ses dépendances. Le fichier spécifié est introuvable.
Source: SAS.Tasks.ProjectReviewer
Hugo,
I've made a small adjustment that should allow it to work in EG 6.1. Can you download another copy and replace the previous version? The new DLL will have today's date (21Jan2014) and a file version of 4.3.0.14021.
Hi Chris,
I discovered your blog and it contains a lot of useful information.
This project reviewer task is really interesting. But I was wondering, would it be possible to also get other information such as CPU time?
My point is that I'm trying to optimize my code and I would like to see the impact of the optimization. However, the SAS server load has a big impact on the real time. So that I can not really compare two different runs of my project.
Again thank you for your post
Seb
Seb, it sounds like you want the metrics available from the FULLSTIMER option. This SAS system option emits many useful measurements into the log. However, SAS Enterprise Guide doesn't aggregate these for you, so if you enable the option in your code, you'll need to peruse the SAS logs in your project to see the measurements and the effects of your optimizations.
Hi Chris,
I'd like to use your custom task "Project Reviewer" for the EG 7.1 x64 with SAS 9.4, but I always get the following exception message. The File"SAS.EG.ProjectElements" has the version 7.100.0.2002. Can you tell me if I am making a mistake? Or could you perhaps update the latest version of your custom task?
have a nice day - Dennis Voigt
Exception Details:
----------------------------------------
Exception type: System.IO.FileNotFoundException
Message: Die Datei oder Assembly "SAS.EG.ProjectElements, Version=4.3.0.0, Culture=neutral, PublicKeyToken=be58efc3b934219b" oder eine Abhängigkeit davon wurde nicht gefunden. Das System kann die angegebene Datei nicht finden.
Source: SAS.Tasks.ProjectReviewer
Target Site: .ctor
Dennis,
I'm sorry that you hit this problem. I have built a special new version for SAS Enterprise Guide 7.1 and added it to the download package. Note that the new version has "71" in the name of the DLL. When you install it, make sure that any previous version (SAS.Tasks.ProjectReviewer.dll) is NOT in the path or already installed. This would cause a conflict that prevents the 7.1 version from loading.
Hi Chris, as always your tips & tricks are very helpful. I had a somewhat related question if you don't mind. My group just upgraded from EG 4.3 to 6.1 in a Grid architecture and one thing I noticed missing is how the log use to display the execution time of each step, even without STIMER. I don't have access to a copy of 4.3 so I'm not sure if there was a setting or what, but I'd love to get this information back in the log without having to add STIMER to everything. thx
Mr. Smith,
I guess the different level of detail in the log is due to the way the Grid environment is set up. There are many options that you (or a SAS admin) can tweak to adjust the way that SAS Grid Computing will behave. See this blog by Eduardo and his thorough paper on the topic.
Pingback: Project Reviewer task and Copy Files task: refreshed for v7.1 - The SAS Dummy
Hi Chris,
love this add-in and have installed it to all our machines.
I did have one problem when trying to install on the first machine. I got a HRESULT 0x08.... which was solved by opening the file properties in windows explorer and clicking on 'Unblock this file'
thanks John
John, thanks for the feedback! I'm so glad that you find this task useful.
Hi Chris really loved his post. Can you give me a link to the source code of the dll or help me through the steps?
Shubham,
The source code for this project uses a tiny bit of unpublished APIs, so I haven't shared it out. However, you can achieve much of this with the ISASProject APIs that are part of the custom task APIs. The SAS Task Property Viewer task uses those same APIs, and I shared that as an example in my Custom Tasks book.
Problem using EG 7.1.2 X64
Message is:
Missingl file or 'assembly 'SAS.EG.ProjectElements, Version=7.100.1.0, Culture=neutral, PublicKeyToken=be58efc3b934219b'
Seems like a version mismatch. Any option for 7.1.2?
thanks for your precious work
Hi Andrea, I had to build a special version for 7.12. I've just placed it in the ZIP package. Re-download and extract the version with the *712 suffix, and REMOVE your old versions of the task DLL.
We are running the latest 7.13 version (7.100.3.5408) (64-bit) and would request that a similar DLL solution be added to the ZIP package for that newest version. Tests against the 3 currently supplied DLL files all failed with our current configuration. Thanks Chris!!
Kevin -- it was on my to-do list, but your comment moved it to the top :)
This is now done -- the updated package has a 713 version. Remove all other versions and use this instead, if you're on EG 7.13.
Hi Chris,
Came across your tool recently and tried to add it to my EG installation: SAS Enterprise Guide 64-bit version: 7.13 HF5 (7.100.3.5486)
As per instructions I copied the SAS.Tasks.ProjectReviewer713.dll to the designated folder. However, the Project Reviewer task did not appear.
When I tried to add it with the add-in manager I get the following error:
Unable to load file /SAS.Tasks.ProjectReviewer713.dll
Error: Could not load file or assembly 'SAS.Tasks.ProjectReviewer713' or one of its dependencies. Operation is not supported. (Exception from HRESULT: 0x80131515)
What am I doing wrong?
Resa, This is the security “sandbox” that Windows imposes on DLLs that you download from the Internet. You need to Unblock this DLL with these instructions. Follow the point-and-click unblocking step, or use the streams.exe utility referenced in the post. Don't use the Add-In Manager -- that's probably not going to work for this one.
Hi Chris, thanks for the reply and unblocking the DLL resolved the issue and the add-in works like a charm. Kudos.
I have often told others, now it is time to tell myself to ... RTFM ;-)
How can i download this task for SAS EG 8.1 Version?
I haven't yet built an EG 8.1 version. Fortunately, EG 8.1 does include several enhancements to the Task Status window, so you can see the history of tasks/programs that you've submitted. But I think the report that the task generates is useful, so I'll look at updating it.
I've built an 8.1 and 8.2 version for this task. Instructions for installing are the same, just drop into the %appdata%\SAS\EnterpriseGuide\8\Custom folder. Install just the one version that works with the version of EG that you have.
Is there a 64-bit version of this task for SAS Guide 8.3.3?
There is a version of the task in the ZIP that works with EG 8.3, yes. It's good for 32-bit and 64-bit.
Chris, I followed the instructions and added it to %appdata%\SAS\EnterpriseGuide\8\Custom. I added it in Add_In Manager, but don't know how to implement it. It was really easy to point and click on Tools > Add-In > Project Reviewer, but I don't see Tools > Add-In anymore, just the Add_In Manager.
In EG 8.3 the Custom Tasks don't appear in the Tools menu. You'll find this in the Task browser -- just search for "Project"