So, you want to do two things at once?
When you use SAS Enterprise Guide to create a project, the work items in that project typically run one at a time, in series. For example, consider the following process flow:
In this flow, SAS Enterprise Guide will run the tasks from left-to-right, top-to-bottom. I've labeled the task items with 1,2,3, etc. to show in what order they'll be run.
Even though some items can theoretically run in parallel (for example, items 2, 4, and 5), SAS Enterprise Guide is using just one SAS workspace session, and it can handle just one program at a time. The question becomes, how can we divide this work so that we use multiple SAS sessions to get the work done quicker?
One approach that customers often use is to divide the project into multiple project files, and then run multiple SAS Enterprise Guide sessions at one time. Each SAS Enterprise Guide instance connects to its own SAS workspace session, so you can submit multiple blocks of work that way. However, it can be pretty darn inconvenient to manage the multiple projects simply to get some parallelism to play out.
Here's another approach: ask your SAS administrator (small shops: that might be you) to define a second logical SAS server. For example, if you have a SAS server named "SASApp", ask for a second identical server named "SASApp2". You don't need any additional hardware for this! A logical server is simply a defined way to access the services of your SAS installation, and you can camp as many of these as you want on the same physical hardware. You do want to make sure the two server definitions have "resource affinity". That's a fancy way of saying that they need access to the same data libraries, files, etc. so that they can work in exactly the same way.
(I won't go into all of the details about how to do this, but you use SAS Management Console to get this done. Add a SAS Application Server of type Workspace, select an available port, use the same SAS command and startup files to launch the SAS session, and add the new server to the SAS object spawner Servers list -- and then restart the SAS object spawner to have it take effect.)
With an additional SAS workspace to use, we can now reallocate some of the project work to allow multiple tasks to run simultaneously. Take a look at this revised flow:
The top branch references programs and data on SASApp, while the bottom branch is associated with SASApp2. Items 1a, 2a, and 3a all must run in serial on SASApp, our original server. Meanwhile, items 1b, 2b, and 3b can run in their own parallel sequence on SASApp2, the session that was added. Optionally, you can use custom user-defined links to join the two flows later, ensuring that one cannot proceed until the work is complete from the other one. Here is an example of that:
In this case, item 4b runs on SASApp2, but SAS Enterprise Guide won't run it until the work from item 3a is complete. That's because I linked 3a to 4b, indicating a dependency.
If you need concurrency that really scales, you'll want to look at SAS grid computing, which, of course, you can also leverage in SAS Enterprise Guide with just a few tweaks.
5 Comments
Hi Chris... a bit late replying to this post, sorry, but I found it while searching for something else.
I'd like to suggest what would be a killer feature for people who use EG as an interface for SAS programming... and there are probably a lot; I know that almost everything I write needs to be coded rather than using EG tasks, since I use macros heavily, and other things like hash tables, etc.
It seems ridiculous to have to open multiple projects to submit multiple bits of code to run simultaneously, and the second logical server you suggest above is a hack - you need an amenable admin, and it doesn't help if you want to submit 10 pieces of code or tasks simultaneously.
Why can't EG manage its own "server instances", and instead of choosing which server to submit to, you choose which server instance (or both), and EG could create as many connections as required. A simple management window similar to the task status window could be used to see what is queued on which server and which instance.
Great article! As a SAS administrator, not one I want all my programmers to find out about... That being said, the ability to take advantage of any extra room on a SAS server can be beneficial for programmers and developers if used wisely. Do you have a blog posting about the "Allow Parallel Execution on the Same Server" option in EG 5.1? (and later versions?) I have experimented with this a little bit, but I cannot find an experts take on what you can do with this and its limitations. (The biggest limitation I have found being separate-temporary work directories.) I feel like that option in EG is so close to what some of my EG programmers have been asking for.
Shaun,
Check out Edoardo's post about EG and the SAS Grid environment. Even better, you might find his SAS Global Forum paper to be useful as it describes the many options to do "many things at the same time" in EG.
I find that all macro variables get nuked in the 2nd parallel running program...any way to fix that?
Macro variables won't be shared among "SASApp" and "SASApp2" -- is that what you're expecting? Or are you saying that macro variables in one session aren't present when a second program runs on the same session? Do you have SAS Grid Computing as well? That could be complicating the matter.