You might be the sort of person who loves to wait indefinitely. You visit the DMV regularly to tweak your auto registration. You queue up in the supermarket checkout line behind the customer most likely to require "price checks". You map your daily commute along the routes that offer the least certainty for traffic volume and faulty signals.
But most of us, even as we say we enjoy "living in the moment", like to have some idea about when our current activity will end and we'll be free to move on to the next activity. It's that sense of progress that keeps us going, especially during the work day.
When you run a big long SAS program in SAS Enterprise Guide, and you open the task status window to see how it's going, do you like to see the status as "Running...", whether it's been running for 10 seconds or 10 minutes or an hour? Or would you rather see something more meaningful, such as "10 out 12 steps complete - almost done..."?
If you answered the latter, then you will enjoy the feature in SAS and SAS Enterprise Guide that allows you to instrument your SAS programs to report the current progress within the task status window. This trick is made possible by a SAS global statement, SYSECHO.
The remainder of this post provides instructions and a code example for using the SYSECHO statement. (5 more short paragraphs plus a code block, for those keeping track of blog post status.) You can sprinkle the SYSECHO statement throughout your programs to add more insight into your task status window. However, even though you can use the SYSECHO statement anywhere (it is a global statement), it's most effective if you place it just inside the DATA step or PROC step block.
SAS Enterprise Guide already provides status updates when your program is processing; it reports a status change each time a new step boundary (such as a DATA or PROC step) begins. To preserve a useful SYSECHO status, you need to place the statements after a DATA step or PROC statement. Otherwise, your informative SYSECHO text will be overridden by the more generic "step boundary encountered" status.
When used inside a DATA step, only the last SYSECHO statement will be reflected in the status window. Why? Because SAS global statements are processed at the time that a DATA step is compiled, not as it's being run. So even though your DATA step statements might seem to take a while to run, the SYSECHO statements are not processed in serial within the step -- they are processed all at once, in the beginning.
However, for certain interactive SAS procedures such as PROC SQL, you can use multiple SYSECHO statements to show more status information between statements.
This example program shows a SAS program that has been adorned with SYSECHO statements designed to communicate status information. The program is somewhat contrived, including some SLEEP function calls to simulate longer operations. You can paste this into your SAS Enterprise Guide session and see how it works. Be sure to select View->Task Status to make the task status window visible.
data class (drop=x); sysecho "In the first DATA step"; set sashelp.class; x = sleep(1); run; proc sql; sysecho "Creating the first table, OUT"; create table out as select name, /* Calculation */ (sleep(1)) AS zzzz from class where age>14; sysecho "Creating the second table, OUT2"; create table out2 as select name, /* Calculation */ (sleep(1)) AS zzzz from class where sex="M" and age>14; quit; %macro makedata(count); %do i = 1 %to &count; data outdata&i; sysecho "Making data step &i of &count"; zzz = sleep(3); run; %end; %mend; %makedata(5); |
11 Comments
Hello Chris,
It is necessary to add that SYSECHO should be inserted into LONG executed data step or procedure otherwise it will be overritten by regular status. I think that for user friendly solution it would be good to maitain a list of consecutive statuses in STATUS window for user defined statuses.
Sincerely,
Alexander
I agree; a way to keep track of the SYSECHO statements that have been issued would be a good thing.
I think I'd prefer that they be written to the project log.
Thanks Jack. Per what you saw at WUSS, I'll also be writing a follow-up post about how DOSUBL can make DATA step tracking (and crude debugging) possible within EG.
Pingback: Implement BY processing for your entire SAS program - The SAS Dummy
Pingback: How much time will your process flow take to run? - The SAS Dummy
Love the sysecho statement! Particularly when you are running a large piece of code that takes a long time. I have a simply sysecho macro:
%macro sysecho(msg);
sysecho "&msg %sysfunc(datetime(),datetime19.)";
%mend;
and in my code use (for example): %sysecho(Extract from Oracle);
That way I can keep track of which piece of code is executing, and how long it's taking..
Pingback: Tracking progress in your program with SAS Enterprise Guide: another trick - The SAS Dummy
Hey Chris!
I've maybe gone down the rabbit hole on this but I have quick question/observation. I tried using the SysEcho and while it works great in some situations I have discovered one where it doesn't seem to work. Wanted to get your thoughts on what I've found and see if you had any suggestions/alternatives I should try.
Here's the situation it doesn't seem to work in.
- Main program executes a program via an include statement.
- Included program has multiple RSubmits with each wrapping an include statement that's being executed in parallel. Within these included programs I have SysEcho statements.
The SysEcho statements in the included programs being executed by the RSubmits running in parallel sessions never make it back so to speak to the EG Task Status window which is a bummer. It would have been cool/helpful if they did. Any alternative suggestion for getting info back to the task status window for this type of situation? If not, the SysEcho and DoSubl are still really helpful tricks for getting info on long running programs.
Thanks!
-Andrew
Hi Andrew,
Sounds like you did a good amount of research and testing. The way SYSECHO works for EG -- it raises an event from the SAS Workspace and passes the text along, and then EG displays that in the status window. With RSubmit, the process is running elsewhere...not native in the SAS Workspace...so the event is never triggered in a way to reach EG. That probably confirms what you've already found -- too many layers to hop and the info doesn't make it all of the way back.
Thanks Chris!
That's exactly what I was thinking. Too many hops and programs running elsewhere. Is it all possible to capture that the RSubmit sessions have been spun up and signed off? Just wondering since info is written to the log regarding the sign on and off from the remote sessions. If it was somehow possible to capture those events and display it in the Task Status window that would at least provide some feedback regarding these programs? Does that make sense? I'm using some code and macros from Doug Haigh's paper "Divide and Conquer - Writing Parallel SAS Code To Speed Up Your SAS Program" (SAS1935-2015). As part of the code and macros in the paper you create a bunch of variables related to each session being spun up so was curious if there is some way to possibly leverage those variables or the info being written back to the log in the Task Status. Even this would be really helpful when it comes to knowing what's going on with my programs that are running.
Here are some of the example notes in the log:
NOTE: Remote signon to MYSESS4 commencing (SAS Release 9.04.01M1P120413).
NOTE: Remote signon to MYSESS4 complete.
NOTE: Background remote submit to MYSESS4 in progress.
NOTE: Remote submit to MYSESS4 commencing.
NOTE: Remote submit to MYSESS4 complete.
NOTE: Remote signoff from MYSESS4 commencing.
NOTE: Remote signoff from MYSESS4 complete.
Thanks,
-Andrew
I guess if you could insert some SYSECHO statements after those SIGNOFF points, as long as they are running in the "master" Workspace. We might need to reach out to Doug for advice.