Thanks for reading my previous post about the importance of understanding your performance issues before trying to determine the cause. Compiling all the available information about the problem and finding a repeatable SAS process that can be monitored are the first steps toward fixing the issue. Since we now have that, let’s talk about what needs to be monitored and how.
Which tools to use
As you monitor the SAS process, matching what is happening from both the SAS log and hardware monitor logs will help you understand what needs to be improved.
As a standard practice, my team always starts by assuming the performance issue is coming from a constraint or bottleneck in the hardware infrastructure and then determining where that bottleneck might be. To do this, we start with the repeatable SAS process that we've documented and add the following SAS statements to the start of the SAS job. The output from this job will helps us understand more about what is going on with the SAS process and determine which step or steps are taking the most time.
OPTIONS FULLSTIMER SOURCE SOURCE2 MSGLEVEL=I MPRINT NOTES; PROC OPTIONS GROUP=MEMORY; PROC OPTIONS GROUP=PERFORMANCE; RUN; LIBNAME _ALL_ LIST; PROC OPTIONS OPTION=WORK; PROC OPTIONS OPTION=UTILLOC; RUN; DATA _NULL_; %PUT This job started on &sysdate at &systime; RUN; OPTIONS SASTRACE=”,,,d” SASTRACELOC=saslog NOSTSUFFIX;
If we run the SAS process (with the above code included) and turn on some free hardware monitoring tools, then we will have lots of data to help us determine if there are any hardware constraints to the computer system where SAS is running.
- For Windows systems, we turn on PerfMon.
- For UNIX/Linux systems, we turn on iostat, vmstat, sar and several other monitors.
What metrics to monitor
The areas we look at from the hardware monitor logs are:
- CPU busy
- Memory utilization (i.e. did the process completely use all the memory and swapping occurred)
- IO throughput numbers (including whether or not the disks are busy AND have wait queues).
How to find this information in the hardware monitor logs takes a lot of time to explain, but we have several white papers that you can read to help you start. Here is a list of them:
- Troubleshoot Your Performance Issues: SAS Technical Support Shows You How
- Solving SAS Performance Problems: Employing Host Based Tools
- SAS Performance Monitoring - A Deeper Discussion
When you determine the possible source of the problem, we strongly suggest that if you make a change in your hardware infrastructure, you make only one or two changes at a time. Test the change by rerunning the SAS process to see if the change has helped or hurt performance. It may take many changes to the hardware infrastructure before you see a complete resolution to your performance issue.
Stay tuned for Part III: How to Proactively Prevent Performance Issues. Of course, if you have any questions, please reply to this blog and we will help you out.