My team and I are often called on to help customers optimize SAS system performance, particularly when the root cause of performance problems is hard to track down. This is the first of a two-part article on practices and tools we’ve found most useful.
The first step in diagnosing and solving performance problems is to understand the complaint thoroughly. Compiling all the available information from the user will help you more accurately target diagnostic testing:
- Is there a problem with a specific task?
- Does a task run out of resources causing it to fail?
- Is it just too slow?
- Does it consume far more compute resources than is expected?
- Did it used to run faster than it is today?
Or, are the problems more pervasive across the many processes and users?
First clues to the problem are usually manifest by poor test or application response times. It’s often helpful to witness the problem first-hand to fully understand it.
This step may require a subsequent setup and run of the application or process, particularly if it is non-recurring task or activity. It is best to perform this activity under the same circumstances as the original problem. Ensuring that conditions are duplicated may require:
- Running the exact same procedures or code on the same data. This test run should be documented in an identified test bed so that the test activity and data will remain constant throughout the diagnostic cycle. Documentation should include the problem complaint and target behavior for solution definitions listed above, as well as the executing code or application and the data model documentation.
- Running under the same physical system, system “load” or operating conditions. On a multi-user system this means testing with the same number of users with the same level of activity. You will want to account for cyclical activity factors such as those influenced by time of day, day of week, day of month, business cycle activity, and so forth.
Once the complaint is clearly established and documented, then you can undertake processes to determine the root cause or causes and initiate correction. If the problem is established as recurring or persistent, it can be measured and diagnosed using monitors from within SAS along with a wide variety of hardware monitors. Stay tuned for Part II: What Monitor Tools to Use.
A Practical Approach to Solving Performance issues with SAS
- Troubleshoot Your Performance Issues: SAS Technical Support Shows You How
- Solving SAS Performance Problems: Employing Host Based Tools
- SAS Performance Monitoring - A Deeper Discussion
- Logging 101: Leveraging the SAS Enhanced Logging Facility
Pingback: Server settings: cost savings vs. performance - SAS Users Groups
Pingback: Improving performance: Determine the cause - SAS Users Groups
Would you also suggest checking what other software might have been installed or updated recently since the problem started? In the past I've encountered anti-virus software scanning large SAS tables and other I/O intensive software causing problems for SAS installations.
I'm looking forward to reading part 2 on monitoring tools. I'd also be interested in hearing whether you suggest or use load testing & simulation tools. A while back I used Apache JMeter to load test a SAS Stored Process server and would be keen to hear about any other such tools that you have used or recommend.
There are ways to have your anit-virus software exclude scanning all SAS data file. These are discussed in paper that we have produced regarding tuning your Windows system for SAS. Here is a link to the paper: http://support.sas.com/resources/papers/WindowsServer2008ConfigurationandTuning.pdf There are lots of other good tips for running SAS on Windows in the paper as well.
The simulation tools that we at SAS use the most for load testing is LoadRunner and JMeter. They work really well and have some very nice analysis features to them. The one thing I would strongly recommend you do with your load testing is make sure what you are testing is how your SAS users use SAS. "Made up tests" can give you inacurate results and if you tune for these results you may make the hardware run slower for your actual SAS users. (This has happened many times with SAS customers over the years.)
Thanks for the comments and keep the questions coming.