Talk about a Halloween nightmare. In a frightening example of bad timing, one SAS shop’s completion of a SAS 9.3 upgrade and replication of their development and production servers coincided with a data storage mount failure in the data center. The result? An inability to restart the metadata server and other SAS servers. SAS Technical Support responded quickly to the situation, and in a Web conferencing session with the SAS Administrator, they diagnosed the problem- corrupted metadata, and provided the solution.

I was a silent observer on that session and was impressed by the speed with which the Technical Support engineers pinpointed the source of the trouble and came up with a resolution. But I was even more impressed with how easily and quickly the SAS Administrator was able to maneuver around the UNIX environment and the SAS Management Console, follow the engineers’ instructions, offer suggestions and generally make things easier for the engineers to do their diagnostic work. Best of all, at the end of the session, the engineers complimented the SAS Administrator, saying that she was one of the best they’d ever worked with.

This experience led me to appreciate how the SAS Administrator is, in a real sense, the bridge between the SAS users and the IT organization. A strong, knowledgeable SAS Administrator can truly make the difference between a struggling SAS shop which is isolated from IT and a cohesive team whose mutual efforts benefit the company as a whole. The SAS Admin may come from the SAS world, the business world, or the IT world, but he or she must develop a familiarity with all three to adequately represent the SAS shop.

What skills and sound practices did this SAS Administrator demonstrate that were so impressive?

Expertise in the operating system is much more than a “nice to have.” SAS software runs on Windows, UNIX, and mainframe platforms, and many SAS shops have more than one of these in their data center. The ability to navigate across directories, view directory contents, copy files from one place to another, view file contents, route output to a specified location, change file names and make use of a native editor may not be exercised daily. But when performing an upgrade or migration—and certainly when working with SAS Technical Support to diagnose and fix a problem— these skills are really important. Not that I’m about to muck around in any Unix environments anytime soon, but after that Web conference, I was motivated to pull out my old copy of The UNIX Programming Environment (Kernighan and Pike), blow the dust off the spine, and flip through the pages! Yup, there’s grep, on page 102 …

Where are the logs? SAS generates a plethora of logs. In addition to the SAS programming log that displays information about your SAS program’s syntax, input files and output results, there are remote services logs, metadata server logs, object spawner logs,  Pooled Workspace server logs, SAS Stored Process logs and more.  Knowing where they’re all located and the information they contain is the best way for the SAS Administrator to assess how well things are working, and, in the case of a problem, diagnose the situation. If the problem warrants SAS Technical Support assistance, the Tech Support engineer assigned to the track will certainly ask for these logs as well.

Generate and keep backups. Restoring the metadata was a simple matter in this situation because this SAS shop makes backups of their metadata on a daily basis, and these backups are kept for a prescribed length of time. It was possible to select an image from the day preceding the data storage mount failure and use it to restore the metadata repository. Having backups is critical in a production environment, where any lost metadata updates can be very significant.

Keep a record of the steps you follow and the changes you make during the debugging process so you can back out. Working through the debugging process with SAS Technical Support may include adding additional logging instructions to different configuration files, temporarily changing permissions and adding user groups in the SAS Management Console, and other such tweaking. This SAS Administrator carefully made a backup copy of every file that was modified and, in an open Notepad window, kept a running record of the steps taken. A few times she said, “Wait! Hold it! I want to write that down!” Before the session ended, she reviewed what and why the changes were made with the SAS Technical Support engineers. Her notes served as a script providing the steps for backing out of the temporary changes and ensuring that nothing was overlooked in returning the environment to its standard form. Should a similar problem occur in the future, she can use the script to remind her how to solve it.

What else was going on at the time of the failure? This SAS Administrator has close working ties with the IT people responsible for their company’s databases, networks, hardware, system monitoring and so on.  A few phone calls and emails revealed the storage mount failure, providing a major clue to the source of the problem for the Tech Support engineers to use in devising a solution.

The above skills and practices are not the only ones that SAS Administrators need, but they represent a good subset that are necessary to keep a SAS environment running from day to day. Does your SAS shop, especially if it’s a large one, have a SAS Administrator? Is he or she as comfortable with these tasks as this SAS Administrator was? The SAS Administrator is a key contributor to your organization’s success, and I salute them!