This article is co-authored with my colleague Russ Tyndall, a Senior Principal Technical Support Analyst at SAS.

As defined in the SAS^® 9.4 Stored Processes: Developer's Guide, Third Edition, a stored process "is a SAS program that is stored on a server and defined in metadata, and which can be executed as requested by client applications." One of the benefits of using stored processes is that client applications always have the latest version of the code. In addition, stored processes provide enhanced security and application integrity.

Have you ever submitted a stored process, and instead of the expected output, you saw errors or no output at all? Depending on how you submit the stored process, various logs are available to assist you with debugging.

This article provides guidance for understanding which situations call for which logs, where to find each log, and what you should look for in each log. The article uses two clients as examples: SAS^® Enterprise Guide^® and the SAS^® 9.4 Stored Process Web Application.

The article is divided into two sections:

• Frequently Used Logs
• Infrequently Used Logs

Frequently Used Logs

The logs that are described in this section are the most prevalent logs that you request when a stored process does not execute properly or when there seems to be a problem with the stored-process server.

SAS^® Object Spawner log

What

The SAS Object Spawner log records anytime that the spawner tries to start or stop a stored-process server, a workspace server, or a pooled workspace server. It also records when a request is redirected to a running stored-process server or a pooled workspace server or to a SAS® Grid Manager server.

The syntax for this log's name is ObjectSpawner_yyyy-mm-dd_machine-name_process-ID.log.

Where

The default locations for the object-spawner log are as follows:

• Microsoft Windows operating environments: SAS-configuration-directory\Lev1\ObjectSpawner\Logs
• UNIX operating environments: SAS-configuration-directory/Lev1/ObjectSpawner/Logs

If the logs are not in the directories that are listed above, you can check the logconfig.xml file that resides in the SAS-configuration-directory\Lev1\ObjectSpawner\ directory. That file contains a parameter called FileNamePattern that determines the location of the object-spawner log, as shown in this example:

Why

Here are some reasons why you should check the object-spawner log:

1. Performance problems occur within stored processes.
2. Servers do not start or servers stop working. The stored process server, the workspace server, and the pooled workspace server are all started by the object spawner.
3. Running a stored process returns no results and no errors.
4. Users do not have permission to start the server or do not have ReadMetadata permission on their application server context (for example, the SASApp application server).
5. The connecting user has to wait longer than the availability time-out of 60 seconds, and the stored process fails.
6. A server is not available on which to run a stored process.

The following error is an example of one that you might find in this log:

The launch of the server process failed due to a problem with the processing of the SAS logging facility configuration file (LOGCONFIGLOC).

This error might occur because the file that is specified for the -LOGCONFIGLOC option cannot be processed. The option either is invalid or it cannot be accessed.

For more examples of errors that you might see in the object-spawner log, see the section Object Spawner Messages in "Appendix 1: Object Spawner and SAS OLAP Server Messages" in the SAS^® 9.4 Intelligence Platform: Application Server Administration Guide.

You can also enable more detailed logging within the object-spawner log by following the steps in Enable More Detailed Logging for SAS Object Spawner Troubleshooting in "Chapter 10, Administering Logging for SAS Servers" of the SAS^® 9.4 Intelligence Platform: System Administration Guide, Fourth Edition.

Example and possible cause

Let’s look at an example of how the object-spawner log can be helpful.

Suppose that the stored-process server fails to validate. Within the object-spawner log, you might see an error like the following:

Error authenticating user domain\sassrv in function LogonUser. Error 1326 (The user name or password is incorrect. ).
 
The credentials specified for the SASApp – Stored Process Server (A5WN99NR.AZ000007) server definition failed to authenticate. Therefore this server definition will not be included.

With this type of error, you might have an invalid sassrv password. You can check the password by trying to log on to the server directly with the sassrv user ID. If the logon is successful, open SAS^® Management Console and verify that the password is correct in the metadata, as follows:

1. In SAS^® Management Console, select User Manager.
2. Locate the SAS General Servers group. Then, right-click and select Properties.
3. Click the Accounts tab.
4. Select the sassrv user ID. Then click Edit.
5. Update the password to what was successful on the operating system.
6. Restart the object spawner.
7. Check the logs again to verify that the error is resolved.

Object-spawner console log

What

The object-spawner console log, available only on UNIX platforms, does not actually contain information about the object spawner. Instead, this log contains STDERR and STDOUT messages about the applications that are launched by the object spawner. The syntax for this log's name ObjectSpawner_console_machine-name.log.

Where

Under UNIX, the default location for the object-spawner console log is SAS-configuration-directory/Lev1/ObjectSpawner/Logs.

Why

Here are some reasons why you should check this log:

• SAS modules are missing. The program tries to run SAS modules that are not present in your environment.
• Possible memory issues occur.
• Storage or space issues occur.
• Possible encoding issues occur.
• Library permission issues occur.

The following error is an example of one that you might find in the object-spawner console log:

ERROR: Could not find extension: (sasgis)

This error can occur when someone or some component runs code that either checks for the existence of certain SAS modules or that contains references to certain SAS modules that are not present in your environment. Because these modules do not exist, errors are written to the log. This behavior is expected, and it is not indicative of any larger issue. So, you can ignore these messages.

Example and possible cause

Let’s look at an example of how the object-spawner console log can be helpful.

Suppose that your program generates a segmentation violation, but there is no indication as to why that happens. Within the object-spawner console log, you might see an error like the following:

ERROR: No space left on device

This type of error is an indication that you might be running out of disk space. Try adding more space, point to a network drive, or use smaller files.

SAS^® Stored Process Server log

You need to enable verbose logging to see every stored process that runs through the server. For more details about verbose logging, see SAS Note 34114, "Creating a detailed SAS^® Stored Process Server log by default."

What

The syntax for the SAS Stored Process Server log name is SASApp_STPServer_yyyy-mm-dd_machine-name_process-ID.log.

Where

The default locations for the stored-process server log are as follows:

• Windows: SAS-configuration-directory\Lev1\SASApp\StoredProcessServer\Logs
• UNIX: SAS-configuration-directory/Lev1/SASApp/StoredProcessServer/Logs

Why

Here are the reasons that you might need to check this log:

• Performance problems occur.
• You do not receive any results. That is, the stored process runs, but it does not return an error, a warning, or the expected results.

Here is an example of an error that you might see in the log:

ERROR: A lock is not available for library.dataset.data
ERROR: Lock held by process 26834

This error indicates that the data set that is shown in the error cannot be locked for processing because the lock is held by another process. Further investigation of the stored-process server log that contains process ID 26834 shows the DATA step that is also reading from the same data set, which causes the lock.

Here is another error that you might see in this log:

ERROR: No logical assign for filename _WEBOUT.

Possible causes

The two main reasons for this error are as follows:

• The %STPBEGIN and %STPEND macros are enabled.
• The item Package is selected under the Result capabilities section (via Execution Options ► Result capabilities in either SAS Management Console or SAS Enterprise Guide)

The object-spawner console log is often needed in conjunction with the object-spawner log so that you can evaluate the communication between them.

SAS^® Stored Process web-application log

What

The SAS Stored Process web-application log resides on the middle-tier machine, and the syntax for the log's name is SASStoredProcess9.4.log. (Note: The 9.4 in SASStoredProcess9.4.log will change with new releases.)

Where

The default locations for this log are as follows:

• Windows: SAS-configuration-directory\Lev1\Web\Logs\SASServer1_1
• UNIX: SAS-configuration-directory/Lev1/Web/Logs/SASServer1_1

Why

Here are some scenarios for which you might want to check this log:

• HTTP errors are displayed in the browser when you submit the stored process from the SAS Stored Process Web Application.
• Pages in the stored-process web application (for example, the welcome page, index page, or custom input form) do not load or they take a long time to load.
• A dynamic prompt cannot load.

Examples

The following error occurs when dynamic prompts from a data set and the SAS General Servers user group are denied ReadMetadata permission in SAS Management Console for that specific stored process:

2021-07-30 15:01:43,494 [tomcat-http--10] WARN [sasdemo] com.sas.prompts.valueprovider.dynamic.workspace.PromptColumnValueProvider - Cannot resolve data type
2021-07-30 15:01:43,505 [tomcat-http--10] ERROR [sasdemo] com.sas.prompts.valueprovider.dynamic.workspace.PromptColumnValueProvider - Unable to find physical table
com.sas.storage.exception.ServerConnectionException: Unable to find physical table
	at com.sas.prompts.valueprovider.dynamic.DataProviderUtil.getLibrary(DataProviderUtil.java:163)
	at com.sas.prompts.valueprovider.dynamic.workspace.PromptColumnValueProvider.setConnection(PromptColumnValueProvider.java:815)
	at com.sas.prompts.valueprovider.dynamic.workspace.PromptColumnValueProvider.getValuesAsList(PromptColumnValueProvider.java:647)
	at com.sas.prompts.valueprovider.dynamic.workspace.PromptColumnValueProvider.getValues(PromptColumnValueProvider.java:633)

The following pop-up message occurs because the SAS General Servers user group or the sasdemo user ID are denied permission to the data set that is needed for the prompts for a stored process.

When you run a stored process from the SAS Stored Process Web Application, it is often helpful to add debugging options to the end of the URL by using the reserved macro variable _DEBUG. The following example URL demonstrates how to use the TRACE and LOG options to obtain pertinent information in the log that is produced in the browser.

http://your.web.server:8080/SASStoredProcess/do?_program=/STP_Examples/test1&_debug=trace,log

For a complete list of _DEBUG= values, see List of Valid Debugging Keywords in "Chapter 7: Building a Web Application with SAS^® Stored Processes" in the SAS^® 9.4 Stored Processes: Developer's Guide, Third Edition.

Workspace-server log

What

You must request logging for the workspace-server log because it is not one that is enabled, by default. The syntax for the log's name is SASApp_WorkspaceServer_yyyy-mm-dd_machine-name_process-ID.log.

Where

As mentioned earlier, you must request logging for this log, as follows:

• Windows: In the sasv9_usermods.cfg file that resides in SAS-configuration-directory\Lev1\SASApp\WorkspaceServer\ directory, add the following command to turn on logging:

  -logconfigloc SAS-configuration-directory\Lev1\SASApp\WorkspaceServer\logconfig.trace.xml"

  This setting generates the log file in SAS-configuration-directory\Lev1\SASApp\WorkspaceServer\Logs.

• UNIX: In the sasv9_usermods.cfg file that resides in SAS-configuration-directory/Lev1/SASApp/WorkspaceServer/logconfig.trace.xml"
  to /Lev1/SASApp/WorkspaceServer/ directory, add the following command to turn on logging:

  -LOGCONFIGLOC SAS-configuration-directory/Lev1/SASApp/WorkspaceServer/logconfig.trace.xml"

  This setting generates the log file that resides in SAS-configuration-directory/Lev1/SASApp/WorkspaceServer/Logs.

Why

If you run the stored process from the SAS Stored Process Web Application with the server set to the workspace server, you need to consult this log for any errors or warnings. Running the stored process from SAS Enterprise Guide produces a log directly in the application. You need to consult the workspace server log in either of these circumstances:

• when Default server is selected as the server type in the stored-process properties
• when you submit the stored process from SAS Enterprise Guide

Pooled workspace-server Log

What

The syntax for the pooled workspace-server log's name is SASApp_PooledWSServer_yyyy-mm-dd_machine-name_process-ID.log.

Where

The default locations for this log are as follows:

• Windows: SAS-configuration-directory\ Lev1\SASApp\PooledWorkspaceServer\Logs
• UNIX: SAS-configuration-directory/Lev1/SASApp/PooledWorkspaceServer/Logs

Why

The pooled workspace server loads dynamic prompt values in the SAS Stored Process web application. So, you need this log if a dynamic prompt does not load.

SAS^® Metadata Server Log

What

The syntax for the SAS Metadata Server log' name is SASMeta_MetadataServer_yyyy-mm-dd_machine-name_process-ID.log

Where

The default locations for the metadata-server log are as follows:

• Windows: SAS-configuration-directory\Lev1\SASMeta\MetadataServer\Logs
• UNIX: SAS-configuration-directory/Lev1/SASMeta/MetadataServer/Logs

Why

This log is helpful for issues with the LIBNAME META engine and metadata permissions.

The following error is written to the metadata-server log when no metadata identity is associated with the user name and password.

ERROR [00000779] 28:billyw - User Folders cannot be created or retrieved if connected user ID is not a person identity. Must be connected as a person identity or valid person name must be passed in request.

The following messages might also be written to this log when you try to open SAS Management Console and when the user billyw requires log on as a batch job user rights at the operating system level.

20100122:13.00.36.82: 00002410:NOTE:    User tbanks probably does not have the right to log on as a batch job.
20100122:13.00.36.82: 00002410:ERROR:   Error authenticating user tbanks in function LogonUser.  Error 1385 (Logon failure: the user has not been granted the requested logon type at this computer. ).
20100122:13.00.36.82: 00002410:ERROR:   Access denied.

Infrequently Used Logs

Windows Event Viewer log

The Windows Event Viewer contains several logs that are used less frequently than the aforementioned logs. This fact in no way diminishes their effectiveness when you are troubleshooting stored-process issues. SAS Technical Support will request these logs from you for an issue if it is necessary.

Although the event viewer contains several logs, SAS Technical Support reviews only the application log, the system log, and the security log for errors that are related to start-up and other failures in the SAS Stored Process Server (or any SAS server).

If the sassrv user ID cannot write to the Logs directory, you receive a message in this log.

If a server is stopped or started, those behaviors register in these logs. Hopefully, this review of the various logs that are related to debugging stored-process issues will assist your troubleshooting efforts.

ERROR_yyyy-mm-dd-time.log

What

This web-server log is useful when you have catastrophic failures in loading or running a stored process in a SAS web application. The syntax for this log's name is ERROR_yyyy-mm-dd-time.log.

Where

The default locations for this log are as follows:

• Windows: SAS-configuration-directory\Lev1\Web\WebServer\logs
• UNIX: SAS-configuration-directory/Lev1/Web/WebServer/logs

Why

Here are some reasons why you should check this log:

• Java errors are returned in the browser when you try to load or run a stored process using the SAS Stored Process Web Application.
• Connection errors occur between the web server and the tc Server.

Here is an example of an error that you might find in this log:

[error] (OS 10061)No connection could be made because the target machine actively refused it.  : proxy: HTTP: attempt to connect to 192.168.x.xxx:8080 (machine-name) failed

This error might occur when your servers are down or when they are restarting and are not active yet.

Example and possible cause

Let’s look at an example of how the ERROR_yyyy-mm-dd-time log can be helpful.

Suppose that you try to run a stored process through the SAS Stored Process Web Application and you cannot load any SAS 9.4 web applications after you enter your user ID and password. If you check this log, you might see errors like the following:

No connection could be made because the target machine actively refused it. : proxy: HTTP: attempt to connect to 192.168.x.xxx:8080 (machine-name) failed
 
ap_proxy_connect_backend disabling worker for (machine-name)

The secondary middle-tier node is in a clustered environment, but it configured incorrectly. For a circumvention, see SAS Note 55904, "You cannot access any SAS® 9.4 web applications when the secondary middle-tier node is in a clustered environment."

Localhost_access_log..yyyy-mm-dd.txt

What

This log shows the sequence and query strings of URLs that are submitted through the SAS Web Application Server (for all web applications that use this application server). This log lists activity sequentially and includes HTTP status codes for each URL. The syntax for this log's name is Localhost_access_log..yyyy-mm-dd.txt.

Where

The default locations for this log are as follows:

• Windows: SAS-configuration-directory\Lev1\Web\WebAppServer\SASServer1_1\logs
• UNIX: SAS-configuration-directory/Lev1/Web/WebAppServer/SASServer1_1/logs

Why

Here are some reasons why you should check this log:

• A page cannot load or the URL is invalid.
• Performance problems occur.
• You need a good way to trace the stored-process activity after you click the Run button in web applications only.

Here is an example of a message that you might see in this log:

"POST /SASLogon/v1/tickets HTTP/1.1" 404 796

Example and possible cause

Let’s look at an example of how this log can be helpful. Suppose that you cannot log on to the SAS Stored Process Web Application. If you check this log, you might see a message like the following:

GET /SASStoredProcess/j_spring_cas_security_check?ticket=ST-9994-zpD2sFdBmayXEuoj4cya-cas HTTP/1.1" 401 802

In this example, you can see that the issue is a 401 HTTP return code.

If the servers have just been restarted, try waiting a little longer for them to start up. If that is not the issue, the server might be down. The best approach is to stop and then restart the servers. Because of dependencies, it is important to start the servers in the correct order. You can find the correct order in Overview of Server Operation, in "Chapter 6: Operating Your Servers" of the SAS^® Intelligence Platform: System Administration Guide, Fourth Edition.

Access_yyyy-mm-dd-time.log

What

This log shows the sequence and query strings of URLs that are submitted through the SAS Web Server. The syntax for the name of this log is access_yyyy-mm-dd-time.log.

Where

The default locations for this log are as follows:

• Windows: SAS-configuration-directory\Lev1\Web\WebServer\logs
• UNIX: SAS-configuration-directory/Lev1/Web/WebServer/logs

Why

Here are some reasons why you would check this log:

• A connection error occurs when you submit a stored process.
• Java errors occur in the browser.
• When you evaluate a performance problem, timestamps in the log confirm when the web server received the request and what was in the request.
• The output that is produced by the stored process does not match the expected output based on the prompt selections that are available when you submit the stored process.

Here is an example of a message that you might see in this log:

"GET /SASTheme_default/themes/ThemeXMLFiles.config HTTP/1.1" 503 299

This error might occur because the servers are down or are in the process of restarting and are not active yet.

Example and possible cause

Let’s look at an example of how the access_yyyy-mm-dd-time log can be helpful.

Suppose that you are trying to run a stored process through the SAS web application and you receive a connection error. If you check this log, you might see a message like the following:

"GET /SASLogon/proxy?pgt=TGT-6-aGakal9b2VGBg3dnNTbbwiVALOWqBSem9E3cVWehDD35vegmxI-cas&targetService=http%3A%2F%2FLazySecurityContext HTTP/1.1" 502 407

The message above shows a 502 HTTP return code. This error indicates that the server, while acting as a gateway or proxy, received an invalid response from the upstream server. If the servers were just restarted, try waiting a little longer for them to start back up. If that is not the issue, then a server might be down. The best approach is to stop and restart the servers. Because of dependencies, it is important to start the servers in the correct order. You can find the correct order in Overview of Server Operation, in "Chapter 6: Operating Your Servers" of the SAS^® Intelligence Platform: System Administration Guide, Fourth Edition.

Server.log

What

This log is helpful if a stored-process web application generates an HTPP, browser, or generic error in the browser when it loads a page or results that are returned from a stored process.

Where

The default locations for this log are as follows:

• Windows: SAS-configuration-directory\Lev1\Web\WebAppServer\SASServer1_1\logs
• UNIX: SAS-configuration-directory/Lev1/Web/WebAppServer/SASServer1_1/logs

Why

Here are some reasons why you would check this log:

• The SAS Stored Processes web application is not working.
• Requests time out.
• A configuration change is made to one of the web applications (for example, an increase to the time-out for the stored-process web application).
• A generic error is displayed in the SAS Stored Processes web application when you run a stored process.

Here is an example of a message that you might find in this log:

java.io.IOException: Cannot bind to URL

One possible reason for this type of error is network issues at the site where the stored process is run.

Example and possible cause

Let’s look at an example of how the server.log file can be helpful.

Suppose that you want to run a stored process through the SAS Web Application and you receive a generic error that says The system is experiencing problems. Please contact your system administrator. If you check this log, you might see a message like the following:

ERROR (ContainerBackgroundProcessor[StandardEngine[Catalina]]) [org.apache.catalina.core.ContainerBase] Unexpected death of background thread ContainerBackgroundProcessor[StandardEngine[Catalina]]
java.lang.OutOfMemoryError: Java heap space

There is insufficient memory available to process the request. As a possible workaround, you need to modify the setenv.bat file that resides on Windows in SAS-configuration-directory\Lev1\Web\Webappserver\Sasserver1_1\bin\ or the setenv.sh in SAS-configuration-directory/Lev1/Web/WebAppServer/Sasserver1_1/bin/ on UNIX. You need to edit the JVM_OPTS value by changing the -Xmx4096m -Xms1024m parameter to the following:

-Xmx8092m -Xms8092m

Making this change requires the restart of the SAS Web Application Server if only the webappserver script is updated.

Conclusion

Hopefully, this review of logs that are related to debugging stored-process issues will assist your troubleshooting efforts. When you request help from SAS Technical Support, these are some of the logs that you will be asked to send so that Technical Support can better determine the cause of your problem.

Learn more

• Jobs: Stored processes in Viya
• Creating Stored Processes - SAS Support
• Developer's Guide: SAS Stored Processes