In the past, configuring a new autoload library for SAS Visual Analytics was a manual process involving steps to create very specific folder structures and to copy and edit scripts and SAS programs. No more! Updates to the SAS Deployment Manager have added a new task that creates and configures new autoload libraries automatically, which should help SAS administrators add autoload libraries to their system while reducing the possibility of making a careless mistake.
For those unfamiliar with autoload, I am talking about a directory into which a user can drop a spreadsheet, delimited file, or SAS Data Set and have that file automatically loaded for use in a SAS Visual Analytics environment.
So let’s see how this works. For a distributed environment, we need to make sure we start the SAS Deployment Manager on the compute tier machine. Looking through the tasks, you should see “Configure Autoload Directory for SAS Visual Analytics.”
After connecting to the SAS Metadata Server with administrative credentials, you will be prompted with a SAS LASR Artifact selection. The first two selections enable you to define entirely new autoload locations serviced by brand-new LASR servers while the last selection enables you to add an autoload library to an existing LASR server. I am going to work under the assumption that I am creating a brand new Non-Distributed LASR server that will surface my brand new autoload library so I will select the first option.
In subsequent dialogs I can provide the parameters that define a new LASR server, a new LASR library, and an autoload location where users will deposit data to be loaded.
The SAS Deployment Manager will enter configuration processing, after which we should see confirmation that our autoload configuration was successful.
The summary document generated by the SAS Deployment Manager indicates that we have created a new SAS LASR server and SAS LASR Library, a new drop-zone with the requisite folder structure, and a new set of scripts with which to manage the autoload library.
All that is left for the SAS administrator to do is to execute schedule.bat for the new autoload library to initiate the job that will check the autoload drop-zone for new data. By default that will occur every 15 minutes, but administrators can edit schedule.bat before executing it to adjust that setting.
23 Comments
Scott,
Okay sorry maybe a better question for this newbie, I opened the help button on the D.M. form with port on it and it said to use the port that SAS clients connect to the environment or something similar, sorry but this product uses so many ports I am not sure what that port is, can you tell me how to find the port that this help is talking about? TIA -KJ
Hi KJ,
The port 10014 in my example is the port for the new LASR server that I want created during autoload configuration. If you choose either of the first two selections in the LASR Artifacts dialog you will need to provide a port number that is not already used by any processes, not just those that SAS uses. I usually use SAS Management Console to check which ports my existing LASR servers are using and then choose a new one. The default port for LASR is 10011 so if I'm configuring a Lev1 deployment I'll typically increment the tens place for additional servers (10021, 10031, etc) but that's just a personal preference. SAS usually increments the ones place for port numbers to coincide with the Lev number so leaving the last digit as a 1 follows that pattern. The only technical requirement is that it be unique and unused. I agree, there are lots of ports to keep up with but following a pattern will help keep everything consistent. Thanks for reading.
Scott,
Okay so I have SAS 9.4 M3 SAS VA 7.3 and can see in the Deployment Manager the forms you discuss here, however you do not give any background on the Port you chose and why. I am trying to decide what port # to use on the form you show 10014, is this used for all Autoloads or do I need different one etc? I don't want a port issue with current default auotload if that is a known issue. Thanks for your time. -KJ
Thanks for the post, Scott. Having this task available in the SAS Deployment Manager is a real time-saver.
However, one minor issue I've come across with this process is that when the on-disk artifacts are created, the Formats folder ends up being misnamed as Format (i.e., no -s at the end). When the autoload process runs, it is looking specifically for a folder called Formats, not Format, so any user-defined formats that get dropped into the Format folder never get found, causing any data sets referencing those formats to error out during the autoload process and never get loaded into memory.
The fix is easy (just add the -s onto the Format folder name), but it is something I've learned to watch out for when using the Deployment Manager task to create new autoload drop-zones.
Jake, good catch. In all the times we've done this we never noticed that. I've opened a defect for that so hopefully it will be corrected soon.
Thanks for reading.
-Scott
Here is mine for running it at specific times.
#!/bin/bash
#
# schedule.sh
#
# syslog Schedule AutoLoad.sas
#
RUNSAS_PATH="/opt/sas/config/Lev1/Applications/SASVisualAnalytics/VisualAnalyticsAdministrator/CustomerIntelligence/runsas.sh"
#TIME_INTERVAL_MINUTES=5
#call to runsas.sh
cat <(fgrep -i -v $RUNSAS_PATH <(crontab -l)) /dev/null 2>&1 ") | crontab -
#cat <(fgrep -i -v $RUNSAS_PATH <(crontab -l)) /dev/null 2>&1") | crontab -
Now that we are on new servers and autoload is done in less than 5 seconds we went back and I changed it to every 5 minutes.
Thanks for sharing, Clinton.
Hi Scott, you mention that in autoload folder,"a user can drop a spreadsheet, delimited file, or SAS Data Set". How does SAS determine a delimiter for "delimited file"? It seems that comma-delimited files are loaded correctly, but not "pipe" ( | ) - delimited.
Hi Leonid, yes, you can specify another delimiter by setting the VA.AutoLoad.Import.Delimiter.TXT extended attribute on the LASR Library associated with your autoload location. By default, VA.AutoLoad.Import.Delimiter.TXT = TAB but you can set that to | (no quotes) and it will then pick up your file appropriately. Be careful though, the autoload import gives great weight to the extension of the file you drop into the autoload location. Specifically, it looks for .CSV (comma separated), .TSV or .TAB (tab delimited). Any file with .TXT will use the delimiter you specify in the extended attribute. Therefore, if you want the import to use your pipe delimiter, you'll need to drop the files with a .TXT extension.
Scott,
Is it possible to have files dropped into the autoload folder automatically by a MS Server Integration Services job? I can paste the tables manually but would not want to have to do this on a regular basis. When I try to write a file to this folder from within a SSIS file task, I get an "access denied" error even though I am using the same credentials.
Beverly, I am not aware of that being a problem but your best bet would probably be to open a track with Technical Support. They should be able to let you know if there are any technical issues that prevent that from working. I do know that many customers use ftp/sftp to write files to the autoload drop-zone if that's an option for you. Good luck!
Hi,
I have a requirement where i want to prioritize certain set of tables to load initially say within 3 to 5 mins time interval and remaining set of tables in 10 to 15 mins time interval through Autoload, but one autoload folder can have single value of time period interval to set which is by default 15mins.
Can we have two autoload folders setup during installation with single LASR server with one autoload folder TIME_INT_PARAMETER=3 MINS and other autoload folder with TIME_INT_PARAMETER=15 MINS. If so, How can i configure 2 autoload folders during installation by using sas deployment manager.
I am new to this admiistration part. Can anyone let me know if there is any other possible way to prioritize certain set of tables from autoload folder to LASR.
Thanks in Advance.
Aanan
If I requested new software, would I need to reinstall to use this feature, or would the updated deployment manager work with an existing installation?
what version of SAS has this option i have SAS 9.4 M2 does not have this.
Thanks
Hi Akbar, you'll need 9.4M3 from Oct 2015 or later to pick up this feature.
I have gone through the process but autoload script is running it also creating logs but other directories like Append,Schedule are remain empty also time stamp in SAS VA does not get change.
Nishant, glad you are giving this a try. It is best that you contact SAS Technical Support so that they can help you if something is amiss.
Scott,
What ship event includes this new functionality in the SAS Deployment Manager. We're on Linux X64 940_15w29 including Distributed VA 7.3, and I don't have that option.
Thanks!
David
Hi David...the autoload configuration was added to the SAS Deployment Manager in 940_15w42.
Hi Scott,
Thanks for the post!
I have a question, is necessary to create a new LASR Server and LASR Library?. Existing ones can not be used?
Thanks a lot,
Belén.
Hi Belén, thanks for your question. Fortunately, you can do either. If you look above at the screenshot that shows the "LASR Artifacts" choices, selecting either "Non-Distributed" or "Distributed" implies that you wish to create a new LASR Server and LASR Library and to configure it with an autoload location. Selecting "Pre-existing Library" allows you to configure autoload for an existing LASR Library which is already associated with a particular LASR Server. Keep in mind that the metadata for each LASR Library incudes a single value for VA.Autoload.Location and the SAS Deployment Manager will overwrite any existing value with the location you specify here.
Great post, thank you, Scott! Is it possible to specify autoload interval on a "per file" basis or it is a system-wide setting? Alternatively, can autoload interval be specified on a "per drop location"? Can there be multiple drop locations?
Thanks, Leonid. The autoload interval is a drop-location-specific setting so each drop-location can have different values if necessary. I do not believe the default framework provides for a 'per-file' detection using the standard scripts. The polling interval is set in the schedule.bat | .sh file. And yes, there can be several drop-locations defined within a single environment. So for example, different departments could be set up with unique drop-locations.