SAS Environment Manager Data Mart—the heart of the Service Management Architecture

Several new capabilities and components are available in SAS Environment Manager 2.4, the web-based administration solution for a SAS environment. For me, the most important enhancement is probably the SAS Environment Manager Service Management Architecture Framework, which provides features and functions that enable SAS Environment Manager to fit into a service-oriented architecture (SOA). The Management Architecture Framework includes:

All these components are organized around the heart of the Service Management Architecture Framework: the SAS Environment Manager Data Mart. In this post, I’d like to share more details about its setup and use.

What is the SAS Environment Manager Data Mart?

The SAS Environment Manager Data Mart is a collection of tables that store monitored data in a data infrastructure that is provided by the SAS Environment Manager Extended Monitoring. These data are collected using the various ETL processes and stored in a standard format, making it easy for SAS administrators to run audit reports and perform analyses on performance and usage.

The data in the Data Mart are also used to populate predefined stored process reports that are provided as part of SAS Environment Manager Report Center. SAS administrators may also use the Data Mart tables for custom reporting and analysis, or they can be used to feed SAS Visual Analytics reports for the administrator autoload environment. You can the full documentation on the SAS Environment Manager Data Mart in the SAS® Environment Manager 2.4 User’s Guide.

How is data collected and stored?

The SAS Environment Manager Data Mart contain standardized data from:

  • Metrics data read from monitored resources across the SAS environment
  • Data read from SAS server’s and spawner’s logs
  • Specific data read and extracted from SAS solutions

Each of these data sources has a separate data source and ETL process:

  • Agent-Collected Metrics (ACM) ETL. This ETL Process runs against the SAS Environment Manager Database. It collects these data, standardizes it, and stores it in the SAS Environment Manager Data Mart tables (ACM tables).
  • Audit, Performance and Measurement (APM) ETL. This ETL Process runs against SAS logs. You probably know this component as the downloadable SAS APM package. This package is now totally integrated in SAS Environment Manager. It collects these data, standardizes it and stores it in the SAS Environment Manager Data Mart tables (ARTIFACT tables).
  • Solution kit ETL processes. The solution kit framework can extend the capabilities of SAS Environment Manager to support specific solutions or applications. The framework includes support for collecting and storing operation information about the solution in the SAS Environment Manager Data Mart tables (KIT tables).

Information stored in these tables can be consumed using predefined or custom reports within SAS Environment Manager, SAS Visual Analytics or the application of your choice through direct access to the SAS data sets.

Click on the image below to see a diagram of data flow within the SAS Environment Manager Data Mart.

EV_datamart

How do I set up the data mart?

Installation and configuration. The SAS Environment Manager Service Management Architecture package is deployed and configured on the SAS Application context host using the SAS Deployment Wizard. For multi-context SAS deployments, the SAS Environment Manager Service Architecture package will be installed on one host in one SAS Application context and is treated as a single deployment.

Initialization steps. To begin, initialize SAS Environment Manager Extended Monitoring using a script provided with the software. During this step, different objects and components are created in SAS Environment Manager, and all the required SAS Environment Manager Data Mart tables are created.

The initialization of ACM, APM and Solution Kit ETL processes are independent of one another and require additional steps. The SAS administrator may initialize all or only one or two of them, depending on their auditing needs. For detailed initialization instructions for APM, ARM and Solution Kit ETL processes, please refer to the documentation: SAS® Environment Manager 2.4 User’s Guide (Chapter 9: Initializing and Enabling the Service Management Architecture).

ETL processes. By default, the ETL processes are executed every night at a predetermined time. This schedule is managed by SAS Environment Manager. If you want to execute these processes more frequently, you can schedule new jobs outside SAS Environment Manager, using the master ETL scripts provided with the software.

After you initialize each ETL process,  it’s a good practice is to run it manually to populate the SAS Environment Manager Data Mart. Otherwise, you will have to wait overnight to see the results and determine if data are being collected as you want.

What’s available in the Report Center?

The Report Center provides a convenient access point for the reports that are provided as part of SAS Environment Manager Service Management Architecture.

The Report Center contains predefined reports based on the data that are stored in the SAS Environment Manager Data Mart. These stored processes and reports were created during the SAS Environment Manager Extended Monitoring initialization and remain empty until one or more of the ETL processes is enabled.

The predefined stored processes and report provide a view of the performance and status of your SAS environment and its resources. These are only samples of the full range of reports that can be created against the SAS Environment Manager Data Mart data. SAS administrators can create custom stored processes and reports to meet their organization’s needs and store them in an appropriate metadata folder structure.

For detailed information about Report Center, please refer to the documentation: SAS® Environment Manager 2.4 User’s Guide (Chapter 10: Using the Report Center).

I hope this article has been helpful in introducing you to my favorite new features in SAS Environment Manager.

Post a Comment

SAS arrays—be not afraid!

SAS Technical Support Problem SolversArrays are a powerful SAS programming tool. They can be used to simplify coding for repetitive calculations, to transpose data and to evaluate variables in a non-sequential manner. Sometimes users are intimidated by the term array, but in SAS, an array is simply a grouping of variables that lasts for the duration of the DATA step where the array is defined.

In this blog post, I’ll provide the code and explanations for using SAS arrays to solve these common programming problems:

  • Find the closest value to a constant value in a group of variables.
  • Determine which variable in a group meets certain criteria and report corresponding variables.
  • Transpose data and manipulate a group of values.

Find the closest value in a group of variables to a constant value.

In this example, we want to know, for each value of ID, what was the closest visit date to 01Jan2014? By defining an array date with the variables visit1-visit5, we can use a DO loop to evaluate each visit date to see whether it is closer than the previous ones. If so, we redefine the closest and diff variables to contain the closest visit date and the difference between that date and 01Jan2014.

When variables are listed in the array definition statement, SAS first looks for existing variables with those names. If they do not exist, SAS creates them. The default for variables created in an array is numeric type, as it is for variables created otherwise.

The dim() function returns the dimension of an array. You can use this in place of hardcoding the end value.

The string '01jan2014'd is a date constant and is the syntax to refer to a specific date.

data visits;
input id (visit1-visit5) (:date9.);
datalines;
1 11aug2012 14sep2014 04nov2013 13feb2014 30jun2013
2 23aug2014 02jan2014 01jan2012 08nov2014 15may2013
run;

data results (keep = id closest diff);
format id 8. visit1-visit5 closest date9.;
set visits;
array date[*] visit1-visit5;
closest = date[1];
diff = abs(date[1] - '01jan2014'd);
do i = 2 to dim(date);
  if abs(date[i] - '01jan2014'd) < diff then do;
      closest = date[i];
      diff = abs(date[i] - '01jan2014'd);
  end;
end;
run;

proc print; run;

For each ID, the date closest to 01Jan2014 and the difference in number of days is displayed.

SAS output

What variable in a group meets a certain criteria, and what are the corresponding variables?

In this example, we want to know, for each respondent ID, which flavor received the lowest score and what was the date and location of that score? 

The definition for the flav array is using a name range list to refer to all variables in order between apple and kiwi inclusive.

We can use a special SAS name list like _character_ to refer to all character variables currently defined in the DATA step. The LENGTH statement for a new character variable flav_min has to be after the array definition statement if we do not want it included in the array of character variables.

Since the variables for the date array are not listed, SAS first looks for, and if needed, creates variables with the names date1-date5.

The OF operator can be used with certain functions to perform that function on the variable list or array specified.

Once we know the minimum value, we loop through the variables to determine which element matches the minimum value. Then we use the vname() function to return the name of the variable. We find the corresponding variables in the loc and date arrays by using the same i subscript to obtain those values.

The LEAVE statement stops processing the current loop. We do not need to continue processing when we have found the value that matches the minimum. (In this case, we are assuming that there is only one value that matches the minimum.)

data flavors;
input id (apple orange banana grape kiwi) (:8.) (loc1-loc5) (:$1.) (date1-date5) (:date9.);
datalines;
1 100 98 75 84 92 A A B C D 10jul2014 12jul2014 01aug2014 10aug2014 12sep2014
2 80 78 83 88 72 B C A E D 10jul2014 12jul2014 01aug2014 10aug2014 12sep2014
;

data flavor_min (keep = id flav_min date_min loc_min);
  retain id flav_min loc_min date_min;
  format date_min date9.;
  set flavors;
  array flav[5] apple--kiwi;    
  array loc[5] _character_;                      
  array date[5];
  length flav_min $10;
  minflav = min(OF flav[*]); 
  do i = 1 to dim(flav);
  if minflav = flav[i] then do;
    flav_min = vname(flav[i]);   
    date_min = date[i];
    loc_min = loc[i];
    leave;                                   
  end;
  end;
run;

proc print; run;

For each ID, the resulting output displays the flavor with the lowest score and the corresponding location and date where that score occurred.

SAS output

Transpose data and manipulate a group of values.

In this example, for each ID, we want to keep only the last value of each identical series and set the remaining values to missing so that there is only one observation per ID in the output. In this case, the data is originally what is sometimes called “long”, that is, many observations per BY group.  We want to make it “wide”, or one observation per BY group.

Instead of using the default lower and upper bounds for the array, we specify the array bounds 2007:2014. That way, we can use the value of year to indicate which array element should be updated.

If we do not specify the variable names in the array, they will be defined yr1-yr7 even though the bounds of the array have been defined differently. We explicitly name them yr2007-yr2014 because those are the variable names we want. These are character variables, so we have to specify that in the array definition statement with a $ after the array name.

We use the CALL MISSING() routine to set all the values of the array to missing at the start of each BY group.

Since we want to transpose the data, we read the value for each observation into an element of the array corresponding the year.  We can use the year value as the subscript for the array and set the value of the correct yrn variable to value.

Once we have read the entire BY group into the array, we’ll use a DO loop to check each value of the array against the next value.  If they are the same, we set the current one to missing. We do an explicit OUTPUT at the end of the DO loop because we don’t want to do the default output at the end of each iteration of the data set. We just want to output one observation for the entire BY group.

data years;
  input id year value $1.;
  datalines;
  1 2007 A
  1 2008 A
  1 2010 B
  1 2011 B 
  1 2012 B
  1 2013 A 
  1 2014 A
run;

data years_out (drop = year i value);
  set years;
  by id;
  array yr[2007:2014] $ yr2007-yr2014;
  retain yr:;
  if first.id then do;
  call missing (of yr[*]);
  end;
  yr[year] = value;
  if last.id then do;
  do i = 2007 to 2013;
    if yr[i] = yr[i+1] then yr[i] = '';
  end;
  output;
  end;
run;

proc print; run;

The output data set has only one observation for each ID with repeated values set to missing until the last one in a series.

SAS output

Just getting started with arrays?

The best place to start looking for information about arrays is the section on Array Processing in the Language Reference: Concepts document.

There are more examples in the  Statements: Reference document under ARRAY statement and Array Reference Statement.

Post a Comment

Which SAS Migration Utility do I use?

During a migration to a new release of SAS software, the SAS Migration Utility is used to analyze and package your source software. There are two ways that you can get the SAS Migration Utility. It’s available by selecting SAS Migration Utility from the Downloads page on support.sas.com. The SAS Migration Utility also ships with your SAS software order and is delivered in the SAS Software Depot.

Does it make a difference how I obtain the SAS Migration Utility?

The downloadable SAS Migration Utility is offered as a convenience to help you prepare for a migration. If you’re considering a migration, you can download the version from support.sas.com to inventory your current SAS deployment. It’s a best practice to run the SAS Migration Utility and review the analysis before you order your new software.

During the migration, you should use the version of the SAS Migration Utility that is included in the SAS Software Depot. It is always compatible with the versions of the products ordered and should be used to package your source, or current, deployment. The SAS Deployment Wizard will check that the package was created by a compatible version of the SAS Migration Utility. In the SAS Software Depot, the SAS Migration Utility is located in the <depot-home>\utilities directory.

How to select the right SAS 9.4 migration utility

SAS 9.4 supports automated migration from SAS 9.2, SAS 9.3 and SAS 9.4 using the SAS Migration Utility and the SAS Deployment Wizard. In a SAS 9.4 SAS Software Depot, there will be three SAS Migration Utility directories under the \utilities directory.

You may be confused about which SAS Migration Utility to use.  Do I use smu93 to migrate to SAS 9.3?  Or is that the utility I use when migrating from SAS 9.3. The rule, once you know it, is really quite simple. It can be stated in two parts:

  1. Locate the \utilities directory in the SAS Software Depot that contains the software for your target environment.
  2. Within that directory, select the version of the SAS Migration Utility that matches with your source environment.

In short, get the SAS Migration Utility from the SAS Software Depot of your target deployment and run the executable that matches the operating system and SAS release of your source deployment.

Step-by-step example

To illustrate with an example, let’s say I want to migrate from SAS 9.2 to SAS 9.4. I go to the \utilities directory of my SAS 9.4 SAS Software Depot. Within that directory, I’ll choose the directory smu92 because I want to analyze and package a SAS 9.2 source deployment.

list of directories in SAS Software Depot

In the \utilities\smu92 directory, you will find four versions of the SAS Migration Utility that can package SAS 9.2 deployments. Select the version which matches the operating system of your source deployment. If you have multiple operating systems in your source SAS Deployment, you will need multiple SAS Migration Utility executables.

  • Unix/Linux: smu92
  • Z0S: smu92.zos
  • Windows 32-bit: smu92_32.exe
  • Windows 64-bit: smu92_x64.exe

SMU2

Post a Comment

Junior Professional Award – helping new SAS users attend SAS Global Forum 2015

winnerRecently, I spoke with Sally Carson about the Junior Professional Award, a SAS Global Forum 2015 program for people who have used SAS three years or less. The award offers free registration, free training and lots of opportunity to learn and become involved. 

This is the third year that Junior Professional awards have been offered for SAS Global Forum. Junior Professional programs started in 2009 at US regional conferences and have really caught fire. Since 2013, the SAS Global Forum program has had almost 250 applicants, and recipients have attended the conference from as far away as China and New Zealand.

Sally is a member of the SAS Global Users Group Executive Board and is the 2015 Junior Professional Program Coordinator. Developing programs that support new SAS users has been a passion of Sally’s for a long time, and the Junior Professional Program is one of her particular favorites. Here’s a little bit of our conversation: Read More »

Post a Comment

SAS macro errors: the answers you’re looking for

Have you ever received an error or warning in SAS macro and did not know what to do next or even where to look?  Now there is an answer! And debugging your SAS macros just got easier.

All macro errors and warnings are now documented in the SAS 9.4 Macro Language Reference, Third Edition. You can find them here:

Each error or warning entry begins with the text of the message followed by a two-colum table that lists each possible cause and a suggested solution for each cause. Here are samples from each section of the appendix that illustrate how the information is presented.

Let us know if this new feature in the SAS Macro Facility documentation helps you to investigate your own possible cause and solution for any macro error you may be encountering. Read More »

Post a Comment

SAS Global Forum 2015 -- content a big success!

As the team of nearly 50 peer-reviewers finalize their review and recommendations, I want to reflect on the amazing accomplishments of the SAS Global Forum 2015 teams: 

  • Nearly 700 sessions were submitted by the SAS user community and SAS experts!

Content areas from data mining to administration have been submitted for varying levels of proficiency and across many disciplines and job areas. There truly is something for all SAS users!

Popular delivery methods such as hands-on workshops, e-posters and breakouts will be complemented this year by table talks on a variety of topics of interest to SAS users.

This is an exciting and busy time for the SAS Global Forum 2015 volunteers. Please enjoy the content summary available now as we work to have the entire list online in January! You’ll have the opportunity to build out your conference agenda approximately two months before arriving in Dallas.

Now that you can see a few of the professional development and learning opportunities waiting for you, we need to get you to SAS Global Forum 2015.  Register today and invite your colleagues. Team discounts are available. There are also opportunities for student and faculty scholarships as well as Junior Professional Awards for those that have been using SAS for three years or less.

Post a Comment

Renewing SAS solutions: a two-step process

Did you know that applying a new SAS license file for many SAS solutions is a two step process?

Because many of SAS’ most of popular solutions (including SAS® Visual Analytics, SAS® Enterprise MinerTM and SAS® Customer Intelligence) depend on the middle-tier architecture for primary user access, information about licensed products and expiration information is maintained in metadata.  So to renew licensing properly for another year, SAS administrators need to update the SID file information in metadata in addition to updating the licensing information stored in the CORE catalog.

As SAS administrator, when you receive a new SID file, you must first apply the new license using either the Renew SAS Software utility on Windows or by running the sassetup script on UNIX systems.  This step updates the CORE catalog with licensed products and expiration dates, and it must be performed on every machine that has a Base SAS installation.

For sites that license SAS solutions, the SAS administrator then needs to run the SAS Deployment Manager and select the “Update SID File in Metadata” option.  This can be done from any machine in the deployment and only needs to be performed once, regardless of the number of machines involved.

renewsolutions

You’ll be prompted for metadata connection credentials, the configuration directory, and the fully qualified path to the new SID file.  The SAS Deployment Manager takes care of the rest and the new licensing information will be updated in metadata.  All that’s left to do is to restart all of the SAS processes to pick up the new license.

For more information

Technical Support maintains a list of all SAS solutions that require this two step process in Usage Note 49750.

Similarly, there are detailed instructions for renewing all types of SAS installations available from support.sas.com.  You can search by Base SAS release, product and product release.

Post a Comment

5 characteristics you meet in the cloud

NIST 5 characteristics of cloud computingIf someone asks you whether SAS runs in the cloud, there are exactly two wrong answers: "yes" and "no". Instead, this question should spark a discussion. It should be a discussion about which of the five characteristics of cloud computing they are interested in. The answers will point you in one direction or another.

If you have seen any SAS presentations about the cloud, you probably have seen this diagram. This terminology comes to us from the National Institute of Standards and Technology, and more precisely, from The NIST Definition of Cloud that has been adopted by SAS.

“Why do we need definitions and standards?,” some of you may ask.  If ever there was a term loaded with hype and misunderstanding, it is “cloud”. The NIST provides us with a common language to talk about it.

In today’s installment of this three-part series, I will discuss the five essential characteristics of cloud computing from NIST, and try to illustrate them with simple examples applied to situations most would be familiar with. Read More »

Post a Comment

How to tune storage arrays and clustered file systems for use with SAS

For those of you who have followed my SAS Administration blogs, you will know that setting up your IO subsystem (the entire infrastructure from the network/fibre channels in your physical server, across your connections, to the fibre adapters into the storage array, and finally to the physical disk drives in the storage array) is very near and dear to my heart. Based on this desire, as my team learns information on better ways to configure the IO system infrastructure, we like to document it in new white papers, or by updating existing white papers. Read More »

Post a Comment

Is there a data steward in your life?

If you’re a SAS user, no matter what your role—programmer, analyst, statistician or administrator—data is important.

But I’m willing to bet that somewhere in your organization, there’s a group of people who not only care about the data—it’s their passion, their raison d’être. Whether they sit in IT or your own business unit, these people make sure that the data that runs applications and fuels analytics is fit for the purpose. Without them, your data would be a big mess.

Those rock stars of the data world are often called data stewards. They’re the go-to people when you have a question about the how data is collected, managed, stored and curated. They go above and beyond to help out their own groups – and often any others in the organization – to make sure that information is consistent, accurate and reliable.

And on Dec. 9, SAS sponsors International Data Stewards Day – a chance to celebrate those who have the monumental task of managing corporate information. Read More »

Post a Comment