The SAS Deep Learning action set is a powerful tool for creating and deploying deep learning models. It works seamlessly when your deep learning models have been created by using SAS. Sometimes, however, you must work with a model that was created with some other popular package, like PyTorch. You could recreate the PyTorch model inside of SAS by using SAS Deep Learning, but that adds extra work and time to your project. Fortunately, it is not necessary because SAS created an action set called dlModelZoo that supports importing PyTorch models.

The dlModelZoo action set can import PyTorch models and use those models alongside the other powerful modeling capabilities of dlModelZoo. This handy feature lets you skip the extra step of recreating the model in SAS Deep Learning. It enables you to leverage the PyTorch model along with many other dlModelZoo capabilities. This includes training, scoring, and even tuning hyperparameters. In this post, we will demonstrate how to import PyTorch models into dlModelZoo and introduce you to some of its modeling capabilities.

PyTorch model

First, an artificial neural network model in PyTorch is created to split images into distinct objects. We won’t be labeling the objects, just creating image masks for different objects in the scene. The specifics of the model aren’t important. What is important is that the artificial neural network, created in PyTorch, can be exported into SAS Deep Learning by using TorchScript. Later it can be trained and scored using dlModelZoo and SAS Deep Learning.

To get started with the example, we’ll use a subset of data from kaggle. The data contain simulated images from the viewpoint of a driving car. Figure 1 is an example image from the data set.

To separate the different objects in the scene, we need to train the weights of an existing PyTorch model that was designed for a segmentation problem. Many deep learning models written in PyTorch are meant to handle this kind of problem. The specific model that we use here is called Deeplab. The output of our model will be a set of masks that distinguish different objects in the scene. Based on an example in Figure 1, the output of the model is depicted in Figure 2. Each different shade represents a different object in the scene.

To import these images to SAS, a table needs to be created containing two columns. The first column shows the path to the original image. The second column shows the path to the desired masked image. An example of a representative table is shown in Table 1.

Table 1: Example of a representative image table

Loading the PyTorch model

Now that we have processed the images, we can get to the important part! Let’s import a PyTorch model into SAS. You need to export your model from PyTorch in a way that can be used by the dlModelZoo action set.  The tool we will be using is called TorchScript.  The process is simple. You just need the following two lines of Python code:

model_scripted = torch.jit.script(model) # Export to TorchScript
model_scripted.save(‘deeplab.pt') # Save

For this example, we export the model into a file named “deeplab.pt” by using the two lines above. The PyTorch model has been exported in a way that SAS can understand, but we still need to provide more details about the model. To describe the model to dlModelZoo, we need to create a yaml string. Most of this string will be the same for all models. Inline comments shown in the following code describe what is needed to be modified for your specific model.

documents = """ #key is case-sensitive, value is not case-sensitive
---
  sas:
  dlx:
  label: "my_torchscript" #referenced in action calls
  dataset:
  type: "Segmentation"
  preProcessing: #a section to place any preprocessing of the input, in our case I am just resizing
    - modelInput: 
        label: input_tensor1
        imageTransformation:
          resize:
              type: TO_FIX_DIM
              size: 256 256 
              targetSize: 256 256
  model:
    type: "TORCHSCRIPT" # TORCHSCRIPT if importing a .pt file with model and weights
                                           # TORCHNATIVE if you want to use a predefined model with a .pt file that contains only weights
    majorVersion: 1
    minorVersion: 2
    caslib: "modellib" # just the caslib where the model exists
    path: "deeplab-model/deeplab.pt" # the location of the TorchScript model
    classNumber: 16    # number of classes to predict (in our case, number of objects to mask)
    inputs:
      - label: input_tensor1
        size:
        - 0
    outputs:
      - label: output_tensor1
        size: 
        - 0
---
"""

Once the yaml string has been created, dlModelZoo knows how to import the model and use it. All that remains to do is to call the training and scoring actions from dlModelZoo.

Training and scoring (and tuning)

We have imported the model into SAS, and now we’d like to fine-tune our model with our specific data in SAS. Assuming that the CAS action session object “s” is loaded, we will use the dlmzTrain Action to train the model as shown in the following code:

res = s.dlmztrain(
    #The location of the data
    table="metadata",
    inputs="image_path",
    targets="label_path",  
    #Where to put the results
    modelOut="trained_model",
    optimizer=dict(
        loss="cross_entropy",
        #The training algorithm to use
        algorithm=dict(
            learningRate=5e-2,
            momentum=0,
            method="sgd",
        ),
        maxEpochs=100,
    ),
    #Here is where that yaml string from before is used
    extraOptions=dict(yaml=documents, label="my_torchscript"),
)

If we wanted to tune the learning rate hyperparameter in that way, we would replace the single value with a range as demonstrated here:

learningRate=dict(lb = 1e-6, ub=5e-1, logScale="LN"),

Lb and ub represent the lower and upper bounds for the hyperparameter of learningRate. Logscale=”LN” indicates that you want to scale the learning rate by the natural log function and tune that. As soon as a range is provided for at least one hyperparameter, dlmzTrain will tune and train the model.

Scoring is just as easy. This code shows an example by using the dlmzScore Action:

res = s.dlmzscore(
    model="trained_model", #this was the output of dlmzTrain
    table="metadata",
    inputs="image_path",
    targets="label_path",
    tableOut=dict(name="output", replace=True),
    #and here I use the label from the scoring section of the yaml string
    extraOptions=dict(yaml=documents, label="my_torchscript")
)

Predefined models

The example in the code uses TorchScript to import a model from PyTorch into SAS Deep Learning. In most cases, this is not necessary because SAS has a variety of predefined models that can be used right out of the box. All that needs to be done is to replace “TORCHSCRIPT” with “TORCHNATIVE” in the model->type section of the yaml description. Also, make sure that if you have pretrained model weights for a pretrained model the path to your model file containing those weights is correct. Once this change to the yaml string is completed, there will be access to many different neural network models that cover many different deep learning tasks. These include ENet, Dilated RNN, LeNet, MobileNet, ResNet, ShuffleNet, UNet, VGG, and YOLOV5.

Summary

This example shows how to import PyTorch models into SAS Deep Learning. We have described how to load a TorchScript exported PyTorch model into SAS, train that model, tune hyperparameters, and score against a validation data set by using the dlModelZoo action set. The dlModelZoo action set has also a variety of predefined models available. The predfined models can be used without being imported. They will likely serve well for most deep learning modeling needs. Hopefully, this post will encourage you to try out the dlModelZoo action set. Have fun!