Images represent one of the largest drivers of data growth. Consumer devices, medical image scanners, imaging satellites, etc. generate astounding amounts of data daily. Efficiently processing these large swaths of raw data, using computer vision techniques to uncover decision-making insights, represents one of the biggest challenges of the present-day world.

SAS^® Viya^® is one of the solutions that SAS offers to address this challenge. Specifically, SAS^® Visual Data Mining and Machine Learning empowers customers with computer vision techniques to analyze image data and its associated metadata at scale. It offers traditional approaches like Canny edge detection. But it also offers advanced methods, such as deep learning-based artificial intelligence and image analytics. Performant image analytics at scale is one area that differentiates SAS Viya from alternatives, like open-source for example.

The purpose of this post is to give you an overview of loadImages, the SAS Cloud Analytic Services (CAS) action for loading images. The loadImages action is the critical first step in most image analytics pipelines in SAS Viya. I will take you behind the scenes of this action to provide a glimpse of how the action achieves performant image loading.

Overview of the loadImages action

The loadImages action is a highly sophisticated component of image analytics in SAS Viya. The overarching scope of the action is to load images and associated metadata from a source external to the CAS server. Then distribute the loaded data among the nodes in the server. To appreciate the design considerations around performant image loading, it’s helpful to go over the complexity of what the action supports.

As of this writing, the action supports 12 different image formats, used across both photographic and biomedical image domains. The former domain largely involves 2D, 3-channel (RGB) images such as the ones produced by consumer smartphones. JPEG, the most popular image format in the photographic domain, is supported by this action. The latter domain mostly involves 2D or 3D single-channel (grayscale) images like the ones generated by a CT scanner in a hospital. The industry standard for image storage in medicine is called Digital Imaging and Communications in Medicine (DICOM). The loadImages action also supports this standard.

The action not only loads images but also parses (aka decodes) the images to extract and output the metadata associated with them. Photographic metadata examples include widths and heights of images, file paths, and image labels embedded in directory structures. Also supported, exclusively for the biomedical domain, are DICOM tags such as a patient’s age and date of the scan.

A typical invocation of the loadImages action from a Python client is shown in the code snippet below.

   mySession.loadimages(
        path='myPath',
        caslib='myLib',
        recuse=True,
        casout='myImages'
)

This code snippet will prompt the action to load images under the path myPath in caslib myLib into the output CAS table myImages in session mySession. The path can point to a single file, directory, URL, or ZIP file. At this time, only a PATH caslib, i.e., a server-side directory that the CAS controller can access, is supported by the action. If the path is a directory, it will recursively traverse the directory to load all images under it. The action will unzip the file and load each image contained therein if it is a ZIP file.

If an additional parameter pathIsList is set to True, the path parameter can also point to a text file that contains a list of file or directory paths. In this case, the action will load images under each of the paths listed in the text file.

Note, the user does not need to know in advance the kind of files or images present under the path. The contents under the path could be photographic images, biomedical images, ZIP files, etc. For each file under the path, the action will automatically detect its type and perform the loading accordingly.

Based on this complex mix of features supported by the loadImages action, the computations within the action can be categorized into the following groups.

1. Listing: generating the list of file paths to read, e.g., by traversing a directory
2. Reading: reading images from the file system
3. Distributing: sending and receiving of images between the nodes in the server
4. Processing: decoding of images, metadata processing, and writing to the output table

Now, the computations for one image are independent of those for another. For example, the processing of one image can be done independently of the reading of another image. This observation was the key to designing a performant implementation of loadImages via thread-level parallelization.

The following sections examine two different implementations of the action based on the level of threading. We then compare the performance of the two implementations to demonstrate performance improvement achieved by the multithreading of loadImages. I will reference the groups listed above in the following sections.

The loadImages action pre-SAS Viya 4: a single-threaded implementation

The implementation of loadImages prior to SAS Viya 4 followed a single-threaded strategy. The figure below illustrates the two execution modes of the CAS server. These are namely the Symmetric Multi-Processing (SMP) mode and Massively Parallel Processing (MPP) mode.

In the SMP mode, there is only one node in the CAS server, the controller. A single thread on the controller performs the computations in groups 1, 2, and 4. Note that image distribution is not applicable in SMP mode since there is only one node.

In MPP mode, there is a controller node and one or more worker nodes. All nodes run a single thread. The thread on the controller performs the computations listed in groups 1-3. That would be the file path list generation, image reading, and image sending. The threads on worker nodes perform the computations listed in groups 3 and 4, the receiving and processing of images. As described in The Architecture of the SAS Cloud Analytic Services in SAS Viya, data exchange between nodes are performed via GCCOMM, SAS’ implementation of cluster communications.

The loadImages action starting in SAS Viya 4: a multithreaded implementation

A multithreaded implementation of the loadImages action was introduced in SAS Viya 4. The figure below shows the strategy for the SMP and MPP execution modes of CAS. As illustrated, each server node runs multiple threads. The loadImages action, as with any other action, calculates the number of threads on each node depending on the customer license and the number of CPU cores in the cluster.

In SMP mode, one of the threads on the controller, called the main thread, performs the computation in group 1. That would be the file path list generation. The main thread then queues in the list of file paths. This queue is consumed by one or more threads called worker threads that perform computations in groups 2 and 4. That would be the reading and processing of the images.

In MPP mode, the main thread on the controller functions exactly as in the SMP mode. It queues in a list of file paths for the worker threads. First, the worker threads consume this queue. Then the worker threads read the images (the computation in group 2). Then finally send them evenly to the worker nodes (the computation in group 3).

The main thread on each worker nodes receives the images (the computation in group 3). Then it queues them up for the worker threads on that node. These worker threads on the worker node consume the image queue and process the images (the computation in group 4). As in the single-threaded strategy, all data exchange between nodes are performed via GCCOMM.

Loading of images from files that contain multiple images, specifically ZIP files, deserves a special mention in the multithreaded strategy. To maximize CPU utilization, the action employs multiple worker threads for each ZIP  file. One worker thread unzips a ZIP file, extracts images from it, and writes them to the image queue. Other worker threads consume these images. Next, they either send the images (from the controller in MPP) or process the images (in the controller node in SMP or worker nodes in MPP). The algorithm underlying the multithreaded implementation guarantees that there will be no deadlocks. This is even though the same worker thread reads from and writes to the same queue.

Performance comparison of the two implementations of loadImages

To demonstrate the benefit of multithreading, the time performance of the loadImages action was assessed by loading one million 250 x 250 RGB JPEG images. To minimize variabilities caused by network traffic load, the data set was stored directly on the hard drive of the controller node. Both SMP and MPP modes were evaluated. The MPP mode utilized 16 nodes.

Each node was powered by 80 threads, running on a 40-core hyperthreaded Intel® Xeon® CPU. The amount of RAM available to the servers was sufficient to contain the one million image data set in an in-memory CAS table.  Since the single-threaded implementation of loadImages existed pre-SAS Viya 4, before the containerization of SAS Viya, non-containerized deployments of CAS were used in our experiment, to isolate the effects of threading.

The figure below shows the real response time of the action in four different combinations of execution modes and thread counts. Since the server hardware was shared by developers at SAS for various applications, the performance measurement was repeated five times for each combination. The time value charted below for each combination corresponds to the fastest execution observed among the repetitions for that combination.

Clearly, the loadImages response time is much faster, about 2x faster in SMP and 6x faster in MPP, in the multithreaded implementation. In SMP, the multithreaded implementation loaded one million images in 44 seconds.

The loadImages action is slower in MPP than in SMP because of the grid communication overhead in MPP, where the data read by the controller node from a PATH caslib have to be distributed among the worker nodes. However, any downstream processing of the loaded and distributed data, using the processImages action, will execute much faster in MPP than in SMP.

Conclusion

This post demonstrates that the loadImages action response time is critical to the performance of any image analytics pipeline built on SAS Viya. A multithreaded implementation of the loadImages action introduced in SAS Viya 4 has boosted the action performance by a factor of up to six. The experiments performed for this post show that the loadImages action is capable of loading one million images in one minute.

From a customer perspective, the efficiency and scalability offered by the loadImages action can help reduce computing infrastructure costs. Given that loading images into memory is a critical first step in most image analytics pipelines, this cost reduction will apply across a variety of applications. In some cases where response time is critical, the high performance of loadImages will even allow customers to solve problems that are otherwise infeasible.

Finally, while this post has focused on loadImages, it is but one example of how SAS’ customers can benefit from the performant analytics offered in SAS Viya. You can find other examples in previous blog posts, including the ones on reverse image search and parallel processing of common data science tasks. If performance analytics is important for you, SAS Viya can offer a competitive edge over its alternatives, including open source.