Blogs

Blogs

Streaming analytics with SAS Event Stream Processing and Apache Nifi

5
By Topics | Internet of Things

Do you have a need for real-time, streaming analytics?

What technology are you considering?

How are you going to enable rapid development and deployment of your analytical models all in real-time?

SAS has partnered the the Hortonworks HDF team to develop a nifi processor that allows SAS machine learning models and advanced analytics to be embedded within the nifi process flow. This technology integration connects Apache Nifi to SAS Event Stream Processing. HDF can be used as the platform for data collection and movement, while SAS can leveraged to deploy online/in-stream analytics.

A Nifi flow diagram with ESP paths highlighted
Integrating SAS Event Stream Processing with Apache Nifi: This image shows the two new SAS Processors for Nifi (click to enlarge).

What is SAS® Event Stream Processing?
SAS Event Stream Processing (ESP) is a real-time, low-latency, high-throughput event processing solution. It has a variety of functions that enable users to configure and deploy real-time analytical models. These functions include machine learning algorithms (such as neutral networks, gradient boosting, decision trees, etc.), text analysis for categorizing text and extracting entities, performing sentiment analysis, advanced pattern matching, in-stream joins, data quality, and much more. This real-time model, or topology, can be developed using a drop-and-drag interface. For advanced users, there's the ability to integrate through Java or C APIs, RESTful web services, out-of-the-box adapters to kafka, hdfs, flume, Cassandra, mqtt, and much more.

What is Apache Nifi?

Apache nifi is a core part of the Hortonworks Data Flow (HDF) offering and a great technology for collecting, processing, and moving data. Initially developed at the NSA, it supports secure, encrypted communication, bi-directional communication, data provenance to track lineage, and a configurable front-end with a variety features.

How does this integration work?

There are two new processors that can be added to the nifi interface:

ListenESP: This processor initiates a listener within nifi that receives events from the SAS Event Stream Processing data stream.

PutESP: This processor sends events from nifi to the SAS Event Stream Processing data stream.

Why does this matter?

  • SAS has a rich set of tools that can be used to quickly build and deploy machine learning models.
  • Drop-and-drag interfaces enable rapid prototyping of analytical models, which generate score code that can be deployed in real-time.
  • Leverage the open architecture of SAS Event Stream Processing and the flexibility to connect to 3rd party systems, APIs, external technology, and (of course) other SAS products.
  • SAS Event Stream Processing and Apache Nifi are now integrated to provide a powerful real-time analytical platform for your business needs.

Last month I was in San Jose at the Hadoop Summit demoing the SAS Event Stream Processing and Nifi integration. You can read more about using these technologies together in the Hortonworks post, Moving streaming analytics out of the data center. For a general introduction to Hadoop, download the e-book, "An early adopter's guide to Hadoop."

Tags big data event stream processing hortonworks Nifi streaming analytics
Share

About Author

Dan Zaratsian

Sr. Solutions Architect

Dan Zaratsian is a Sr. Solutions Architect with SAS' Global Analytics Practice, specializing in real-time event stream processing, text analytics, and machine learning. He works with a variety of technologies, both open-source and enterprise software, in order to design, program, and implement analytical solutions for clients. Dan holds a M.S. in Analytics from North Carolina State's Institute for Advanced Analytics and a Bachelor’s degree in Electrical Engineering from the University of Akron.

5 Comments

  1. Kiran Surve on

    Hi Dan I am currently implementing SAS ESP at a client . I want to know a few insights in configuration . I was trained on the same by SAS labs . Please tell me the implementation approach

  2. Sundaresh on

    Hi Dan,

    Thanks - this is great information. Would you have an opinion as to whether this framework lends itself well to an search - auto-suggest feature on website. Using some javascript/html services to capture searches, I can ingest them into an ESP window which scores for a suggestion based on a model, but the flow tends to be uni-directional (i.e. i can write this to some database / file ).

    Will the NiFi integration solve the last mile problem - i.e. getting the result back to the search bar? I am also thinking of using a Ajax request (which most sites do ) but am not sure if Ajax requests could be made to an ESP project.
    regards
    Sundar

Back to Top