Event stream processing and/or real-time processing


Using cellphone and considering event stream processing versus real-time processingEvent stream processing (ESP) and real-time processing (RTP) so often come up in the same conversation that it begs the question if they are one and the same. The short answer is yes and/or no. But since I don’t need the other kind of ESP to know that you won’t find that answer helpful, permit me to respond to your query by streaming a few more, hopefully eventful, words.

RTP typically refers to a system or application that is time-sensitive, which means its output is guaranteed within a real-time deadline. For example, a trading system that guarantees the query of a stock’s current price will be returned within 10 milliseconds. Although the stock price is likely fluctuating, this system doesn’t execute the query until requested. But it does return the result quickly enough to represent the current stock price.

ESP, on the other hand, typically refers to continuously querying data in motion (i.e., event streams) as it flows through a system or application. There are usually no compulsory time limitations in ESP. Instead the goal is to detect meaningful patterns within event streams and determine if an individual event within the stream should trigger an immediate action. For example, a fraud monitoring system that continuously queries credit card transaction flows to detect patterns of unusual activity and stop a fraudulent transaction before it’s executed.

ESP – More real-time than near-real-time

Many systems or applications only enable analysis after an event has already happened and the data describing it has already been stored. This often triggers the semantic battle of describing a system or application as near-real-time as opposed to real-time, to indicate that there is some latency involved. By contrast, instead of storing data, ESP stores queries and streams data through them. By analyzing events before their data is stored, ESP provides immediate insights for immediate action. So, if you really wanted to further the semantic battle, you could call ESP more real-time than near-real-time.

ESP is often described as capturing the real-time value of data (hence the confusion with RTP) before it’s lost in the time lag between analysis and action. ESP enables you to continuously know what events are occurring in streaming data so you can appropriately react to those events as they happen. As David Loshin blogged, ESP not only automates reactions to events but also encompasses analytics to be smart about those reactions.

Indeed, there’s a lot of overlap between ESP and RTP, which is why they often come up in the same conversation. Here’s a simple (meaning, not true in all cases) Shakespearean rule of thumb: the query’s the thing wherein I’ll catch the distinction of this thing. In other words, on-demand queries versus continuous querying. If the query happens at a particular point in time, it’s RTP. If the query happens all the time, it’s ESP.

Learn more in – Channeling Streaming Data for Competitive Advantage

About Author

Jim Harris

Blogger-in-Chief at Obsessive-Compulsive Data Quality (OCDQ)

Jim Harris is a recognized data quality thought leader with 25 years of enterprise data management industry experience. Jim is an independent consultant, speaker, and freelance writer. Jim is the Blogger-in-Chief at Obsessive-Compulsive Data Quality, an independent blog offering a vendor-neutral perspective on data quality and its related disciplines, including data governance, master data management, and business intelligence.

Related Posts

Leave A Reply

Back to Top