Event stream processing – Tip 1: Don’t be overwhelmed


I believe most people become overwhelmed when considering the data that can be created during event processing. Number one, it is A LOT of data – and number two, the data needs real-time analysis. For the past few years, most of us have been analyzing data after we collected it, not during the event itself.

Christchurch is becoming a "sensing" city

We are surrounded by technology and applications. Most every device can be connected to another device for sharing – or moving – data (except my cell phone, but it could be the operator – ha!). My favorite application is one that does real-time analytics of traffic slowdowns and street closures. My husband and I use one such app when we decide to head up the mountains on a Friday during rush hour. We check it to see where the traffic is slow, and to decide if we should use an alternate route. In this case, sensors transmit event information to a central processor for consumption by the two of us on an application on our cell phones.

If you are working on an initiative to analyze event data, don’t be overwhelmed. Instead, consider this:

  • Not all data needs to be analyzed. When we analyze streaming data it is for a specific purpose. The rules are built in the processing engine to analyze and react on ONLY what we have built.
    • Create questions that you need answers to using this type of processing. For example, on the production floor, how many products deviated from the specifications every hour?
  • While there can be A LOT of data, it is usually short lived, and not worth keeping for years.
  • Usually the results of the event processing data are combined with other data stores for later analysis.
  • The rules can change over time.

About Author

Joyce Norris-Montanari

President of DBTech Solutions, Inc

Joyce Norris-Montanari, CBIP-CDMP, is president of DBTech Solutions, Inc. Joyce advises clients on all aspects of architectural integration, business intelligence and data management. Joyce advises clients about technology, including tools like ETL, profiling, database, quality and metadata. Joyce speaks frequently at data warehouse conferences and is a contributor to several trade publications. She co-authored Data Warehousing and E-Business (Wiley & Sons) with William H. Inmon and others. Joyce has managed and implemented data integrations, data warehouses and operational data stores in industries like education, pharmaceutical, restaurants, telecommunications, government, health care, financial, oil and gas, insurance, research and development and retail. She can be reached at jmontanari@earthlink.net.

Leave A Reply

Back to Top