Event stream processing – Tips 2 and 3: Understand the life cycle of the data, collection and consumption


Determining the life cycle of event stream data requires us to first understand our business and how fast it changes. If event data is analyzed, it makes sense that the results of that analysis would feed another process. For example, a customer relationship management (CRM) system or campaign management system like SalesForce.com. Here are some questions I would ask:

  1. What systems are being fed by this analysis?
    1. How fast do the results need to be fed to this system? We should probably list those systems, as well as what data elements those systems require. This will help us determine how much data manipulation is needed to complete the task.
    2. Does human analysis need to take place? If so, it might be necessary to propagate this data to another data store for future analysis.
  2. How long is this event?
  3. What data elements of this event will be required for the data warehouse? How long will I retain this information for after-event analysis?

White paper cover for Understanding Data Streams in IoTThe above relates immediately to the collection and consumption of event data. If we are collecting real-time data – and assessing, aggregating, correlating and analyzing it – consider these questions:

  1. Where are we storing this data for analysis? This data store would have read and write activity constantly.
  2. Are we off-loading some of this data at a specific hourly interval for further after-event analysis? If so, what type of data store is required? This would be entirely based on how it's accessed and for what purpose. This data could be off-loaded and used for integration into the data warehouse on an hourly basis.
  3. What analysis programs or reporting tools are used for consumption of this data? This will determine design specifications.

About Author

Joyce Norris-Montanari

President of DBTech Solutions, Inc

Joyce Norris-Montanari, CBIP-CDMP, is president of DBTech Solutions, Inc. Joyce advises clients on all aspects of architectural integration, business intelligence and data management. Joyce advises clients about technology, including tools like ETL, profiling, database, quality and metadata. Joyce speaks frequently at data warehouse conferences and is a contributor to several trade publications. She co-authored Data Warehousing and E-Business (Wiley & Sons) with William H. Inmon and others. Joyce has managed and implemented data integrations, data warehouses and operational data stores in industries like education, pharmaceutical, restaurants, telecommunications, government, health care, financial, oil and gas, insurance, research and development and retail. She can be reached at jmontanari@earthlink.net.

Leave A Reply

Back to Top