VISP: An Ecosystem for Elastic Data Stream Processing for the Internet of Things

Presentation of our research paper "VISP: An Ecosystem for Elastic Data Stream Processing for the Internet of Things" at the IEEE EDOC 2016 conference in Vienna, Austria.

Transcript

1. VISP: An Ecosystem for Elastic Data Stream Processing for the Internet of Things Christoph Hochreiner, Michael Vögler, Philipp Waibel and Schahram Dustdar
2. Motivational Scenario
3. Motivational Scenario 3 Headquarter Factory Analytics Provider Machine Machine Manufacturing Machine Transform Temperature Temperature Sensor Rotation Speed Sensor Amount of Cycles Pressure Sensor Transform Pressure Filter Temperature Filter Pressure Filter Rotation Recordings Reason on KPI Update Production Plan Update Maintenance Plan Record Historic Data Reason on Shutdown Notify User Sensor Stream Processing Operator External Stream Processing Operator User Interaction Dataflow
4. Motivational Scenario 4 Headquarter Factory Analytics Provider Machine Machine Manufacturing Machine Transform Temperature Temperature Sensor Rotation Speed Sensor Amount of Cycles Pressure Sensor Transform Pressure Filter Temperature Filter Pressure Filter Rotation Recordings Reason on KPI Update Production Plan Update Maintenance Plan Record Historic Data Reason on Shutdown Notify User Sensor Stream Processing Operator External Stream Processing Operator User Interaction Dataflow Hybrid deployment ‣ Deployment restrictions - legal restrictions ‣ Performance requirements - low latency
5. Motivational Scenario 5 Headquarter Factory Analytics Provider Machine Machine Manufacturing Machine Transform Temperature Temperature Sensor Rotation Speed Sensor Amount of Cycles Pressure Sensor Transform Pressure Filter Temperature Filter Pressure Filter Rotation Recordings Reason on KPI Update Production Plan Update Maintenance Plan Record Historic Data Reason on Shutdown Notify User Sensor Stream Processing Operator External Stream Processing Operator User Interaction Dataflow Reuse of stream processing operators ‣ Allow decomposition of stream processing topologies ‣ Create a marketplace for individual stream processing operators ‣ Develop monetization schemes to attract stream processing operator developers
6. Motivational Scenario 6 Headquarter Factory Analytics Provider Machine Machine Manufacturing Machine Transform Temperature Temperature Sensor Rotation Speed Sensor Amount of Cycles Pressure Sensor Transform Pressure Filter Temperature Filter Pressure Filter Rotation Recordings Reason on KPI Update Production Plan Update Maintenance Plan Record Historic Data Reason on Shutdown Notify User Sensor Stream Processing Operator External Stream Processing Operator User Interaction Dataflow Ease of designing new stream processing topologies ‣ Provide an ecosystem ranging from the design to the enactment ‣ Provide a topology builder to support the domain experts
7. Related Systems 7 Grovestreams IFTTT Apache Edgent ThingStore ThingWorx Amazon Web Services IoT Microsoft Stream Analytics Google Cloud Dataflow Runtime ✔ ✔ ✔ ✔ ✔ ✔ ✔ Marketplace ✔ ✔ ✔ Topology Builder ✔ ✔ On-Premise Deployment ✔
8. VISP Vienna Ecosystem for Elastic Stream Processing
9. VISP 9 Operator Instance VISP Runtime Cloud Resource Handling Computational Resources Operator Instance Management Elasticity Usage Monitor Reasoner Communication Message Infrastructure Operator Instance Operator Instance Deployment Failure Management Availability Monitoring Processing Logic System Monitor Configuration API Failure Monitor Shared State Operator Instan VISP Runtime Cloud Resource Handling Computational Reso VISP Marketplace Operator Registry Configuration Storage Configuration Operator Instance Management Operator Image Storage Topology Builder Billing Infrastructure Topology Configuration API Elasticity Usage Monitor Reasoner Communication Message Infrastructure Operator Instance Operator Instance Deployment Failure Management Availability Monitoring Processing Logic System Monitor Configuration AP Failure Monitor Shared State VISP Runtime Cloud Resource Handling VISP Marketplace Operator Registry Configuration Storage Configuration Operator Instance Management Operator Image Storage Topology Builder Billing Infrastructure Topology Configuration API Elasticity Usage Monitor Reasoner Operator Instance Deployment Failure Management Availability Monitoring
10. Stream Processing Topology Realization 10 Topology Design Topology Upload to VISP Runtime Topology Distribution to affected VISP Runtimes Operator Image Provision and Billing Operations Operator Instantiation Message Infrastructure Configuration
11. Topology Design ‣Design the desired topology ‣Search the marketplace for existing suitable operators 11 Operator Instance VISP Runtime Cloud Resource Handling Computational Resources VISP Marketplace Operator Registry Configuration Storage Configuration Operator Instance Management Operator Image Storage Topology Builder Billing Infrastructure Topology Configuration API Elasticity Usage Monitor Reasoner Communication Message Infrastructure Operator Instance Operator Instance Deployment Failure Management Availability Monitoring Processing Logic System Monitor Configuration API Failure Monitor Shared State Topology Design Topology Upload to VISP Runtime Topology Distribution to affected VISP Runtimes Operator Image Provision and Billing Operations Operator Instantiation Message Infrastructure Configuration
12. Operator Image Operator images represent processing operators including their execution environment Requirements ‣Connect to message infrastructure (AMQP/MQTT) ‣Implement failure monitor ‣Packaged in a predefined format (Docker images) 12 Topology Design Topology Upload to VISP Runtime Topology Distribution to affected VISP Runtimes Operator Image Provision and Billing Operations Operator Instantiation Message Infrastructure Configuration
13. Topology Definition Topology description language ‣Based on SPL by IBM ‣Considers hybrid deployment ‣Considers arbitrary operator types 13 Topology Design Topology Upload to VISP Runtime Topology Distribution to affected VISP Runtimes Operator Image Provision and Billing Operations Operator Instantiation Message Infrastructure Configuration
14. Operator Procurement ‣Obtain the operator images from the operator registry ‣Perform payment activities for the operator images 14 Operator Instance VISP Runtime Cloud Resource Handling Computational Resources VISP Marketplace Operator Registry Configuration Storage Configuration Operator Instance Management Operator Image Storage Topology Builder Billing Infrastructure Topology Configuration API Elasticity Usage Monitor Reasoner Communication Message Infrastructure Operator Instance Operator Instance Deployment Failure Management Availability Monitoring Processing Logic System Monitor Configuration API Failure Monitor Shared State Topology Design Topology Upload to VISP Runtime Topology Distribution to affected VISP Runtimes Operator Image Provision and Billing Operations Operator Instantiation Message Infrastructure Configuration
15. Operator Instantiation ‣Obtain computational resource from a cloud ‣Instantiate the operators on these computational resources 15 Operator Instance VISP Runtime Cloud Resource Handling Computational Resources VISP Marketplace Operator Registry Configuration Storage Configuration Operator Instance Management Operator Image Storage Topology Builder Billing Infrastructure Topology Configuration API Elasticity Usage Monitor Reasoner Communication Message Infrastructure Operator Instance Operator Instance Deployment Failure Management Availability Monitoring Processing Logic System Monitor Configuration API Failure Monitor Shared State Topology Design Topology Upload to VISP Runtime Topology Distribution to affected VISP Runtimes Operator Image Provision and Billing Operations Operator Instantiation Message Infrastructure Configuration
16. Topology Configuration ‣Wiring of operator instances according to the topology based on the message infrastructure 16 Operator Instance VISP Runtime Cloud Resource Handling Computational Resources VISP Marketplace Operator Registry Configuration Storage Configuration Operator Instance Management Operator Image Storage Topology Builder Billing Infrastructure Topology Configuration API Elasticity Usage Monitor Reasoner Communication Message Infrastructure Operator Instance Operator Instance Deployment Failure Management Availability Monitoring Processing Logic System Monitor Configuration API Failure Monitor Shared State Topology Design Topology Upload to VISP Runtime Topology Distribution to affected VISP Runtimes Operator Image Provision and Billing Operations Operator Instantiation Message Infrastructure Configuration
17. Topology Operations ‣Constant monitoring on the availability of the operator instances and the load of these ‣Replicating individual operators on demand to provide failover mechanism or increase the processing capabilities 17 Operator Instance VISP Runtime Cloud Resource Handling Computational Resources VISP Marketplace Operator Registry Configuration Storage Configuration Operator Instance Management Operator Image Storage Topology Builder Billing Infrastructure Topology Configuration API Elasticity Usage Monitor Reasoner Communication Message Infrastructure Operator Instance Operator Instance Deployment Failure Management Availability Monitoring Processing Logic System Monitor Configuration API Failure Monitor Shared State Topology Design Topology Upload to VISP Runtime Topology Distribution to affected VISP Runtimes Operator Image Provision and Billing Operations Operator Instantiation Message Infrastructure Configuration
18. Implementation
19. Implementation 19 Operator Instance VISP Runtime Cloud Resource Handling Computational Resources VISP Marketplace Operator Registry Configuration Storage Configuration Operator Instance Management Operator Image Storage Topology Builder Billing Infrastructure Topology Configuration API Elasticity Usage Monitor Reasoner Communication Message Infrastructure Operator Instance Operator Instance Deployment Failure Management Availability Monitoring Processing Logic System Monitor Configuration API Failure Monitor Shared State https://github.com/chochreiner/VISP-Runtime
20. Use Case Evaluation
21. Use Case Scenario 21 VISP Runtime at Factory 1 VISP Runtime at Factory 2 VISP Runtime at Factory 3 External Operators VISP Runtime on Cloud Resources User VISP Marketplace Machine Machine Manufacturing Machine Transform Temperature Temperature Sensor Rotation Speed Sensor Amount of Cycles Pressure Sensor Transform Pressure Filter Temperature Filter Pressure Filter Rotation Recordings Reason on KPI Update Production Plan Update Maintenance Plan Record Historic Data Reason on Shutdown Notify User Sensor Stream Processing Operator External Stream Processing Operator User Interaction Dataflow
22. Use Case Walkthrough ‣Design the stream processing topology ‣Implement the required stream processing operators and upload them to the marketplace ‣Deploy the four VISP runtime instances ‣Upload the topology to one VISP runtime instance ‣Let the VISP runtime obtain the operator images and instantiate the operator instances ‣Process data with VISP 22
23. Conclusion
24. Contributions ‣Concept for a holistic ecosystem covering the lifecycle of stream processing topologies ‣Show the feasibility of reusable stream processing operators and the hybrid deployment across multiple clouds ‣Implemented an open source toolkit, which can be used to evaluate future aspects for data stream processing topologies 24
25. Outlook ‣Resource elasticity and on demand provisioning ‣Business models for IoT scenarios ‣Usability aspects for designing future topology builders 25
26. Q & A Christoph Hochreiner c.hochreiner@infosys.tuwien.ac.at