Share
Embed
VISP: An Ecosystem for Elastic Data Stream Processing for the Internet of Things
Presentation of our research paper "VISP: An Ecosystem for Elastic Data Stream Processing for the Internet of Things" at the IEEE EDOC 2016 conference in Vienna, Austria.
Christoph Hochreiner
September 09, 2016
Transcript
26 slides
-
VISP: An Ecosystem for Elastic Data Stream Processing for the
Internet of Things Christoph Hochreiner, Michael Vögler, Philipp Waibel and Schahram Dustdar -
Motivational Scenario 3 Headquarter Factory Analytics Provider Machine Machine Manufacturing
Machine Transform Temperature Temperature Sensor Rotation Speed Sensor Amount of Cycles Pressure Sensor Transform Pressure Filter Temperature Filter Pressure Filter Rotation Recordings Reason on KPI Update Production Plan Update Maintenance Plan Record Historic Data Reason on Shutdown Notify User Sensor Stream Processing Operator External Stream Processing Operator User Interaction Dataflow -
Motivational Scenario 4 Headquarter Factory Analytics Provider Machine Machine Manufacturing
Machine Transform Temperature Temperature Sensor Rotation Speed Sensor Amount of Cycles Pressure Sensor Transform Pressure Filter Temperature Filter Pressure Filter Rotation Recordings Reason on KPI Update Production Plan Update Maintenance Plan Record Historic Data Reason on Shutdown Notify User Sensor Stream Processing Operator External Stream Processing Operator User Interaction Dataflow Hybrid deployment ‣ Deployment restrictions - legal restrictions ‣ Performance requirements - low latency -
Motivational Scenario 5 Headquarter Factory Analytics Provider Machine Machine Manufacturing
Machine Transform Temperature Temperature Sensor Rotation Speed Sensor Amount of Cycles Pressure Sensor Transform Pressure Filter Temperature Filter Pressure Filter Rotation Recordings Reason on KPI Update Production Plan Update Maintenance Plan Record Historic Data Reason on Shutdown Notify User Sensor Stream Processing Operator External Stream Processing Operator User Interaction Dataflow Reuse of stream processing operators ‣ Allow decomposition of stream processing topologies ‣ Create a marketplace for individual stream processing operators ‣ Develop monetization schemes to attract stream processing operator developers -
Motivational Scenario 6 Headquarter Factory Analytics Provider Machine Machine Manufacturing
Machine Transform Temperature Temperature Sensor Rotation Speed Sensor Amount of Cycles Pressure Sensor Transform Pressure Filter Temperature Filter Pressure Filter Rotation Recordings Reason on KPI Update Production Plan Update Maintenance Plan Record Historic Data Reason on Shutdown Notify User Sensor Stream Processing Operator External Stream Processing Operator User Interaction Dataflow Ease of designing new stream processing topologies ‣ Provide an ecosystem ranging from the design to the enactment ‣ Provide a topology builder to support the domain experts -
Related Systems 7 Grovestreams IFTTT Apache Edgent ThingStore ThingWorx Amazon
Web Services IoT Microsoft Stream Analytics Google Cloud Dataflow Runtime ✔ ✔ ✔ ✔ ✔ ✔ ✔ Marketplace ✔ ✔ ✔ Topology Builder ✔ ✔ On-Premise Deployment ✔ -
VISP 9 Operator Instance VISP Runtime Cloud Resource Handling Computational
Resources Operator Instance Management Elasticity Usage Monitor Reasoner Communication Message Infrastructure Operator Instance Operator Instance Deployment Failure Management Availability Monitoring Processing Logic System Monitor Configuration API Failure Monitor Shared State Operator Instan VISP Runtime Cloud Resource Handling Computational Reso VISP Marketplace Operator Registry Configuration Storage Configuration Operator Instance Management Operator Image Storage Topology Builder Billing Infrastructure Topology Configuration API Elasticity Usage Monitor Reasoner Communication Message Infrastructure Operator Instance Operator Instance Deployment Failure Management Availability Monitoring Processing Logic System Monitor Configuration AP Failure Monitor Shared State VISP Runtime Cloud Resource Handling VISP Marketplace Operator Registry Configuration Storage Configuration Operator Instance Management Operator Image Storage Topology Builder Billing Infrastructure Topology Configuration API Elasticity Usage Monitor Reasoner Operator Instance Deployment Failure Management Availability Monitoring -
Stream Processing Topology Realization 10 Topology Design Topology Upload to
VISP Runtime Topology Distribution to affected VISP Runtimes Operator Image Provision and Billing Operations Operator Instantiation Message Infrastructure Configuration -
Topology Design ‣Design the desired topology ‣Search the marketplace for
existing suitable operators 11 Operator Instance VISP Runtime Cloud Resource Handling Computational Resources VISP Marketplace Operator Registry Configuration Storage Configuration Operator Instance Management Operator Image Storage Topology Builder Billing Infrastructure Topology Configuration API Elasticity Usage Monitor Reasoner Communication Message Infrastructure Operator Instance Operator Instance Deployment Failure Management Availability Monitoring Processing Logic System Monitor Configuration API Failure Monitor Shared State Topology Design Topology Upload to VISP Runtime Topology Distribution to affected VISP Runtimes Operator Image Provision and Billing Operations Operator Instantiation Message Infrastructure Configuration -
Operator Image Operator images represent processing operators including their execution
environment Requirements ‣Connect to message infrastructure (AMQP/MQTT) ‣Implement failure monitor ‣Packaged in a predefined format (Docker images) 12 Topology Design Topology Upload to VISP Runtime Topology Distribution to affected VISP Runtimes Operator Image Provision and Billing Operations Operator Instantiation Message Infrastructure Configuration -
Topology Definition Topology description language ‣Based on SPL by IBM
‣Considers hybrid deployment ‣Considers arbitrary operator types 13 Topology Design Topology Upload to VISP Runtime Topology Distribution to affected VISP Runtimes Operator Image Provision and Billing Operations Operator Instantiation Message Infrastructure Configuration -
Operator Procurement ‣Obtain the operator images from the operator registry
‣Perform payment activities for the operator images 14 Operator Instance VISP Runtime Cloud Resource Handling Computational Resources VISP Marketplace Operator Registry Configuration Storage Configuration Operator Instance Management Operator Image Storage Topology Builder Billing Infrastructure Topology Configuration API Elasticity Usage Monitor Reasoner Communication Message Infrastructure Operator Instance Operator Instance Deployment Failure Management Availability Monitoring Processing Logic System Monitor Configuration API Failure Monitor Shared State Topology Design Topology Upload to VISP Runtime Topology Distribution to affected VISP Runtimes Operator Image Provision and Billing Operations Operator Instantiation Message Infrastructure Configuration -
Operator Instantiation ‣Obtain computational resource from a cloud ‣Instantiate the
operators on these computational resources 15 Operator Instance VISP Runtime Cloud Resource Handling Computational Resources VISP Marketplace Operator Registry Configuration Storage Configuration Operator Instance Management Operator Image Storage Topology Builder Billing Infrastructure Topology Configuration API Elasticity Usage Monitor Reasoner Communication Message Infrastructure Operator Instance Operator Instance Deployment Failure Management Availability Monitoring Processing Logic System Monitor Configuration API Failure Monitor Shared State Topology Design Topology Upload to VISP Runtime Topology Distribution to affected VISP Runtimes Operator Image Provision and Billing Operations Operator Instantiation Message Infrastructure Configuration -
Topology Configuration ‣Wiring of operator instances according to the topology
based on the message infrastructure 16 Operator Instance VISP Runtime Cloud Resource Handling Computational Resources VISP Marketplace Operator Registry Configuration Storage Configuration Operator Instance Management Operator Image Storage Topology Builder Billing Infrastructure Topology Configuration API Elasticity Usage Monitor Reasoner Communication Message Infrastructure Operator Instance Operator Instance Deployment Failure Management Availability Monitoring Processing Logic System Monitor Configuration API Failure Monitor Shared State Topology Design Topology Upload to VISP Runtime Topology Distribution to affected VISP Runtimes Operator Image Provision and Billing Operations Operator Instantiation Message Infrastructure Configuration -
Topology Operations ‣Constant monitoring on the availability of the operator
instances and the load of these ‣Replicating individual operators on demand to provide failover mechanism or increase the processing capabilities 17 Operator Instance VISP Runtime Cloud Resource Handling Computational Resources VISP Marketplace Operator Registry Configuration Storage Configuration Operator Instance Management Operator Image Storage Topology Builder Billing Infrastructure Topology Configuration API Elasticity Usage Monitor Reasoner Communication Message Infrastructure Operator Instance Operator Instance Deployment Failure Management Availability Monitoring Processing Logic System Monitor Configuration API Failure Monitor Shared State Topology Design Topology Upload to VISP Runtime Topology Distribution to affected VISP Runtimes Operator Image Provision and Billing Operations Operator Instantiation Message Infrastructure Configuration -
Implementation 19 Operator Instance VISP Runtime Cloud Resource Handling Computational
Resources VISP Marketplace Operator Registry Configuration Storage Configuration Operator Instance Management Operator Image Storage Topology Builder Billing Infrastructure Topology Configuration API Elasticity Usage Monitor Reasoner Communication Message Infrastructure Operator Instance Operator Instance Deployment Failure Management Availability Monitoring Processing Logic System Monitor Configuration API Failure Monitor Shared State https://github.com/chochreiner/VISP-Runtime -
Use Case Scenario 21 VISP Runtime at Factory 1 VISP
Runtime at Factory 2 VISP Runtime at Factory 3 External Operators VISP Runtime on Cloud Resources User VISP Marketplace Machine Machine Manufacturing Machine Transform Temperature Temperature Sensor Rotation Speed Sensor Amount of Cycles Pressure Sensor Transform Pressure Filter Temperature Filter Pressure Filter Rotation Recordings Reason on KPI Update Production Plan Update Maintenance Plan Record Historic Data Reason on Shutdown Notify User Sensor Stream Processing Operator External Stream Processing Operator User Interaction Dataflow -
Use Case Walkthrough ‣Design the stream processing topology ‣Implement the
required stream processing operators and upload them to the marketplace ‣Deploy the four VISP runtime instances ‣Upload the topology to one VISP runtime instance ‣Let the VISP runtime obtain the operator images and instantiate the operator instances ‣Process data with VISP 22 -
Contributions ‣Concept for a holistic ecosystem covering the lifecycle of
stream processing topologies ‣Show the feasibility of reusable stream processing operators and the hybrid deployment across multiple clouds ‣Implemented an open source toolkit, which can be used to evaluate future aspects for data stream processing topologies 24 -
Outlook ‣Resource elasticity and on demand provisioning ‣Business models for
IoT scenarios ‣Usability aspects for designing future topology builders 25