Share
Embed
Transcript
119 slides
-
Outline ‣ Preliminaries ‣ Motivational Scenario ‣ Contributions • The
VISP Ecosystem • The Vienna Topology Description Language • Cost-efficient Enactment of Stream Processing Applications ‣ Conclusion 2 -
Data Stream Processing Stream Processing Application (SPA) Data Source Data
Sink Raw Data Value Added Data 4 -
Stream Processing Application (SPA) Data Stream Processing Data Source Data
Sink Transformator Filter Aggregator 5 -
Stream Processing Application (SPA) O1 Stream Processing Engine (SPE) Data
Stream Processing Data Source Data Sink Transformator Filter Aggregator O1 O2 O3 6 -
Motivational Scenario Goal Monitor manufacturing machines that are located in
different geographic locations in a central location to optimize the overall productivity. 8 -
Motivational Scenario Manufacturing Machine Manufacturing Machine Manufacturing Machine Availability Sensor
(S1) Production Data (S2) Temperature Sensor (S3) Filter Availability (O1) Parse and Distribute Data (O3) Calculate Performance (O4) Calculate Availability (O5) Calculate Quality (O6) Monitor Temperature (O2) Calculate OEE (O7) Generate Report (O8) Inform User (O9) 9 -
Manufacturing Machine Manufacturing Machine Manufacturing Machine Availability Sensor (S1) Production
Data (S2) Temperature Sensor (S3) Filter Availability (O1) Parse and Distribute Data (O3) Calculate Performance (O4) Calculate Availability (O5) Calculate Quality (O6) Monitor Temperature (O2) Calculate OEE (O7) Generate Report (O8) Inform User (O9) De-Centralized Deployment + Network efficient - Redundant infrastructure - Hard to manage 10 -
- Not network efficient + Hardware efficient + Easy to
manage Manufacturing Machine Manufacturing Machine Manufacturing Machine Availability Sensor (S1) Production Data (S2) Temperature Sensor (S3) Filter Availability (O1) Parse and Distribute Data (O3) Calculate Performance (O4) Calculate Availability (O5) Calculate Quality (O6) Monitor Temperature (O2) Calculate OEE (O7) Generate Report (O8) Inform User (O9) Centralized Deployment 11 -
Research Question 1 How can geographically distributed data streams be
processed efficiently? 12 -
Creation of Stream Processing Applications Manufacturing Machine Manufacturing Machine Manufacturing
Machine Availability Sensor (S1) Production Data (S2) Temperature Sensor (S3) Filter Availability (O1) Parse and Distribute Data (O3) Calculate Performance (O4) Calculate Availability (O5) Calculate Quality (O6) Monitor Temperature (O2) Calculate OEE (O7) Generate Report (O8) Inform User (O9) ‣ SPAs are mostly implemented from scratch ‣ Service level objectives (SLOs) for operators are not supported ‣ No support for distributed SPEs State of the Art 13 -
‣ Domain experts can create SPAs ‣ Reuse of existing
operators ‣ Consider non-functional aspects like the geographic location or SLOs Creation of Stream Processing Applications Manufacturing Machine Manufacturing Machine Manufacturing Machine Availability Sensor (S1) Production Data (S2) Temperature Sensor (S3) Filter Availability (O1) Parse and Distribute Data (O3) Calculate Performance (O4) Calculate Availability (O5) Calculate Quality (O6) Monitor Temperature (O2) Calculate OEE (O7) Generate Report (O8) Inform User (O9) Goal 14 -
Research Question 2 How can SPAs be described to allow
a distributed deployment and respect SLOs? 15 -
Operation of Stream Processing Applications ‣ Ensure enough computational resources
for SPAs to process the data without delay ‣ Minimize the cost for computational resources ‣ Minimize potential penalty cost for delayed data 16 -
Research Question 3 How can SPAs be optimally executed on
computational resources? 17 -
Contribution 1 A system design for a holistic stream processing
ecosystem in distributed environments C. Hochreiner, S. Schulte, S. Dustdar, and F. Lecue, Elastic Stream Processing for Distributed Environments, IEEE Internet Computing, 2015. C. Hochreiner, M. Vögler, P. Waibel, and S. Dustdar, VISP: An Ecosystem for Elastic Data Stream Processing for the Internet of Things, EDOC 2016. 19 -
Manufacturing Machine Manufacturing Machine Manufacturing Machine Availability Sensor (S1) Production
Data (S2) Temperature Sensor (S3) Filter Availability (O1) Parse and Distribute Data (O3) Calculate Performance (O4) Calculate Availability (O5) Calculate Quality (O6) Monitor Temperature (O2) Calculate OEE (O7) Generate Report (O8) Inform User (O9) Distributed Deployment - Needs a new system design + Network & hardware efficient + Easy to manage 20 -
Challenges for a New System Design ‣ Distributed deployment of
SPAs ‣ Create an ecosystem that supports the reuse of operators to ease the creation of new SPAs ‣ Elastic provisioning of processing capabilities at runtime 21 -
IBM System S Apache Spark / Storm AWS, Google Cloud,
Microsoft Azure ThingWorx ThingStore [1] Distributed Apache Storm [2] StreamCloud [3] Heterogenous Clouds ✓ Resource Elasticity ( ✓ ) ✓ ✓ ✓ ✓ Cost Efficiency ( ✓ ) ✓ ✓ Marketplace ( ✓ ) ✓ ✓ Related Stream Processing Engines and Platforms for the IoT 22 -
System Design for the VISP Ecosystem Infrastructure Data Sources VISP
Marketplace VISP Runtime Operational Layer Communication Layer Infrastructure Layer Stream Processing Application Cross-cutting Concerns 23 -
Spain United Kingdom Germany O1 O2 O8 O9 O3 O4
O5 O6 O7 Sweden O1 O2 O1 O2 O3 O4 O5 O6 O7 O3 O4 O5 O6 O7 O1 O2 S1 S2 S3 S1 S2 S3 S1 S2 S3 S1 S2 S3 SN Sensor ON Operator Dataflow Evaluation Scenario 24 -
Evaluation Setup VISP Runtime VISP Runtime VISP Runtime VISP Runtime
VISP Marketplace ‣ Define the topology for the SPA ‣ Deploy the VISP Runtime instances in an OpenStack- based private cloud 25 -
Evaluation Walk Through VISP Runtime VISP Runtime VISP Runtime VISP
Runtime VISP Marketplace 1. Upload the topology to one VISP Runtime 26 -
Evaluation Walk Through VISP Runtime VISP Runtime VISP Runtime VISP
Runtime VISP Marketplace 2. Topology is distributed among all VISP Runtimes 27 -
Evaluation Walk Through VISP Runtime VISP Runtime VISP Runtime VISP
Runtime VISP Marketplace 3. Operators are fetched from the VISP Marketplace and the data flow is configured by each VISP Runtime 28 -
Evaluation Walk Through VISP Runtime VISP Runtime VISP Runtime VISP
Runtime VISP Marketplace 4. Data is processed 29 -
Contribution Summary ‣ Introduced a system design to enable the
distributed operation of SPAs based on the autonomous coordination among individual runtimes ‣ Proposed the VISP Marketplace to support the reuse of operators and incentivize 3rd party developers to contribute ‣ Achieve network-efficiency by already filtering data at the edge of the network ‣ Achieve resource-efficiency by getting rid of redundant deployments 30 -
Contribution 2 An approach to define quality of service- and
location-specific aspects for SPAs C. Hochreiner, M. Nardelli, B. Knasmueller, S. Schulte, and S. Dustdar, VTDL: A Notation for Data Stream Processing Applications, SOSE 2018. 31 -
Non-functional Aspects Define non-functional aspects for the SPA Aspects for
Designing Stream Processing Applications Basic Functionality Define the orchestration of the individual operators and the data flow through the SPA 32 -
Non-functional Aspects ‣ Deployment preferences to support distributed deployments ‣
SLOs to describe performance aspects of operators ‣ Definition of failure compensation strategies ‣ Support for operator annotations to perform integrity checks or semantic match-making ‣ Model different data transfer modes among the operators 33 -
Related Work on Topology Description Languages Dataflow Apache Beam SPL
CQL Apache Spark / Storm Deployment Preferences ( ✓ ) QoS Aspects ( ✓ ) Failure Compensation ( ✓ ) Semantic Annotation ✓ Data Transfer Aspects better motivate the + present the desig vtDL 34 -
Manufacturing Machine Manufacturing Machine Manufacturing Machine Availability Sensor (S1) Production
Data (S2) Temperature Sensor (S3) Filter Availability (O1) Parse and Distribute Data (O3) Calculate Performance (O4) Calculate Availability (O5) Calculate Quality (O6) Monitor Temperature (O2) Calculate OEE (O7) Generate Report (O8) Inform User (O9) Vienna Topology Description Language (VTDL) $temperature = Source() { concreteLocation : ":::::ffff:8083:c001/cpu", type : "temperatureSensor", outputFormat : "temperature" } $monitor = Operator($temperature) { allowedLocations : *, concreteLocation : ":::::ffff:8083:c001/cpu", inputFormat : "temperature", type : "monitorTemperature", outputFormat : "alert", stateful : "false", replicationAllowed : "true", responseTime : "0.5", compensation : "mailto:[email protected]" } $informUser = Operator($monitor) { allowedLocations : ":::::ffff:8083:c001" ":::::ffff:8083:c002", inputFormat : "alert, report", type : "informUser", outputFormat : "message", queueLength : "200", protocol : "microbatch/50items", compensation : "deploy:www.visp.io/backup.vtdl" } 35 -
Spain United Kingdom Germany O1 O2 O8 O9 O3 O4
O5 O6 O7 Sweden O1 O2 O1 O2 O3 O4 O5 O6 O7 O3 O4 O5 O6 O7 O1 O2 S1 S2 S3 S1 S2 S3 S1 S2 S3 S1 S2 S3 SN Sensor ON Operator Dataflow Evaluation Scenario 1 - New Topology in One Location 36 -
Spain United Kingdom Germany O1 O2 O8 O9 O3 O4
O5 O6 O7 Sweden O1 O2 O1 O2 O3 O4 O5 O6 O7 O3 O4 O5 O6 O7 O1 O2 S1 S2 S3 S1 S2 S3 S1 S2 S3 S1 S2 S3 SN Sensor ON Operator Dataflow Evaluation Scenario 2 - New Topology across Four Locations Amazon EC2 Tokyo Amazon EC2 Frankfurt Amazon EC2 Ohio 37 -
Spain United Kingdom Germany O1 O2 O8 O9 O3 O4
O5 O6 O7 Sweden O1 O2 O1 O2 O3 O4 O5 O6 O7 O3 O4 O5 O6 O7 O1 O2 S1 S2 S3 S1 S2 S3 S1 S2 S3 S1 S2 S3 SN Sensor ON Operator Dataflow Evaluation Scenario 3 - Network Disruption 38 -
Spain United Kingdom Germany O1 O2 O8 O9 O3 O4
O5 O6 O7 Sweden O1 O2 O1 O2 O3 O4 O5 O6 O7 O3 O4 O5 O6 O7 O1 O2 S1 S2 S3 S1 S2 S3 S1 S2 S3 S1 S2 S3 SN Sensor ON Operator Dataflow Old dataflow New Dataflow Evaluation Scenario 3 - Network Disruption 39 -
Spain United Kingdom Germany O1 O2 O8 O9 O3 O4
O5 O6 O7 Sweden O1 O2 O1 O2 O3 O4 O5 O6 O7 O3 O4 O5 O6 O7 O1 O2 S1 S2 S3 S1 S2 S3 S1 S2 S3 S1 S2 S3 SN Sensor ON Operator Dataflow Evaluation Scenario 4 - Load Reduction 40 -
Evaluation Scenario 4 - Load Reduction Spain United Kingdom Germany
O1 O2 O8 O9 O3 O4 O5 O6 O7 Sweden O1 O2 O1 O2 O3 O4 O5 O6 O7 O3 O4 O5 O6 O7 O1 O2 S1 S2 S3 S1 S2 S3 S1 S2 S3 S1 S2 S3 SN Sensor ON Operator Dataflow Old dataflow New Dataflow 41 -
Spain United Kingdom Germany O1 O2 O8 O9 O3 O4
O5 O6 O7 Sweden O1 O2 O1 O2 O3 O4 O5 O6 O7 O3 O4 O5 O6 O7 O1 O2 S1 S2 S3 S1 S2 S3 S1 S2 S3 S1 S2 S3 SN Sensor ON Operator Dataflow Evaluation Scenario 5 - Bugfix for an Operator 42 -
Evaluation Baseline (Apache Storm) VTDL-based New topology in one location
54.98 s 54.98 s New topology across four locations 68.10 s 55.95 s Network disruption — 7.53 s Load reduction 54.05 s 3.30 s Bugfix for an operator 55.62 s 5.69 s 43 -
Contribution Summary ‣ Introduced a topology description approach that supports
non-functional aspects ‣ Evaluate failure compensation mechanism across different VISP Runtimes ‣ Showed a reduced downtime when using the VTDL-based approach compared to state of the art ones 44 -
Contribution 3 Cost-efficient resource provisioning approaches for SPAs C. Hochreiner,
M. Vögler, S. Schulte, and S. Dustdar, Elastic Stream Processing for the Internet of Things, CLOUD 2016. C. Hochreiner, M. Vögler, S. Schulte, S. Dustdar, Cost-efficient enactment of stream processing topologies, PeerJ Computer Science, 2017. 45 -
Optimization Model min X p2P pi + u · N
+ d · N <latexit sha1_base64="1Y8N28G+CXvJS/CnK7ZQG+jHk1A=">AAACInicbZDLSgMxFIYz3q23qks3B4sgCGUqgpdV0Y0rqWBV6JQhk0nbYJIZkjNiGfoubnwVNy68rQQfxrRW0NYfAh//OYeT80epFBZ9/8ObmJyanpmdmy8sLC4trxRX1y5tkhnG6yyRibmOqOVSaF5HgZJfp4ZTFUl+Fd2c9OtXt9xYkegL7Ka8qWhbi5ZgFJ0VFo8AAuR3aFSuhO5BYDMV5ikEQkOtB2koYAcyCFicIJw5jn+4EBZLftkfCMahMoQSGaoWFt+COGGZ4hqZpNY2Kn6KzZwaFEzyXiHILE8pu6Ft3nCoqeK2mQ9u7MGWc2JoJcY9jTBwf0/kVFnbVZHrVBQ7drTWN/+rNTJsHTRzodMMuWbfi1qZBEygHxjEwnCGsuuAMiPcX4F1qKEMXaz9ECqjJ49Dfbd8WPbP90rV42Eac2SDbJJtUiH7pEpOSY3UCSP35JE8kxfvwXvyXr3379YJbzizTv7I+/wCd9yh0g==</latexit> <latexit sha1_base64="1Y8N28G+CXvJS/CnK7ZQG+jHk1A=">AAACInicbZDLSgMxFIYz3q23qks3B4sgCGUqgpdV0Y0rqWBV6JQhk0nbYJIZkjNiGfoubnwVNy68rQQfxrRW0NYfAh//OYeT80epFBZ9/8ObmJyanpmdmy8sLC4trxRX1y5tkhnG6yyRibmOqOVSaF5HgZJfp4ZTFUl+Fd2c9OtXt9xYkegL7Ka8qWhbi5ZgFJ0VFo8AAuR3aFSuhO5BYDMV5ikEQkOtB2koYAcyCFicIJw5jn+4EBZLftkfCMahMoQSGaoWFt+COGGZ4hqZpNY2Kn6KzZwaFEzyXiHILE8pu6Ft3nCoqeK2mQ9u7MGWc2JoJcY9jTBwf0/kVFnbVZHrVBQ7drTWN/+rNTJsHTRzodMMuWbfi1qZBEygHxjEwnCGsuuAMiPcX4F1qKEMXaz9ECqjJ49Dfbd8WPbP90rV42Eac2SDbJJtUiH7pEpOSY3UCSP35JE8kxfvwXvyXr3379YJbzizTv7I+/wCd9yh0g==</latexit> <latexit sha1_base64="1Y8N28G+CXvJS/CnK7ZQG+jHk1A=">AAACInicbZDLSgMxFIYz3q23qks3B4sgCGUqgpdV0Y0rqWBV6JQhk0nbYJIZkjNiGfoubnwVNy68rQQfxrRW0NYfAh//OYeT80epFBZ9/8ObmJyanpmdmy8sLC4trxRX1y5tkhnG6yyRibmOqOVSaF5HgZJfp4ZTFUl+Fd2c9OtXt9xYkegL7Ka8qWhbi5ZgFJ0VFo8AAuR3aFSuhO5BYDMV5ikEQkOtB2koYAcyCFicIJw5jn+4EBZLftkfCMahMoQSGaoWFt+COGGZ4hqZpNY2Kn6KzZwaFEzyXiHILE8pu6Ft3nCoqeK2mQ9u7MGWc2JoJcY9jTBwf0/kVFnbVZHrVBQ7drTWN/+rNTJsHTRzodMMuWbfi1qZBEygHxjEwnCGsuuAMiPcX4F1qKEMXaz9ECqjJ49Dfbd8WPbP90rV42Eac2SDbJJtUiH7pEpOSY3UCSP35JE8kxfvwXvyXr3379YJbzizTv7I+/wCd9yh0g==</latexit> 49 -
Optimization Model Upscaling ‣ 150 items waiting to be processed
by an operator ‣ CPU usage of an operator is higher than 90% min X p2P pi + u · N + d · N <latexit sha1_base64="1Y8N28G+CXvJS/CnK7ZQG+jHk1A=">AAACInicbZDLSgMxFIYz3q23qks3B4sgCGUqgpdV0Y0rqWBV6JQhk0nbYJIZkjNiGfoubnwVNy68rQQfxrRW0NYfAh//OYeT80epFBZ9/8ObmJyanpmdmy8sLC4trxRX1y5tkhnG6yyRibmOqOVSaF5HgZJfp4ZTFUl+Fd2c9OtXt9xYkegL7Ka8qWhbi5ZgFJ0VFo8AAuR3aFSuhO5BYDMV5ikEQkOtB2koYAcyCFicIJw5jn+4EBZLftkfCMahMoQSGaoWFt+COGGZ4hqZpNY2Kn6KzZwaFEzyXiHILE8pu6Ft3nCoqeK2mQ9u7MGWc2JoJcY9jTBwf0/kVFnbVZHrVBQ7drTWN/+rNTJsHTRzodMMuWbfi1qZBEygHxjEwnCGsuuAMiPcX4F1qKEMXaz9ECqjJ49Dfbd8WPbP90rV42Eac2SDbJJtUiH7pEpOSY3UCSP35JE8kxfvwXvyXr3379YJbzizTv7I+/wCd9yh0g==</latexit> <latexit sha1_base64="1Y8N28G+CXvJS/CnK7ZQG+jHk1A=">AAACInicbZDLSgMxFIYz3q23qks3B4sgCGUqgpdV0Y0rqWBV6JQhk0nbYJIZkjNiGfoubnwVNy68rQQfxrRW0NYfAh//OYeT80epFBZ9/8ObmJyanpmdmy8sLC4trxRX1y5tkhnG6yyRibmOqOVSaF5HgZJfp4ZTFUl+Fd2c9OtXt9xYkegL7Ka8qWhbi5ZgFJ0VFo8AAuR3aFSuhO5BYDMV5ikEQkOtB2koYAcyCFicIJw5jn+4EBZLftkfCMahMoQSGaoWFt+COGGZ4hqZpNY2Kn6KzZwaFEzyXiHILE8pu6Ft3nCoqeK2mQ9u7MGWc2JoJcY9jTBwf0/kVFnbVZHrVBQ7drTWN/+rNTJsHTRzodMMuWbfi1qZBEygHxjEwnCGsuuAMiPcX4F1qKEMXaz9ECqjJ49Dfbd8WPbP90rV42Eac2SDbJJtUiH7pEpOSY3UCSP35JE8kxfvwXvyXr3379YJbzizTv7I+/wCd9yh0g==</latexit> <latexit sha1_base64="1Y8N28G+CXvJS/CnK7ZQG+jHk1A=">AAACInicbZDLSgMxFIYz3q23qks3B4sgCGUqgpdV0Y0rqWBV6JQhk0nbYJIZkjNiGfoubnwVNy68rQQfxrRW0NYfAh//OYeT80epFBZ9/8ObmJyanpmdmy8sLC4trxRX1y5tkhnG6yyRibmOqOVSaF5HgZJfp4ZTFUl+Fd2c9OtXt9xYkegL7Ka8qWhbi5ZgFJ0VFo8AAuR3aFSuhO5BYDMV5ikEQkOtB2koYAcyCFicIJw5jn+4EBZLftkfCMahMoQSGaoWFt+COGGZ4hqZpNY2Kn6KzZwaFEzyXiHILE8pu6Ft3nCoqeK2mQ9u7MGWc2JoJcY9jTBwf0/kVFnbVZHrVBQ7drTWN/+rNTJsHTRzodMMuWbfi1qZBEygHxjEwnCGsuuAMiPcX4F1qKEMXaz9ECqjJ49Dfbd8WPbP90rV42Eac2SDbJJtUiH7pEpOSY3UCSP35JE8kxfvwXvyXr3379YJbzizTv7I+/wCd9yh0g==</latexit> 50 -
Optimization Model min X p2P pi + u · N
+ d · N <latexit sha1_base64="1Y8N28G+CXvJS/CnK7ZQG+jHk1A=">AAACInicbZDLSgMxFIYz3q23qks3B4sgCGUqgpdV0Y0rqWBV6JQhk0nbYJIZkjNiGfoubnwVNy68rQQfxrRW0NYfAh//OYeT80epFBZ9/8ObmJyanpmdmy8sLC4trxRX1y5tkhnG6yyRibmOqOVSaF5HgZJfp4ZTFUl+Fd2c9OtXt9xYkegL7Ka8qWhbi5ZgFJ0VFo8AAuR3aFSuhO5BYDMV5ikEQkOtB2koYAcyCFicIJw5jn+4EBZLftkfCMahMoQSGaoWFt+COGGZ4hqZpNY2Kn6KzZwaFEzyXiHILE8pu6Ft3nCoqeK2mQ9u7MGWc2JoJcY9jTBwf0/kVFnbVZHrVBQ7drTWN/+rNTJsHTRzodMMuWbfi1qZBEygHxjEwnCGsuuAMiPcX4F1qKEMXaz9ECqjJ49Dfbd8WPbP90rV42Eac2SDbJJtUiH7pEpOSY3UCSP35JE8kxfvwXvyXr3379YJbzizTv7I+/wCd9yh0g==</latexit> <latexit sha1_base64="1Y8N28G+CXvJS/CnK7ZQG+jHk1A=">AAACInicbZDLSgMxFIYz3q23qks3B4sgCGUqgpdV0Y0rqWBV6JQhk0nbYJIZkjNiGfoubnwVNy68rQQfxrRW0NYfAh//OYeT80epFBZ9/8ObmJyanpmdmy8sLC4trxRX1y5tkhnG6yyRibmOqOVSaF5HgZJfp4ZTFUl+Fd2c9OtXt9xYkegL7Ka8qWhbi5ZgFJ0VFo8AAuR3aFSuhO5BYDMV5ikEQkOtB2koYAcyCFicIJw5jn+4EBZLftkfCMahMoQSGaoWFt+COGGZ4hqZpNY2Kn6KzZwaFEzyXiHILE8pu6Ft3nCoqeK2mQ9u7MGWc2JoJcY9jTBwf0/kVFnbVZHrVBQ7drTWN/+rNTJsHTRzodMMuWbfi1qZBEygHxjEwnCGsuuAMiPcX4F1qKEMXaz9ECqjJ49Dfbd8WPbP90rV42Eac2SDbJJtUiH7pEpOSY3UCSP35JE8kxfvwXvyXr3379YJbzizTv7I+/wCd9yh0g==</latexit> <latexit sha1_base64="1Y8N28G+CXvJS/CnK7ZQG+jHk1A=">AAACInicbZDLSgMxFIYz3q23qks3B4sgCGUqgpdV0Y0rqWBV6JQhk0nbYJIZkjNiGfoubnwVNy68rQQfxrRW0NYfAh//OYeT80epFBZ9/8ObmJyanpmdmy8sLC4trxRX1y5tkhnG6yyRibmOqOVSaF5HgZJfp4ZTFUl+Fd2c9OtXt9xYkegL7Ka8qWhbi5ZgFJ0VFo8AAuR3aFSuhO5BYDMV5ikEQkOtB2koYAcyCFicIJw5jn+4EBZLftkfCMahMoQSGaoWFt+COGGZ4hqZpNY2Kn6KzZwaFEzyXiHILE8pu6Ft3nCoqeK2mQ9u7MGWc2JoJcY9jTBwf0/kVFnbVZHrVBQ7drTWN/+rNTJsHTRzodMMuWbfi1qZBEygHxjEwnCGsuuAMiPcX4F1qKEMXaz9ECqjJ49Dfbd8WPbP90rV42Eac2SDbJJtUiH7pEpOSY3UCSP35JE8kxfvwXvyXr3379YJbzizTv7I+/wCd9yh0g==</latexit> 51 Downscaling ‣ 5 items waiting to be processed by an operator ‣ CPU usage of an operator is less than 10% -
Evaluation Setup ‣ Each VM hosts one operator instance ‣
BTU for VMs is 60 minutes Distance Speed Average Speed Aggregation Analysis Monitor 52 -
Evaluation Results 0 5 10 15 20 25 0 20
40 60 80 100 120 100 200 300 400 500 600 700 800 900 Number of Operator Instances Incoming Data Items Time in Minutes Under-provisioning Over-provisioning Elastic-provisioning Streaming Data Data arrival pattern is based on day-night cycles of taxi rides within Beijing 53 -
Evaluation Results 0 5 10 15 20 25 0 20
40 60 80 100 120 100 200 300 400 500 600 700 800 900 Number of Operator Instances Incoming Data Items Time in Minutes Under-provisioning Over-provisioning Elastic-provisioning Streaming Data Overall data processing takes the longest due to too little computational resources 54 -
Evaluation Results Data is processed without any delay at high
costs 0 5 10 15 20 25 0 20 40 60 80 100 120 100 200 300 400 500 600 700 800 900 Number of Operator Instances Incoming Data Items Time in Minutes Under-provisioning Over-provisioning Elastic-provisioning Streaming Data 55 -
total makespan is the same as for the over-provisioning and
5% faster compared to under-provisioning Evaluation Results 0 5 10 15 20 25 0 20 40 60 80 100 120 100 200 300 400 500 600 700 800 900 Number of Operator Instances Incoming Data Items Time in Minutes Under-provisioning Over-provisioning Elastic-provisioning Streaming Data 56 -
Observations ‣ Threshold-based provisioning can lead to oscillation ‣ Resources
may not be used up to their full potential ‣ Threshold-based approaches can only react to changing data volumes which leads to delays ‣ VMs are too coarse grained for resource provisioning 57 -
Optimization Model Min <latexit sha1_base64="Iww38d60nYMRtuU88ICQJ2ruV0o=">AAAB/HicbVDLSgNBEJyNrxhf8XHzMhgET2FXBPUW9OJFiOCaQHYJs5NJMmRmdjPTK8Yl+CtePKh49UO8+TdOHgdNLGgoqrrp7ooSwQ247reTW1hcWl7JrxbW1jc2t4rbO3cmTjVlPo1FrOsRMUxwxXzgIFg90YzISLBa1Lsc+bV7pg2P1S0MEhZK0lG8zSkBKzWLewGwB9Ayu+YKB/1+Slp42CyW3LI7Bp4n3pSU0BTVZvEraMU0lUwBFcSYhucmEGZEA6eCDQtBalhCaI90WMNSRSQzYTa+fogPrdLC7VjbUoDH6u+JjEhjBjKynZJA18x6I/E/r5FC+yzMuEpSYIpOFrVTgSHGoyhwi2tGQQwsIVRzeyumXaIJBRtYwYbgzb48T/zj8nnZvTkpVS6maeTRPjpAR8hDp6iCrlAV+YiiR/SMXtGb8+S8OO/Ox6Q150xndtEfOJ8/+J6VGg==</latexit> <latexit sha1_base64="Iww38d60nYMRtuU88ICQJ2ruV0o=">AAAB/HicbVDLSgNBEJyNrxhf8XHzMhgET2FXBPUW9OJFiOCaQHYJs5NJMmRmdjPTK8Yl+CtePKh49UO8+TdOHgdNLGgoqrrp7ooSwQ247reTW1hcWl7JrxbW1jc2t4rbO3cmTjVlPo1FrOsRMUxwxXzgIFg90YzISLBa1Lsc+bV7pg2P1S0MEhZK0lG8zSkBKzWLewGwB9Ayu+YKB/1+Slp42CyW3LI7Bp4n3pSU0BTVZvEraMU0lUwBFcSYhucmEGZEA6eCDQtBalhCaI90WMNSRSQzYTa+fogPrdLC7VjbUoDH6u+JjEhjBjKynZJA18x6I/E/r5FC+yzMuEpSYIpOFrVTgSHGoyhwi2tGQQwsIVRzeyumXaIJBRtYwYbgzb48T/zj8nnZvTkpVS6maeTRPjpAR8hDp6iCrlAV+YiiR/SMXtGb8+S8OO/Ox6Q150xndtEfOJ8/+J6VGg==</latexit> <latexit sha1_base64="Iww38d60nYMRtuU88ICQJ2ruV0o=">AAAB/HicbVDLSgNBEJyNrxhf8XHzMhgET2FXBPUW9OJFiOCaQHYJs5NJMmRmdjPTK8Yl+CtePKh49UO8+TdOHgdNLGgoqrrp7ooSwQ247reTW1hcWl7JrxbW1jc2t4rbO3cmTjVlPo1FrOsRMUxwxXzgIFg90YzISLBa1Lsc+bV7pg2P1S0MEhZK0lG8zSkBKzWLewGwB9Ayu+YKB/1+Slp42CyW3LI7Bp4n3pSU0BTVZvEraMU0lUwBFcSYhucmEGZEA6eCDQtBalhCaI90WMNSRSQzYTa+fogPrdLC7VjbUoDH6u+JjEhjBjKynZJA18x6I/E/r5FC+yzMuEpSYIpOFrVTgSHGoyhwi2tGQQwsIVRzeyumXaIJBRtYwYbgzb48T/zj8nnZvTkpVS6maeTRPjpAR8hDp6iCrlAV+YiiR/SMXtGb8+S8OO/Ox6Q150xndtEfOJ8/+J6VGg==</latexit> h#
· hcost <latexit sha1_base64="Ojz3NR4aHMmkixGq35HRXv1L4zw=">AAAB/HicbVDLSsNAFJ34rPUVHzs3g0FwVRIR1F3RjcsKxhaaWCbTSTN0kgkzN0INxV9x40LFrR/izr9x+lho64ELh3Pu5d57olxwDa77bS0sLi2vrFbWqusbm1vb9s7unZaFosynUkjViohmgmfMBw6CtXLFSBoJ1oz6VyO/+cCU5jK7hUHOwpT0Mh5zSsBIHXs/uQ+cAAe0KwEnnZJKDcOO7bg1dww8T7wpcdAUjY79FXQlLVKWARVE67bn5hCWRAGngg2rQaFZTmif9Fjb0IykTIfl+PohPjJKF8dSmcoAj9XfEyVJtR6kkelMCSR61huJ/3ntAuLzsORZXgDL6GRRXAgMEo+iwF2uGAUxMIRQxc2tmCZEEQomsKoJwZt9eZ74J7WLmntz6tQvp2lU0AE6RMfIQ2eojq5RA/mIokf0jF7Rm/VkvVjv1sekdcGazuyhP7A+fwA7KpSl</latexit> <latexit sha1_base64="Ojz3NR4aHMmkixGq35HRXv1L4zw=">AAAB/HicbVDLSsNAFJ34rPUVHzs3g0FwVRIR1F3RjcsKxhaaWCbTSTN0kgkzN0INxV9x40LFrR/izr9x+lho64ELh3Pu5d57olxwDa77bS0sLi2vrFbWqusbm1vb9s7unZaFosynUkjViohmgmfMBw6CtXLFSBoJ1oz6VyO/+cCU5jK7hUHOwpT0Mh5zSsBIHXs/uQ+cAAe0KwEnnZJKDcOO7bg1dww8T7wpcdAUjY79FXQlLVKWARVE67bn5hCWRAGngg2rQaFZTmif9Fjb0IykTIfl+PohPjJKF8dSmcoAj9XfEyVJtR6kkelMCSR61huJ/3ntAuLzsORZXgDL6GRRXAgMEo+iwF2uGAUxMIRQxc2tmCZEEQomsKoJwZt9eZ74J7WLmntz6tQvp2lU0AE6RMfIQ2eojq5RA/mIokf0jF7Rm/VkvVjv1sekdcGazuyhP7A+fwA7KpSl</latexit> <latexit sha1_base64="Ojz3NR4aHMmkixGq35HRXv1L4zw=">AAAB/HicbVDLSsNAFJ34rPUVHzs3g0FwVRIR1F3RjcsKxhaaWCbTSTN0kgkzN0INxV9x40LFrR/izr9x+lho64ELh3Pu5d57olxwDa77bS0sLi2vrFbWqusbm1vb9s7unZaFosynUkjViohmgmfMBw6CtXLFSBoJ1oz6VyO/+cCU5jK7hUHOwpT0Mh5zSsBIHXs/uQ+cAAe0KwEnnZJKDcOO7bg1dww8T7wpcdAUjY79FXQlLVKWARVE67bn5hCWRAGngg2rQaFZTmif9Fjb0IykTIfl+PohPjJKF8dSmcoAj9XfEyVJtR6kkelMCSR61huJ/3ntAuLzsORZXgDL6GRRXAgMEo+iwF2uGAUxMIRQxc2tmCZEEQomsKoJwZt9eZ74J7WLmntz6tQvp2lU0AE6RMfIQ2eojq5RA/mIokf0jF7Rm/VkvVjv1sekdcGazuyhP7A+fwA7KpSl</latexit> + P h2H h cpu P i2I\itype=o o cpu P h2H h cpu <latexit sha1_base64="FC2LMoTMydIHirzI5Lws8bpZXkI=">AAACUXicdVFBSyMxFH4dXe3W1a3ucS8Py4IglqksqAeh6KV764K1QqcMmTRjg5lkSDJCCfMbBfHiD/Gyh91N2zms1n0Q+Pi+772XfElywY0Nw+dasLb+YWOz/rGx9Wl753Nzd+/aqEJTNqBKKH2TEMMEl2xguRXsJteMZIlgw+Tucq4P75k2XMkrO8vZOCO3kqecEuupuMkPMZqkmlAXmSKL3RQjLrFXopvGjuZFWeIRLiW+kH5gREmOPHbzcSWeo/JmVZnL/44p42YrbIeLwlXQqUALqurHzcdoomiRMWmpIMaMOmFux45oy6lgZSMqDMsJvSO3bOShJBkzY7eIpMRvnplgqrQ/0uKC/bfDkcyYWZZ4Z0bs1LzV5uR72qiw6enYcZkXlkm6XJQWAq3CeSA44ZpRK2YeEKq5vyvSKfEBW/8LDR9C5+2TV8HguH3WDn9+b3UvqjTq8BX24QA6cAJd6EEfBkDhAV7gN/ypPdV+BRAES2tQq3q+wKsKtv4C+FK0xA==</latexit> <latexit sha1_base64="FC2LMoTMydIHirzI5Lws8bpZXkI=">AAACUXicdVFBSyMxFH4dXe3W1a3ucS8Py4IglqksqAeh6KV764K1QqcMmTRjg5lkSDJCCfMbBfHiD/Gyh91N2zms1n0Q+Pi+772XfElywY0Nw+dasLb+YWOz/rGx9Wl753Nzd+/aqEJTNqBKKH2TEMMEl2xguRXsJteMZIlgw+Tucq4P75k2XMkrO8vZOCO3kqecEuupuMkPMZqkmlAXmSKL3RQjLrFXopvGjuZFWeIRLiW+kH5gREmOPHbzcSWeo/JmVZnL/44p42YrbIeLwlXQqUALqurHzcdoomiRMWmpIMaMOmFux45oy6lgZSMqDMsJvSO3bOShJBkzY7eIpMRvnplgqrQ/0uKC/bfDkcyYWZZ4Z0bs1LzV5uR72qiw6enYcZkXlkm6XJQWAq3CeSA44ZpRK2YeEKq5vyvSKfEBW/8LDR9C5+2TV8HguH3WDn9+b3UvqjTq8BX24QA6cAJd6EEfBkDhAV7gN/ypPdV+BRAES2tQq3q+wKsKtv4C+FK0xA==</latexit> <latexit sha1_base64="FC2LMoTMydIHirzI5Lws8bpZXkI=">AAACUXicdVFBSyMxFH4dXe3W1a3ucS8Py4IglqksqAeh6KV764K1QqcMmTRjg5lkSDJCCfMbBfHiD/Gyh91N2zms1n0Q+Pi+772XfElywY0Nw+dasLb+YWOz/rGx9Wl753Nzd+/aqEJTNqBKKH2TEMMEl2xguRXsJteMZIlgw+Tucq4P75k2XMkrO8vZOCO3kqecEuupuMkPMZqkmlAXmSKL3RQjLrFXopvGjuZFWeIRLiW+kH5gREmOPHbzcSWeo/JmVZnL/44p42YrbIeLwlXQqUALqurHzcdoomiRMWmpIMaMOmFux45oy6lgZSMqDMsJvSO3bOShJBkzY7eIpMRvnplgqrQ/0uKC/bfDkcyYWZZ4Z0bs1LzV5uR72qiw6enYcZkXlkm6XJQWAq3CeSA44ZpRK2YeEKq5vyvSKfEBW/8LDR9C5+2TV8HguH3WDn9+b3UvqjTq8BX24QA6cAJd6EEfBkDhAV7gN/ypPdV+BRAES2tQq3q+wKsKtv4C+FK0xA==</latexit> + P h2H h memory P i2I\itype=o o memory P h2H h memory <latexit sha1_base64="dQikx1vGD9GpzBE8abB6+iA0g7o=">AAACWnicfVHLSgMxFM2Mr7a+6mPn5mIRBLFMRVAXguim7hSsCp0yZNKMDeYxJBmhhPlJV7rwV8T0AT7xQuBwzrn3JidpzpmxUfQahDOzc/MLlWptcWl5ZbW+tn5rVKEJ7RDFlb5PsaGcSdqxzHJ6n2uKRcrpXfp4MdLvnqg2TMkbO8xpT+AHyTJGsPVUUtd7EPczjYmLTSESN4CYSWiX4AaJE1QoPSxL2IeJysbqJcQE58ASN5pYwiko71ef/vK/YWVSb0TNaFzwG7SmoIGmdZXUn+O+IoWg0hKOjem2otz2HNaWEU7LWlwYmmPyiB9o10OJBTU9N86mhB3P9CFT2h9pYcx+7XBYGDMUqXcKbAfmpzYi/9K6hc2Oe47JvLBUksmirOBgFYxigT7TlFg+9AATzfxdgQywT9r676j5EFo/n/wbdA6aJ83o+rBxdj5No4K20DbaRS10hM5QG12hDiLoBb0HC0EleAtnwmq4OLGGwbRnA32rcPMDTjG2FA==</latexit> <latexit sha1_base64="dQikx1vGD9GpzBE8abB6+iA0g7o=">AAACWnicfVHLSgMxFM2Mr7a+6mPn5mIRBLFMRVAXguim7hSsCp0yZNKMDeYxJBmhhPlJV7rwV8T0AT7xQuBwzrn3JidpzpmxUfQahDOzc/MLlWptcWl5ZbW+tn5rVKEJ7RDFlb5PsaGcSdqxzHJ6n2uKRcrpXfp4MdLvnqg2TMkbO8xpT+AHyTJGsPVUUtd7EPczjYmLTSESN4CYSWiX4AaJE1QoPSxL2IeJysbqJcQE58ASN5pYwiko71ef/vK/YWVSb0TNaFzwG7SmoIGmdZXUn+O+IoWg0hKOjem2otz2HNaWEU7LWlwYmmPyiB9o10OJBTU9N86mhB3P9CFT2h9pYcx+7XBYGDMUqXcKbAfmpzYi/9K6hc2Oe47JvLBUksmirOBgFYxigT7TlFg+9AATzfxdgQywT9r676j5EFo/n/wbdA6aJ83o+rBxdj5No4K20DbaRS10hM5QG12hDiLoBb0HC0EleAtnwmq4OLGGwbRnA32rcPMDTjG2FA==</latexit> <latexit sha1_base64="dQikx1vGD9GpzBE8abB6+iA0g7o=">AAACWnicfVHLSgMxFM2Mr7a+6mPn5mIRBLFMRVAXguim7hSsCp0yZNKMDeYxJBmhhPlJV7rwV8T0AT7xQuBwzrn3JidpzpmxUfQahDOzc/MLlWptcWl5ZbW+tn5rVKEJ7RDFlb5PsaGcSdqxzHJ6n2uKRcrpXfp4MdLvnqg2TMkbO8xpT+AHyTJGsPVUUtd7EPczjYmLTSESN4CYSWiX4AaJE1QoPSxL2IeJysbqJcQE58ASN5pYwiko71ef/vK/YWVSb0TNaFzwG7SmoIGmdZXUn+O+IoWg0hKOjem2otz2HNaWEU7LWlwYmmPyiB9o10OJBTU9N86mhB3P9CFT2h9pYcx+7XBYGDMUqXcKbAfmpzYi/9K6hc2Oe47JvLBUksmirOBgFYxigT7TlFg+9AATzfxdgQywT9r676j5EFo/n/wbdA6aJ83o+rBxdj5No4K20DbaRS10hM5QG12hDiLoBb0HC0EleAtnwmq4OLGGwbRnA32rcPMDTjG2FA==</latexit> + X v2V v · P <latexit sha1_base64="rWldUUTRw7lvvczrI32Ii0/gU20=">AAACAnicbVDLSgMxFM34rPVVdaebYBEEocyIoO6KblxWcNpCZyiZTKYNzWNIMoUyFNz4K25cqLj1K9z5N6btLLT1QOBwzrnc3BOljGrjut/O0vLK6tp6aaO8ubW9s1vZ229qmSlMfCyZVO0IacKoIL6hhpF2qgjiESOtaHA78VtDojSV4sGMUhJy1BM0oRgZK3Urh2cw0Bnv5kMYUAGbY2gJjqWBjW6l6tbcKeAi8QpSBQVs/iuIJc44EQYzpHXHc1MT5kgZihkZl4NMkxThAeqRjqUCcaLDfHrDGJ5YJYaJVPYJA6fq74kcca1HPLJJjkxfz3sT8T+vk5nkKsypSDNDBJ4tSjIGjYSTQmBMFcGGjSxBWFH7V4j7SCFsbG1lW4I3f/Ii8c9r1zX3/qJavynaKIEjcAxOgQcuQR3cgQbwAQaP4Bm8gjfnyXlx3p2PWXTJKWYOwB84nz9aA5ZA</latexit> <latexit sha1_base64="rWldUUTRw7lvvczrI32Ii0/gU20=">AAACAnicbVDLSgMxFM34rPVVdaebYBEEocyIoO6KblxWcNpCZyiZTKYNzWNIMoUyFNz4K25cqLj1K9z5N6btLLT1QOBwzrnc3BOljGrjut/O0vLK6tp6aaO8ubW9s1vZ229qmSlMfCyZVO0IacKoIL6hhpF2qgjiESOtaHA78VtDojSV4sGMUhJy1BM0oRgZK3Urh2cw0Bnv5kMYUAGbY2gJjqWBjW6l6tbcKeAi8QpSBQVs/iuIJc44EQYzpHXHc1MT5kgZihkZl4NMkxThAeqRjqUCcaLDfHrDGJ5YJYaJVPYJA6fq74kcca1HPLJJjkxfz3sT8T+vk5nkKsypSDNDBJ4tSjIGjYSTQmBMFcGGjSxBWFH7V4j7SCFsbG1lW4I3f/Ii8c9r1zX3/qJavynaKIEjcAxOgQcuQR3cgQbwAQaP4Bm8gjfnyXlx3p2PWXTJKWYOwB84nz9aA5ZA</latexit> <latexit sha1_base64="rWldUUTRw7lvvczrI32Ii0/gU20=">AAACAnicbVDLSgMxFM34rPVVdaebYBEEocyIoO6KblxWcNpCZyiZTKYNzWNIMoUyFNz4K25cqLj1K9z5N6btLLT1QOBwzrnc3BOljGrjut/O0vLK6tp6aaO8ubW9s1vZ229qmSlMfCyZVO0IacKoIL6hhpF2qgjiESOtaHA78VtDojSV4sGMUhJy1BM0oRgZK3Urh2cw0Bnv5kMYUAGbY2gJjqWBjW6l6tbcKeAi8QpSBQVs/iuIJc44EQYzpHXHc1MT5kgZihkZl4NMkxThAeqRjqUCcaLDfHrDGJ5YJYaJVPYJA6fq74kcca1HPLJJjkxfz3sT8T+vk5nkKsypSDNDBJ4tSjIGjYSTQmBMFcGGjSxBWFH7V4j7SCFsbG1lW4I3f/Ii8c9r1zX3/qJavynaKIEjcAxOgQcuQR3cgQbwAQaP4Bm8gjfnyXlx3p2PWXTJKWYOwB84nz9aA5ZA</latexit> 58 -
BTU-based Approach ‣ Scale up, when operators require more resources
based on thresholds ‣ Only scale down at the end of a BTU to keep already paid processing capabilities for future data spikes ‣ Use Docker container to employ fine grained resource provisioning 59 -
Downscaling Procedure BTU of VM ends Create a down-scaling plan
Release operators Migrate operators Release VM Lease VM for another BTU 60 -
Evaluation Setup ‣ Threshold-based approach serves as the baseline ‣
BTUs: 10 min, 30 min, 60 min Spain United Kindom Germany O1 O2 O8 O9 O3 O4 O5 O6 O7 Sweden O1 O2 O1 O2 O3 O4 O5 O6 O7 O3 O4 O5 O6 O7 O1 O2 S1 S2 S3 S1 S2 S3 S1 S2 S3 S1 S2 S3 61 -
Data Arrival Pattern 0 20 40 60 80 100 120
0 2 4 6 8 Stepwise Pattern Time (Minutes) Manufacturing Machines 0 20 40 60 80 100 120 0 2 4 6 8 2−Level Pattern Time (Minutes) Manufacturing Machines 0 20 40 60 80 100 120 0 2 4 6 8 Random Walk Pattern 1 Time (Minutes) Manufacturing Machines 0 20 40 60 80 100 120 0 2 4 6 8 Random Walk Pattern 2 Time (Minutes) Manufacturing Machines 62 -
Evaluation Results - Threshold-based Baseline 0 20 40 60 80
100 120 0 2 4 6 8 Time (Minutes) Amount of Manufacturing Machines and VMs 0 10 20 30 40 50 60 70 Amount of Operator Instances 0 0 Pattern VMs Operator Instances 63 -
Evaluation Results - BTU 30 0 0 Pattern VMs Operator
Instances 0 20 40 60 80 100 120 0 2 4 6 8 Time (Minutes) Amount of Manufacturing Machines and VMs 0 10 20 30 40 50 60 70 Amount of Operator Instances 64 -
Evaluation Metrics Compliance Metrics ‣ Real-time: average data processing time
without any load ‣ Near real-time: 2 x the realtime SLO ‣ Relaxed: 5 x the realtime SLO Cost ‣ Resource cost: cost for leasing a VM for 10 min amounts for 1 cost unit ‣ Penalty cost: delaying 10000 data items amounts for 1 cost unit ‣ Total cost = Resource cost + penalty cost 65 -
Evaluation Results - Compliance Threshold- based BTU 10 BTU 30
BTU 60 Real-time Compliance 40 % 48 % 50 % 55 % Near real- time Compliance 68 % 84 % 88 % 93 % Relaxed Compliance 72 % 88 % 91 % 95 % on the results for the other —> mention here the results m walk 1 and rw 2 66 -
Threshold-based BTU-based BTU 10 BTU 30 BTU 60 BTU 10
BTU 30 BTU 60 Resource Cost 66 86 122 82 96 104 Real-time Compliance 157 178 214 169 178 176 Near real-time Compliance 115 135 171 108 155 115 Relaxed Compliance 108 128 165 102 111 112 Evaluation Results - Cost 67 -
Contribution Summary ‣ Introduced, implemented and evaluated a threshold-based provisioning
approach ‣ Analyzed the limitations of threshold-based provisioning approaches ‣ Proposed a lazy downscaling approach to improve the SLA compliance, reduce the management overhead and achieve a cost reduction of up to 36% 68 -
Research Questions Revisited RQ1 How can geographically distributed data streams
be processed efficiently? C1 A system design for a holistic stream processing ecosystem in distributed environments 70 -
Research Questions Revisited RQ2 How can SPAs be described to
allow a distributed deployment model and respect service level objectives? C2 An approach to define quality of service- and location-specific aspects for SPAs 71 -
Research Questions Revisited C3 Cost-efficient resource provisioning approaches for SPAs
RQ3 How can SPAs be optimally executed on computational resources? 72 -
Future Work ‣Optimal deployment of operators across different geographic locations
‣Development of suitable monitoring metrics ‣Complex failure compensation strategies for SPAs ‣Data privacy and data ownership 73 -
Final Notes Publications ‣Journal 4 / 7 (FGCS, IC, PeerJ,
TSC … ) ‣Conference, Workshops 9 / 18 (CLOUD, EDOC, SOSE, UCC …) ‣Technical Report, Proceedings 1 / 3 Teaching ‣Advanced Internet Computing ‣Co-Supervision: 2 bachelor theses, 3 master theses, 2 projects in SE & IC 74 -
Acknowledgements ‣Prof. Dr. Schahram Dustdar and Assistant. Prof. Dr.-Ing. Stefan
Schulte for guiding me through my PhD ‣Prof. Dr. Harald Gall and Prof. Dr. Gertrude Kappel for serving as reviewers ‣All current and previous members of the DSG for insightful discussions FP7 SIMPLICITY (318201) H2020 CREMA (637066) Innovative Project 75 -
References [1] Akpınar, K., Hua, K. A., & Li, K.
ThingStore: a platform for internet-of-things application development and deployment. DEBS 2015. [2] Cardellini, V., Grassi, V., Lo Presti, F., & Nardelli, M. Distributed QoS-aware scheduling in Storm. DEBS 2015. [3] Gulisano, V., Jimenez-Peris, R., Patino-Martinez, M., Soriente, C., & Valduriez, P. Streamcloud: An elastic and scalable data streaming system. Transactions on Parallel and Distributed Systems, 2012 [4] Lim, H., Han, Y., & Babu, S. How to Fit when No One Size Fits. CIDR, 2013 [5] Schneider, S., Andrade, H., Gedik, B., Biem, A., & Wu, K. L. Elastic scaling of data parallel operators in stream processing. IPDPS 2009. [6] Gedik, B., Schneider, S., Hirzel, M., & Wu, K. L. Elastic scaling for data stream processing. Transactions on Parallel and Distributed Systems, 2014 [7] Aniello, L., Baldoni, R., & Querzoni, L. Adaptive online scheduling in storm. DEBS 2013 [8] Xu, J., Chen, Z., Tang, J., & Su, S. T-storm: Traffic-aware online scheduling in storm. ICDCS 2014 [9] Xu, L., Peng, B., & Gupta, I. Stela: Enabling stream processing systems to scale-in and scale-out on-demand. IC2E 2016 [10] Heinze, T., Roediger, L., Meister, A., Ji, Y., Jerzak, Z., & Fetzer, C. Online parameter optimization for elastic data stream processing. SCC 2015 [11] Castro Fernandez, R., Migliavacca, M., Kalyvianaki, E., & Pietzuch, P. Integrating scale out and fault tolerance in stream processing using operator state management. SIGMOD 2013 77 -
Methodology Based on the design science methodology ‣Identify a challenge
‣Conduct a literature survey ‣Define a research hypotheses and performance indicators ‣Design and implementation of a novel algorithm or system design ‣Evaluation of the research hypotheses based on the research prototype 79 -
System Design Infrastructure Data Sources VISP Marketplace Cloud Resources Edge
Resources VISP Runtime Stream Processing Application Topology Configuration Operator Replication Infrastructure Management Operator Provisioning Messaging Operator Migration Operator Placement Shared Data Failure Detection Privacy Security Topology Enactment Manufacturing Machines Wearable Devices News Portals Failure Compensation Monitoring Monetization Operator Distribution Topology Builder Operator Registry Runtime Synchronization Stock Market SLA Compliance Operational Layer Communication Layer Infrastructure Layer Cross-cutting Concerns 81 -
Distributed Deployment of VISP Runtimes VISP Runtime Data Source 1
Data Source N • • • VISP Runtime VISP Runtime • • • Topology Configuration Operator Replication Infrastructure Management Monitoring Messaging Computational Resources Virtual Machine 1 Operator 1 Operator N • • • Virtual Machine N Operator 1 Operator N • • • • • • Shared Data Operator Image 1 Operator Image N • • • Operator Repository VISP Runtime VISP Runtime • • • R R 83 -
Features of the VTDL Attribute Values Default Value Description type
string — Operator logic allowedLocations IP+ — Possible locations poolPreferences string+ general Hardware preferences concreteLocation IP/poolIdentifier* — Preselected location inputFormat string+ — Accepted inputs outputFormat string — Data output format responseTime numerical 2s Response time queueLength numerical 100 Buffered items stateful { true | false } true Operator statefulness replicationAllowed { true | false } false Operator replication protocol {stream | microbatch: <X>items/<X>ms} stream Processing mode compensation { redeploySingle | redeployTopology | deploy<URL} | mailto:<email> | none } none Failure recovery mode 85 -
Topology Update Procedure (S13) Continue processing (S10) Update message infrastructure
(S7) Distribute update operations (S6) Initiate update Processing in place / No topology in place No compensation required for failure Processing stopped Compensation required for failure Managing VISP Runtime All involved VISP Runtimes All involved VISP Runtimes synchronized by managing VISP Runtime (S1) Receive new topology (S2) Check integrity for new topology (S3) Topology grounding (S4) Distribute new topology (S5) Check that no update is in place (S8) Validate resource availability (S9) Stop processing of involved operators (S11) Apply operator updates (S12) Check integrity 86 -
Evaluation VTDL Approach Baseline Approach duration (s) interactions duration (s)
interactions 1. New topology in one location 54.98 3 54.98 3 2. New topology across four locations 40.95 3 68.10 12 3. Network disruption 7.53 0 — — 4. Load reduction 3.30 3 54.05 8 5. Bugfix for an operator 5.69 3 35.62 4 87 -
Deployment of SPAs O5 O4 O3 O2 O1 O6 O9
O7 O8 O5 O4 O3 O2 O1 O6 O9 O7 O8 O2 O1 O6 O5 O4 O3 O2 O1 O6 O9 O7 O8 O2 O1 O6 O1 O1 O3 O5 O4 O6 O9 O7 O8 O2 O1 O6 O1 O2 O1 O3 O5 O4 H1 H2 H1 H2 H3 H1 H2 H3 H4 H1 H2 H3 H4 Stage 1 Stage 2 Stage 3 Stage 4 Data Volume over Time Amount of Manufacturing Machines 89 -
Elastic Resource Provisioning Model 1 min X p2P pi +
X p2P piBT U + u · N + d · N <latexit sha1_base64="gPDVtNvhSDVp6sXC9eN/+3LxsS0=">AAACPnicbZBNSwMxEIazflu/qh69DBZBEMpWBPVW9OJJKrgqdMuSzaYaTLJLMiuWZf+ZF/+CN69ePKh49Whaq/g1EHh43xkm88aZFBZ9/94bGR0bn5icmq7MzM7NL1QXl05smhvGA5bK1JzF1HIpNA9QoORnmeFUxZKfxpf7ff/0ihsrUn2MvYx3FD3XoisYRSdF1ROAEPk1GlUooUsIba6iIoNQaGiVkEUCNj7FL60QUbF3HJSl83IIWZIiHDpOPrkSVWt+3R8U/IXGEGpkWK2oehcmKcsV18gktbbd8DPsFNSgYJKXlTC3PKPskp7ztkNNFbedYnB/CWtOSaCbGvc0wkD9PlFQZW1Pxa5TUbywv72++J/XzrG70ymEznLkmn0s6uYSMIV+mJAIwxnKngPKjHB/BXZBDWXoIu+H0Ph98l8INuu7df9oq9bcG6YxRVbIKlknDbJNmuSAtEhAGLkhD+SJPHu33qP34r1+tI54w5ll8qO8t3eoMa0C</latexit> <latexit sha1_base64="gPDVtNvhSDVp6sXC9eN/+3LxsS0=">AAACPnicbZBNSwMxEIazflu/qh69DBZBEMpWBPVW9OJJKrgqdMuSzaYaTLJLMiuWZf+ZF/+CN69ePKh49Whaq/g1EHh43xkm88aZFBZ9/94bGR0bn5icmq7MzM7NL1QXl05smhvGA5bK1JzF1HIpNA9QoORnmeFUxZKfxpf7ff/0ihsrUn2MvYx3FD3XoisYRSdF1ROAEPk1GlUooUsIba6iIoNQaGiVkEUCNj7FL60QUbF3HJSl83IIWZIiHDpOPrkSVWt+3R8U/IXGEGpkWK2oehcmKcsV18gktbbd8DPsFNSgYJKXlTC3PKPskp7ztkNNFbedYnB/CWtOSaCbGvc0wkD9PlFQZW1Pxa5TUbywv72++J/XzrG70ymEznLkmn0s6uYSMIV+mJAIwxnKngPKjHB/BXZBDWXoIu+H0Ph98l8INuu7df9oq9bcG6YxRVbIKlknDbJNmuSAtEhAGLkhD+SJPHu33qP34r1+tI54w5ll8qO8t3eoMa0C</latexit> <latexit sha1_base64="gPDVtNvhSDVp6sXC9eN/+3LxsS0=">AAACPnicbZBNSwMxEIazflu/qh69DBZBEMpWBPVW9OJJKrgqdMuSzaYaTLJLMiuWZf+ZF/+CN69ePKh49Whaq/g1EHh43xkm88aZFBZ9/94bGR0bn5icmq7MzM7NL1QXl05smhvGA5bK1JzF1HIpNA9QoORnmeFUxZKfxpf7ff/0ihsrUn2MvYx3FD3XoisYRSdF1ROAEPk1GlUooUsIba6iIoNQaGiVkEUCNj7FL60QUbF3HJSl83IIWZIiHDpOPrkSVWt+3R8U/IXGEGpkWK2oehcmKcsV18gktbbd8DPsFNSgYJKXlTC3PKPskp7ztkNNFbedYnB/CWtOSaCbGvc0wkD9PlFQZW1Pxa5TUbywv72++J/XzrG70ymEznLkmn0s6uYSMIV+mJAIwxnKngPKjHB/BXZBDWXoIu+H0Ph98l8INuu7df9oq9bcG6YxRVbIKlknDbJNmuSAtEhAGLkhD+SJPHu33qP34r1+tI54w5ll8qO8t3eoMa0C</latexit> pi <latexit sha1_base64="p6RESMUprFwWDqzMCniPeYau4Sw=">AAAB7HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqN6KXjxWMG2hDWWz3bRrN7thdyOU0P/gxYOKV3+QN/+NmzYHbX0w8Hhvhpl5YcKZNq777ZTW1jc2t8rblZ3dvf2D6uFRW8tUEeoTyaXqhlhTzgT1DTOcdhNFcRxy2gknt7nfeaJKMykezDShQYxHgkWMYGOlNkoGDFUG1Zpbd+dAq8QrSA0KtAbVr/5QkjSmwhCOte55bmKCDCvDCKezSj/VNMFkgke0Z6nAMdVBNr92hs6sMkSRVLaEQXP190SGY62ncWg7Y2zGetnLxf+8XmqiqyBjIkkNFWSxKEo5MhLlr6MhU5QYPrUEE8XsrYiMscLE2IDyELzll1eJf1G/rrv3l7XmTZFGGU7gFM7BgwY04Q5a4AOBR3iGV3hzpPPivDsfi9aSU8wcwx84nz+je44L</latexit> <latexit sha1_base64="p6RESMUprFwWDqzMCniPeYau4Sw=">AAAB7HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqN6KXjxWMG2hDWWz3bRrN7thdyOU0P/gxYOKV3+QN/+NmzYHbX0w8Hhvhpl5YcKZNq777ZTW1jc2t8rblZ3dvf2D6uFRW8tUEeoTyaXqhlhTzgT1DTOcdhNFcRxy2gknt7nfeaJKMykezDShQYxHgkWMYGOlNkoGDFUG1Zpbd+dAq8QrSA0KtAbVr/5QkjSmwhCOte55bmKCDCvDCKezSj/VNMFkgke0Z6nAMdVBNr92hs6sMkSRVLaEQXP190SGY62ncWg7Y2zGetnLxf+8XmqiqyBjIkkNFWSxKEo5MhLlr6MhU5QYPrUEE8XsrYiMscLE2IDyELzll1eJf1G/rrv3l7XmTZFGGU7gFM7BgwY04Q5a4AOBR3iGV3hzpPPivDsfi9aSU8wcwx84nz+je44L</latexit> <latexit sha1_base64="p6RESMUprFwWDqzMCniPeYau4Sw=">AAAB7HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqN6KXjxWMG2hDWWz3bRrN7thdyOU0P/gxYOKV3+QN/+NmzYHbX0w8Hhvhpl5YcKZNq777ZTW1jc2t8rblZ3dvf2D6uFRW8tUEeoTyaXqhlhTzgT1DTOcdhNFcRxy2gknt7nfeaJKMykezDShQYxHgkWMYGOlNkoGDFUG1Zpbd+dAq8QrSA0KtAbVr/5QkjSmwhCOte55bmKCDCvDCKezSj/VNMFkgke0Z6nAMdVBNr92hs6sMkSRVLaEQXP190SGY62ncWg7Y2zGetnLxf+8XmqiqyBjIkkNFWSxKEo5MhLlr6MhU5QYPrUEE8XsrYiMscLE2IDyELzll1eJf1G/rrv3l7XmTZFGGU7gFM7BgwY04Q5a4AOBR3iGV3hzpPPivDsfi9aSU8wcwx84nz+je44L</latexit> piBT U <latexit sha1_base64="FmGu3g7XaE19MK2k4P833H+fNoM=">AAAB8XicbVBNS8NAEN3Ur1q/qh69LBbBU0lEUG+lXjxWaGyhDWGz3bRLN7thdyKUkJ/hxYOKV/+NN/+N2zYHrT4YeLw3w8y8KBXcgOt+OZW19Y3Nrep2bWd3b/+gfnj0YFSmKfOpEkr3I2KY4JL5wEGwfqoZSSLBetH0du73Hpk2XMkuzFIWJGQsecwpASsN0jDnYd7u+kUR1htu010A/yVeSRqoRCesfw5HimYJk0AFMWbguSkEOdHAqWBFbZgZlhI6JWM2sFSShJkgX5xc4DOrjHCstC0JeKH+nMhJYswsiWxnQmBiVr25+J83yCC+DnIu0wyYpMtFcSYwKDz/H4+4ZhTEzBJCNbe3YjohmlCwKdVsCN7qy3+Jf9G8abr3l41Wu0yjik7QKTpHHrpCLXSHOshHFCn0hF7QqwPOs/PmvC9bK045c4x+wfn4Bt8WkS0=</latexit> <latexit sha1_base64="FmGu3g7XaE19MK2k4P833H+fNoM=">AAAB8XicbVBNS8NAEN3Ur1q/qh69LBbBU0lEUG+lXjxWaGyhDWGz3bRLN7thdyKUkJ/hxYOKV/+NN/+N2zYHrT4YeLw3w8y8KBXcgOt+OZW19Y3Nrep2bWd3b/+gfnj0YFSmKfOpEkr3I2KY4JL5wEGwfqoZSSLBetH0du73Hpk2XMkuzFIWJGQsecwpASsN0jDnYd7u+kUR1htu010A/yVeSRqoRCesfw5HimYJk0AFMWbguSkEOdHAqWBFbZgZlhI6JWM2sFSShJkgX5xc4DOrjHCstC0JeKH+nMhJYswsiWxnQmBiVr25+J83yCC+DnIu0wyYpMtFcSYwKDz/H4+4ZhTEzBJCNbe3YjohmlCwKdVsCN7qy3+Jf9G8abr3l41Wu0yjik7QKTpHHrpCLXSHOshHFCn0hF7QqwPOs/PmvC9bK045c4x+wfn4Bt8WkS0=</latexit> <latexit sha1_base64="FmGu3g7XaE19MK2k4P833H+fNoM=">AAAB8XicbVBNS8NAEN3Ur1q/qh69LBbBU0lEUG+lXjxWaGyhDWGz3bRLN7thdyKUkJ/hxYOKV/+NN/+N2zYHrT4YeLw3w8y8KBXcgOt+OZW19Y3Nrep2bWd3b/+gfnj0YFSmKfOpEkr3I2KY4JL5wEGwfqoZSSLBetH0du73Hpk2XMkuzFIWJGQsecwpASsN0jDnYd7u+kUR1htu010A/yVeSRqoRCesfw5HimYJk0AFMWbguSkEOdHAqWBFbZgZlhI6JWM2sFSShJkgX5xc4DOrjHCstC0JeKH+nMhJYswsiWxnQmBiVr25+J83yCC+DnIu0wyYpMtFcSYwKDz/H4+4ZhTEzBJCNbe3YjohmlCwKdVsCN7qy3+Jf9G8abr3l41Wu0yjik7QKTpHHrpCLXSHOshHFCn0hF7QqwPOs/PmvC9bK045c4x+wfn4Bt8WkS0=</latexit> u <latexit sha1_base64="kF4y8yL2rgRNwd6KnxzHbkHreI8=">AAAB53icbVBNS8NAEJ34WetX1aOXxSJ4KokI6q3oxWMLxhbaUDbbSbt2swm7G6GE/gIvHlS8+pe8+W/ctjlo64OBx3szzMwLU8G1cd1vZ2V1bX1js7RV3t7Z3duvHBw+6CRTDH2WiES1Q6pRcIm+4UZgO1VI41BgKxzdTv3WEyrNE3lvxikGMR1IHnFGjZWaWa9SdWvuDGSZeAWpQoFGr/LV7Scsi1EaJqjWHc9NTZBTZTgTOCl3M40pZSM6wI6lksaog3x26IScWqVPokTZkobM1N8TOY21Hseh7YypGepFbyr+53UyE10FOZdpZlCy+aIoE8QkZPo16XOFzIixJZQpbm8lbEgVZcZmU7YheIsvLxP/vHZdc5sX1fpNkUYJjuEEzsCDS6jDHTTABwYIz/AKb86j8+K8Ox/z1hWnmDmCP3A+fwBPxIzM</latexit> <latexit sha1_base64="kF4y8yL2rgRNwd6KnxzHbkHreI8=">AAAB53icbVBNS8NAEJ34WetX1aOXxSJ4KokI6q3oxWMLxhbaUDbbSbt2swm7G6GE/gIvHlS8+pe8+W/ctjlo64OBx3szzMwLU8G1cd1vZ2V1bX1js7RV3t7Z3duvHBw+6CRTDH2WiES1Q6pRcIm+4UZgO1VI41BgKxzdTv3WEyrNE3lvxikGMR1IHnFGjZWaWa9SdWvuDGSZeAWpQoFGr/LV7Scsi1EaJqjWHc9NTZBTZTgTOCl3M40pZSM6wI6lksaog3x26IScWqVPokTZkobM1N8TOY21Hseh7YypGepFbyr+53UyE10FOZdpZlCy+aIoE8QkZPo16XOFzIixJZQpbm8lbEgVZcZmU7YheIsvLxP/vHZdc5sX1fpNkUYJjuEEzsCDS6jDHTTABwYIz/AKb86j8+K8Ox/z1hWnmDmCP3A+fwBPxIzM</latexit> <latexit sha1_base64="kF4y8yL2rgRNwd6KnxzHbkHreI8=">AAAB53icbVBNS8NAEJ34WetX1aOXxSJ4KokI6q3oxWMLxhbaUDbbSbt2swm7G6GE/gIvHlS8+pe8+W/ctjlo64OBx3szzMwLU8G1cd1vZ2V1bX1js7RV3t7Z3duvHBw+6CRTDH2WiES1Q6pRcIm+4UZgO1VI41BgKxzdTv3WEyrNE3lvxikGMR1IHnFGjZWaWa9SdWvuDGSZeAWpQoFGr/LV7Scsi1EaJqjWHc9NTZBTZTgTOCl3M40pZSM6wI6lksaog3x26IScWqVPokTZkobM1N8TOY21Hseh7YypGepFbyr+53UyE10FOZdpZlCy+aIoE8QkZPo16XOFzIixJZQpbm8lbEgVZcZmU7YheIsvLxP/vHZdc5sX1fpNkUYJjuEEzsCDS6jDHTTABwYIz/AKb86j8+K8Ox/z1hWnmDmCP3A+fwBPxIzM</latexit> d <latexit sha1_base64="493iywG9k3VTthaAR+vadVA7SLQ=">AAAB53icbVBNS8NAEJ34WetX1aOXxSJ4KokI6q3oxWMLxhbaUDabSbt2swm7G6GU/gIvHlS8+pe8+W/ctjlo64OBx3szzMwLM8G1cd1vZ2V1bX1js7RV3t7Z3duvHBw+6DRXDH2WilS1Q6pRcIm+4UZgO1NIk1BgKxzeTv3WEyrNU3lvRhkGCe1LHnNGjZWaUa9SdWvuDGSZeAWpQoFGr/LVjVKWJygNE1TrjudmJhhTZTgTOCl3c40ZZUPax46lkiaog/Hs0Ak5tUpE4lTZkobM1N8TY5poPUpC25lQM9CL3lT8z+vkJr4KxlxmuUHJ5oviXBCTkunXJOIKmREjSyhT3N5K2IAqyozNpmxD8BZfXib+ee265jYvqvWbIo0SHMMJnIEHl1CHO2iADwwQnuEV3pxH58V5dz7mrStOMXMEf+B8/gA2EYy7</latexit> <latexit sha1_base64="493iywG9k3VTthaAR+vadVA7SLQ=">AAAB53icbVBNS8NAEJ34WetX1aOXxSJ4KokI6q3oxWMLxhbaUDabSbt2swm7G6GU/gIvHlS8+pe8+W/ctjlo64OBx3szzMwLM8G1cd1vZ2V1bX1js7RV3t7Z3duvHBw+6DRXDH2WilS1Q6pRcIm+4UZgO1NIk1BgKxzeTv3WEyrNU3lvRhkGCe1LHnNGjZWaUa9SdWvuDGSZeAWpQoFGr/LVjVKWJygNE1TrjudmJhhTZTgTOCl3c40ZZUPax46lkiaog/Hs0Ak5tUpE4lTZkobM1N8TY5poPUpC25lQM9CL3lT8z+vkJr4KxlxmuUHJ5oviXBCTkunXJOIKmREjSyhT3N5K2IAqyozNpmxD8BZfXib+ee265jYvqvWbIo0SHMMJnIEHl1CHO2iADwwQnuEV3pxH58V5dz7mrStOMXMEf+B8/gA2EYy7</latexit> <latexit sha1_base64="493iywG9k3VTthaAR+vadVA7SLQ=">AAAB53icbVBNS8NAEJ34WetX1aOXxSJ4KokI6q3oxWMLxhbaUDabSbt2swm7G6GU/gIvHlS8+pe8+W/ctjlo64OBx3szzMwLM8G1cd1vZ2V1bX1js7RV3t7Z3duvHBw+6DRXDH2WilS1Q6pRcIm+4UZgO1NIk1BgKxzeTv3WEyrNU3lvRhkGCe1LHnNGjZWaUa9SdWvuDGSZeAWpQoFGr/LVjVKWJygNE1TrjudmJhhTZTgTOCl3c40ZZUPax46lkiaog/Hs0Ak5tUpE4lTZkobM1N8TY5poPUpC25lQM9CL3lT8z+vkJr4KxlxmuUHJ5oviXBCTkunXJOIKmREjSyhT3N5K2IAqyozNpmxD8BZfXib+ee265jYvqvWbIo0SHMMJnIEHl1CHO2iADwwQnuEV3pxH58V5dz7mrStOMXMEf+B8/gA2EYy7</latexit> Concrete Operator Instance Remaining BTU for the VM of the concrete Operator Instance Upscaling decision variable Downscaling decision variable 90 -
Resource Setup Under-Provisioning Over-Provisioning Speed 5 8 Average Speed 6
10 Aggregation 2 3 Distance 1 1 Analysis 1 1 Monitor 1 1 91 -
Threshold-based Evaluation Elastic Provisioning Over-Provisioning Under-Provisioning Number of Total Rides
75 75 75 Number of Location Information Items 50742 50742 50742 Cost for Operator Instances 2160.66 2664 1856 (σ = 13.61) (σ = 0.00) (σ = 0.00) Total Makespan in Seconds 6653 6655 6975 (σ = 9.60) (σ = 0.00) (σ = 1.00) Average Duration for the 77 35 355 Report Generation in Seconds (σ = 10.69) (σ = 0.00) (σ = 0.57) Total Number of Delays 21 0 75 (σ = 5.29) (σ = 0.00) (σ = 0.00) SLA Adherence in % 28 % 100 % 0 % (σ = 7.40) (σ = 0.00) (σ= 0.00) 92 -
Upscaling Procedure Upscaling for a specific operator triggered Enough resources
available Not enough resources available Best host selected Not enough resources available Assess utility for hosts Try scaledown for other operator Scaledown successful Scaledown not successful Lease another host Select best host Sufficient processing capabilities for specific operator Add instance for a specific operator 93 -
Downscaling Procedure BTU of host ends Downscaling plan exists Host
leased for another BTU Still running Operator Instances Still running Operator Instances No running Operator Instances Host released Simulate operator downscaling and migration Select best host Prolong leasing Simulation successful Try scaledown of Operator Instances Scaledown not successful Scaledown successful Try migration of Operator Instances Migration is not successful Migration is successful Release host 94 -
Sensor Configuration Emission Rate / min Size (Bytes) Availability Sensor
(S1) 5 95 Production Data (S2) 1 12500 Temperature Sensor (S33) 10 90 98 -
Related Work on Resource Elasticity ‣Resource elasticity for monolithic SPAs
Lim et al.[4] ‣Resource elasticity for individual operators Schneider et al. [5][6] ‣Placement of operators Aneillo et al. [7], Xu et al. [8] ‣Minimization of reconfigurations for the SPE Xu et al. [9] ‣Threshold-based approaches ‣Update of thresholds at runtime Heinze et al. [10] ‣Consideration of stateful operators Castro Fernandez et al. [11] 99 -
Operator Configuration Processing Duration (ms) CPU Shares Memory (MB) Storage
(MB) State Outgoing Ratio O1 600 131 524 68 x 50:1 O2 600 65 440 68 x 100:1 O3 1500 660 452 89 1:3 O4 750 100 430 68 1:1 O5 750 83 502 68 x 1:1 O6 750 77 527 68 x 1:1 O7 700 46 464 68 x 3:1 O8 1300 47 452 70 x 300:1 O9 500 74 466 68 1:0 100 -
Evaluation Results - Compliance (2-level) Threshold BTU 10 BTU 30
BTU 60 Real-time Compliance 40 % (σ = 2%) 48 % (σ = 1%) 50 % (σ = 1%) 55 % (σ = 1%) Near real- time Compliance 68 % (σ = 2%) 84 % (σ = 2%) 88 % (σ = 0%) 93 % (σ = 1%) Relaxed Compliance 72 % (σ = 2%) 88 % (σ = 2%) 91 % (σ = 0%) 95 % (σ = 1%) 101 -
Evaluation Results - Compliance (Stepwise) Threshold BTU 10 BTU 30
BTU 60 Real-time Compliance 40 % (σ = 1%) 49 % (σ = 1%) 52 % (σ = 1%) 53 % (σ = 1%) Near real- time Compliance 67 % (σ = 2%) 85 % (σ = 2%) 90 % (σ = 1%) 93 % (σ = 1%) Relaxed Compliance 71 % (σ = 1%) 89 % (σ = 1%) 93 % (σ = 1%) 95 % (σ = 1%) 102 -
Evaluation Results - Compliance (Random Walk 1) Threshold BTU 10
BTU 30 BTU 60 Real-time Compliance 39 % (σ = 0%) 49 % (σ = 1%) 52 % (σ = 2%) 54 % (σ = 0%) Near real- time Compliance 66 % (σ = 1%) 85 % (σ = 2%) 90 % (σ = 3%) 93 % (σ = 0%) Relaxed Compliance 71 % (σ = 1%) 89 % (σ = 2%) 93 % (σ = 2%) 95 % (σ = 1%) 103 -
Evaluation Results - Compliance (Random Walk 2) Threshold BTU 10
BTU 30 BTU 60 Real-time Compliance 41 % (σ = 0%) 49 % (σ = 2%) 51 % (σ = 1%) 53 % (σ = 0%) Near real- time Compliance 70 % (σ = 1%) 87 % (σ = 2%) 90 % (σ = 1%) 92 % (σ = 1%) Relaxed Compliance 75 % (σ = 1%) 90 % (σ = 2%) 94 % (σ = 1%) 95 % (σ = 0%) 104 -
Evaluation Results - Cost (2-level) Threshold-based BTU-based BTU 10 BTU
30 BTU 60 BTU 10 BTU 30 BTU 60 Resource Cost 66 (σ = 0.00) 86 (σ = 1.73) 122 (σ = 3.46) 82 (σ = 5.13) 96 (σ = 7.94) 104 (σ = 6.93) Real-time Compliance 157 (σ = 2.64) 178 (σ = 4.17) 214 (σ = 4.16) 169 (σ = 6.83) 178 (σ = 6.82) 176 (σ = 4.77) Near real-time Compliance 115 (σ = 3.21) 135 (σ = 4.19) 171 (σ = 2.94) 108 (σ = 6.74) 155 (σ = 8.18) 115 (σ = 6.28) Relaxed Compliance 108 (σ = 2.40) 128 (σ = 3.37) 165 (σ = 2.59) 102 (σ = 6.74) 111 (σ = 8.18) 112 (σ = 6.28) 105 -
Evaluation Results - Cost (Stepwise) Threshold-based BTU-based BTU 10 BTU
30 BTU 60 BTU 10 BTU 30 BTU 60 Resource Cost 58 (σ = 1.73) 79 (σ = 4.58) 120 (σ = 6.00) 72 (σ = 3.79) 92 (σ = 1.73) 98 (σ = 3.84) Real-time Compliance 152 (σ = 1.95) 173 (σ = 4.93) 214 (σ = 6.45) 159 (σ = 0.82) 173 (σ = 0.68) 175 (σ = 4.25) Near real-time Compliance 110 (σ = 1.77) 131 (σ = 4.73) 172 (σ = 6.35) 97 (σ = 0.39) 108 (σ = 0.50) 109 (σ = 3.17) Relaxed Compliance 103 (σ = 1.57) 124 (σ = 1.57) 165 (σ = 6.12) 91 (σ = 1.84) 103 (σ = 1.04) 105 (σ = 2.91) 106 -
Evaluation Results - Cost (Random Walk 1) Threshold-based BTU-based BTU
10 BTU 30 BTU 60 BTU 10 BTU 30 BTU 60 Resource Cost 61 (σ = 1.53) 86 (σ = 1.73) 128 (σ = 6.93) 69 (σ = 3.51) 95 (σ = 1.73) 110 (σ = 3.46) Real-time Compliance 159 (σ = 1.54) 183 (σ = 1.40) 225 (σ = 6.56) 158 (σ = 4.99) 177 (σ = 4.70) 186 (σ = 3.40) Near real-time Compliance 116 (σ = 1.14) 140 (σ = 1.77) 182 (σ = 6.88) 94 (σ = 6.14) 111 (σ = 5.65) 122 (σ = 2.89) Relaxed Compliance 108 (σ = 1.47) 132 (σ = 2.14) 174 (σ = 7.25) 88 (σ = 5.86) 107 (σ = 4.69) 117 (σ = 3.16) 107 -
Evaluation Results - Cost (Random Walk 2) Threshold-based BTU-based BTU
10 BTU 30 BTU 60 BTU 10 BTU 30 BTU 60 Resource Cost 60 (σ = 4.04) 82 (σ = 9.17) 118 (σ = 9.17) 74 (σ = 6.08) 90 (σ = 0.00) 106 (σ = 3.46) Real-time Compliance 164 (σ = 3.84) 187 (σ = 9.88) 223 (σ = 9.88) 173 (σ = 5.35) 184 (σ = 0.99) 196 (σ = 3.33) Near real-time Compliance 113 (σ = 3.47) 136 (σ = 9.93) 172 (σ = 9.93) 100 (σ = 6.65) 109 (σ = 2.47) 121 (σ = 3.59) Relaxed Compliance 104 (σ = 2.49) 127 (σ = 8.29) 163 (σ = 8.29) 93 (σ = 7.15) 102 (σ = 1.61) 116 (σ = 3.48) 108 -
Evaluation Results - Threshold-based Baseline (2-Level) 0 20 40 60
80 100 120 0 2 4 6 8 Time (Minutes) Amount of Manufacturing Machines and VMs 0 10 20 30 40 50 60 70 Amount of Operator Instances 0 0 Pattern VMs Operator Instances 109 -
Evaluation Results - Threshold-based Baseline (Stepwise) 0 0 Pattern VMs
Operator Instances 110 0 20 40 60 80 100 120 0 2 4 6 8 Time (Minutes) Amount of Manufacturing Machines and VMs 0 10 20 30 40 50 60 70 Amount of Operator Instances -
Evaluation Results - Threshold-based Baseline (Random Walk 1) 0 0
Pattern VMs Operator Instances 111 0 20 40 60 80 100 120 0 2 4 6 8 Time (Minutes) Amount of Manufacturing Machines and VMs 0 10 20 30 40 50 60 70 Amount of Operator Instances -
Evaluation Results - Threshold-based Baseline (Random Walk 2) 0 0
Pattern VMs Operator Instances 112 0 20 40 60 80 100 120 0 2 4 6 8 Time (Minutes) Amount of Manufacturing Machines and VMs 0 10 20 30 40 50 60 70 Amount of Operator Instances -
Evaluation Results - BTU 10 (2-Level) 0 0 Pattern VMs
Operator Instances 113 0 20 40 60 80 100 120 0 2 4 6 8 Time (Minutes) Amount of Manufacturing Machines and VMs 0 10 20 30 40 50 60 70 Amount of Operator Instances -
Evaluation Results - BTU 30 (2-Level) 0 0 Pattern VMs
Operator Instances 0 20 40 60 80 100 120 0 2 4 6 8 Time (Minutes) Amount of Manufacturing Machines and VMs 0 10 20 30 40 50 60 70 Amount of Operator Instances 114 -
Evaluation Results - BTU 60 (2-Level) 0 0 Pattern VMs
Operator Instances 115 0 20 40 60 80 100 120 0 2 4 6 8 Time (Minutes) Amount of Manufacturing Machines and VMs 0 10 20 30 40 50 60 70 Amount of Operator Instances -
Evaluation Results - BTU 30 (Stepwise) 0 0 Pattern VMs
Operator Instances 116 0 20 40 60 80 100 120 0 2 4 6 8 Time (Minutes) Amount of Manufacturing Machines and VMs 0 10 20 30 40 50 60 70 Amount of Operator Instances -
Evaluation Results - BTU 30 (Random Walk 1) 0 0
Pattern VMs Operator Instances 117 0 20 40 60 80 100 120 0 2 4 6 8 Time (Minutes) Amount of Manufacturing Machines and VMs 0 10 20 30 40 50 60 70 Amount of Operator Instances -
Evaluation Results - BTU 30 (Random Walk 2) 0 0
Pattern VMs Operator Instances 118 0 20 40 60 80 100 120 0 2 4 6 8 Time (Minutes) Amount of Manufacturing Machines and VMs 0 10 20 30 40 50 60 70 Amount of Operator Instances -
Contribution Overview P1: C. Hochreiner, M. Vögler, S. Schulte, S.
Dustdar, "Cost-efficient enactment of stream processing topologies," PeerJ Computer Science. P2: C. Hochreiner, S. Schulte, S. Dustdar, and F. Lecue, "Elastic Stream Processing for Distributed Environments," IEEE Internet Computing. P3: C. Hochreiner, M. Nardelli, B. Knasmueller, S. Schulte, and S. Dustdar, “VTDL: A Notation for Data Stream Processing Applications," SOSE 2018. B2: R. Ganaus, Deployment Techniques for Dynamic Application Scenarios. P4: C. Hochreiner, M. Vögler, P. Waibel, and S. Dustdar, "VISP: An Ecosystem for Elastic Data Stream Processing for the Internet of Things,” EDOC 2016. P5: C. Hochreiner, M. Vögler, S. Schulte, and S. Dustdar, "Elastic Stream Processing for the Internet of Things," CLOUD 2016. P6: C. Hochreiner, M. Vögler, J. M. Schleicher, C. Inzinger, S. Schulte, and S. Dustdar, "Nomadic Applications Traveling in the Fog," CN4IoT 2017. P7: C. Hochreiner, "VISP Testbed - A Toolkit for Modeling and Evaluating Resource Provisioning Algorithms for Stream Processing Applications," ZEUS 2017. P8: M. Borkowski, C. Hochreiner, and S. Schulte, "Moderated Resource Elasticity for Stream Processing Applications," Auto-DaSP 2017. P9: M. Nardelli, C. Hochreiner, and S. Schulte, "Elastic Provisioning of Virtual Machines for Container Deployment," ACPROSS 2017. M1: T. Biasion, A Network-based Business Model Framework for the Internet of Things. M2: T. Hiessl, Optimizing the Placement of Stream Processing Operators in the Fog. M3: B. Knasmueller, On Fault Tolerance in Stream Processing Systems. M4: F. Rinker, A Monitoring Framework for Stream Processing Applications in Heterogenous Environments. B1: A. Carrasco, Scalability for Data Stream Processing Frameworks. Infrastructure Data Sources VISP Marketplace Cloud Resources Edge Resources VISP Runtime Stream Processing Application Topology Configuration Operator Replication Infrastructure Management Operator Provisioning Messaging Operator Migration Operator Placement Shared Data Failure Detection Privacy Security Topology Enactment Manufacturing Machines Wearable Devices News Portals Failure Compensation Monitoring Monetization Operator Distribution Topology Builder Operator Registry Runtime Synchronization Stock Market SLA Compliance Operational Layer Communication Layer Infrastructure Layer Cross-cutting Concerns P2 P3 P4 P6 P1 P5 P7 P8 P9 M1 M2 M3 M4 B1 B2 119