PhD Defense

Slides from my PhD Defense

Transcript

1. AN ELASTIC DATA STREAM PROCESSING ECOSYSTEM FOR DISTRIBUTED ENVIRONMENTS Christoph Hochreiner
2. Outline ‣ Preliminaries ‣ Motivational Scenario ‣ Contributions • The VISP Ecosystem • The Vienna Topology Description Language • Cost-efficient Enactment of Stream Processing Applications ‣ Conclusion 2
3. Preliminaries 3
4. Data Stream Processing Stream Processing Application (SPA) Data Source Data Sink Raw Data Value Added Data 4
5. Stream Processing Application (SPA) Data Stream Processing Data Source Data Sink Transformator Filter Aggregator 5
6. Stream Processing Application (SPA) O1 Stream Processing Engine (SPE) Data Stream Processing Data Source Data Sink Transformator Filter Aggregator O1 O2 O3 6
7. Motivational Scenario 7
8. Motivational Scenario Goal Monitor manufacturing machines that are located in different geographic locations in a central location to optimize the overall productivity. 8
9. Motivational Scenario Manufacturing Machine Manufacturing Machine Manufacturing Machine Availability Sensor (S1) Production Data (S2) Temperature Sensor (S3) Filter Availability (O1) Parse and Distribute Data (O3) Calculate Performance (O4) Calculate Availability (O5) Calculate Quality (O6) Monitor Temperature (O2) Calculate OEE (O7) Generate Report (O8) Inform User (O9) 9
10. Manufacturing Machine Manufacturing Machine Manufacturing Machine Availability Sensor (S1) Production Data (S2) Temperature Sensor (S3) Filter Availability (O1) Parse and Distribute Data (O3) Calculate Performance (O4) Calculate Availability (O5) Calculate Quality (O6) Monitor Temperature (O2) Calculate OEE (O7) Generate Report (O8) Inform User (O9) De-Centralized Deployment + Network efficient - Redundant infrastructure - Hard to manage 10
11. - Not network efficient + Hardware efficient + Easy to manage Manufacturing Machine Manufacturing Machine Manufacturing Machine Availability Sensor (S1) Production Data (S2) Temperature Sensor (S3) Filter Availability (O1) Parse and Distribute Data (O3) Calculate Performance (O4) Calculate Availability (O5) Calculate Quality (O6) Monitor Temperature (O2) Calculate OEE (O7) Generate Report (O8) Inform User (O9) Centralized Deployment 11
12. Research Question 1 How can geographically distributed data streams be processed efficiently? 12
13. Creation of Stream Processing Applications Manufacturing Machine Manufacturing Machine Manufacturing Machine Availability Sensor (S1) Production Data (S2) Temperature Sensor (S3) Filter Availability (O1) Parse and Distribute Data (O3) Calculate Performance (O4) Calculate Availability (O5) Calculate Quality (O6) Monitor Temperature (O2) Calculate OEE (O7) Generate Report (O8) Inform User (O9) ‣ SPAs are mostly implemented from scratch ‣ Service level objectives (SLOs) for operators are not supported ‣ No support for distributed SPEs State of the Art 13
14. ‣ Domain experts can create SPAs ‣ Reuse of existing operators ‣ Consider non-functional aspects like the geographic location or SLOs Creation of Stream Processing Applications Manufacturing Machine Manufacturing Machine Manufacturing Machine Availability Sensor (S1) Production Data (S2) Temperature Sensor (S3) Filter Availability (O1) Parse and Distribute Data (O3) Calculate Performance (O4) Calculate Availability (O5) Calculate Quality (O6) Monitor Temperature (O2) Calculate OEE (O7) Generate Report (O8) Inform User (O9) Goal 14
15. Research Question 2 How can SPAs be described to allow a distributed deployment and respect SLOs? 15
16. Operation of Stream Processing Applications ‣ Ensure enough computational resources for SPAs to process the data without delay ‣ Minimize the cost for computational resources ‣ Minimize potential penalty cost for delayed data 16
17. Research Question 3 How can SPAs be optimally executed on computational resources? 17
18. Contributions 18
19. Contribution 1 A system design for a holistic stream processing ecosystem in distributed environments C. Hochreiner, S. Schulte, S. Dustdar, and F. Lecue, Elastic Stream Processing for Distributed Environments, IEEE Internet Computing, 2015. C. Hochreiner, M. Vögler, P. Waibel, and S. Dustdar, VISP: An Ecosystem for Elastic Data Stream Processing for the Internet of Things, EDOC 2016. 19
20. Manufacturing Machine Manufacturing Machine Manufacturing Machine Availability Sensor (S1) Production Data (S2) Temperature Sensor (S3) Filter Availability (O1) Parse and Distribute Data (O3) Calculate Performance (O4) Calculate Availability (O5) Calculate Quality (O6) Monitor Temperature (O2) Calculate OEE (O7) Generate Report (O8) Inform User (O9) Distributed Deployment - Needs a new system design + Network & hardware efficient + Easy to manage 20
21. Challenges for a New System Design ‣ Distributed deployment of SPAs ‣ Create an ecosystem that supports the reuse of operators to ease the creation of new SPAs ‣ Elastic provisioning of processing capabilities at runtime 21
22. IBM System S Apache Spark / Storm AWS, Google Cloud, Microsoft Azure ThingWorx ThingStore [1] Distributed Apache Storm [2] StreamCloud [3] Heterogenous Clouds ✓ Resource Elasticity ( ✓ ) ✓ ✓ ✓ ✓ Cost Efficiency ( ✓ ) ✓ ✓ Marketplace ( ✓ ) ✓ ✓ Related Stream Processing Engines and Platforms for the IoT 22
23. System Design for the VISP Ecosystem Infrastructure Data Sources VISP Marketplace VISP Runtime Operational Layer Communication Layer Infrastructure Layer Stream Processing Application Cross-cutting Concerns 23
24. Spain United Kingdom Germany O1 O2 O8 O9 O3 O4 O5 O6 O7 Sweden O1 O2 O1 O2 O3 O4 O5 O6 O7 O3 O4 O5 O6 O7 O1 O2 S1 S2 S3 S1 S2 S3 S1 S2 S3 S1 S2 S3 SN Sensor ON Operator Dataflow Evaluation Scenario 24
25. Evaluation Setup VISP Runtime VISP Runtime VISP Runtime VISP Runtime VISP Marketplace ‣ Define the topology for the SPA ‣ Deploy the VISP Runtime instances in an OpenStack- based private cloud 25
26. Evaluation Walk Through VISP Runtime VISP Runtime VISP Runtime VISP Runtime VISP Marketplace 1. Upload the topology to one VISP Runtime 26
27. Evaluation Walk Through VISP Runtime VISP Runtime VISP Runtime VISP Runtime VISP Marketplace 2. Topology is distributed among all VISP Runtimes 27
28. Evaluation Walk Through VISP Runtime VISP Runtime VISP Runtime VISP Runtime VISP Marketplace 3. Operators are fetched from the VISP Marketplace and the data flow is configured by each VISP Runtime 28
29. Evaluation Walk Through VISP Runtime VISP Runtime VISP Runtime VISP Runtime VISP Marketplace 4. Data is processed 29
30. Contribution Summary ‣ Introduced a system design to enable the distributed operation of SPAs based on the autonomous coordination among individual runtimes ‣ Proposed the VISP Marketplace to support the reuse of operators and incentivize 3rd party developers to contribute ‣ Achieve network-efficiency by already filtering data at the edge of the network ‣ Achieve resource-efficiency by getting rid of redundant deployments 30
31. Contribution 2 An approach to define quality of service- and location-specific aspects for SPAs C. Hochreiner, M. Nardelli, B. Knasmueller, S. Schulte, and S. Dustdar, VTDL: A Notation for Data Stream Processing Applications, SOSE 2018. 31
32. Non-functional Aspects Define non-functional aspects for the SPA Aspects for Designing Stream Processing Applications Basic Functionality Define the orchestration of the individual operators and the data flow through the SPA 32
33. Non-functional Aspects ‣ Deployment preferences to support distributed deployments ‣ SLOs to describe performance aspects of operators ‣ Definition of failure compensation strategies ‣ Support for operator annotations to perform integrity checks or semantic match-making ‣ Model different data transfer modes among the operators 33
34. Related Work on Topology Description Languages Dataflow Apache Beam SPL CQL Apache Spark / Storm Deployment Preferences ( ✓ ) QoS Aspects ( ✓ ) Failure Compensation ( ✓ ) Semantic Annotation ✓ Data Transfer Aspects better motivate the + present the desig vtDL 34
35. Manufacturing Machine Manufacturing Machine Manufacturing Machine Availability Sensor (S1) Production Data (S2) Temperature Sensor (S3) Filter Availability (O1) Parse and Distribute Data (O3) Calculate Performance (O4) Calculate Availability (O5) Calculate Quality (O6) Monitor Temperature (O2) Calculate OEE (O7) Generate Report (O8) Inform User (O9) Vienna Topology Description Language (VTDL) $temperature = Source() { concreteLocation : ":::::ffff:8083:c001/cpu", type : "temperatureSensor", outputFormat : "temperature" } $monitor = Operator($temperature) { allowedLocations : *, concreteLocation : ":::::ffff:8083:c001/cpu", inputFormat : "temperature", type : "monitorTemperature", outputFormat : "alert", stateful : "false", replicationAllowed : "true", responseTime : "0.5", compensation : "mailto:admin@tuwien.ac.at" } $informUser = Operator($monitor) { allowedLocations : ":::::ffff:8083:c001" ":::::ffff:8083:c002", inputFormat : "alert, report", type : "informUser", outputFormat : "message", queueLength : "200", protocol : "microbatch/50items", compensation : "deploy:www.visp.io/backup.vtdl" } 35
36. Spain United Kingdom Germany O1 O2 O8 O9 O3 O4 O5 O6 O7 Sweden O1 O2 O1 O2 O3 O4 O5 O6 O7 O3 O4 O5 O6 O7 O1 O2 S1 S2 S3 S1 S2 S3 S1 S2 S3 S1 S2 S3 SN Sensor ON Operator Dataflow Evaluation Scenario 1 - New Topology in One Location 36
37. Spain United Kingdom Germany O1 O2 O8 O9 O3 O4 O5 O6 O7 Sweden O1 O2 O1 O2 O3 O4 O5 O6 O7 O3 O4 O5 O6 O7 O1 O2 S1 S2 S3 S1 S2 S3 S1 S2 S3 S1 S2 S3 SN Sensor ON Operator Dataflow Evaluation Scenario 2 - New Topology across Four Locations Amazon EC2 Tokyo Amazon EC2 Frankfurt Amazon EC2 Ohio 37
38. Spain United Kingdom Germany O1 O2 O8 O9 O3 O4 O5 O6 O7 Sweden O1 O2 O1 O2 O3 O4 O5 O6 O7 O3 O4 O5 O6 O7 O1 O2 S1 S2 S3 S1 S2 S3 S1 S2 S3 S1 S2 S3 SN Sensor ON Operator Dataflow Evaluation Scenario 3 - Network Disruption 38
39. Spain United Kingdom Germany O1 O2 O8 O9 O3 O4 O5 O6 O7 Sweden O1 O2 O1 O2 O3 O4 O5 O6 O7 O3 O4 O5 O6 O7 O1 O2 S1 S2 S3 S1 S2 S3 S1 S2 S3 S1 S2 S3 SN Sensor ON Operator Dataflow Old dataflow New Dataflow Evaluation Scenario 3 - Network Disruption 39
40. Spain United Kingdom Germany O1 O2 O8 O9 O3 O4 O5 O6 O7 Sweden O1 O2 O1 O2 O3 O4 O5 O6 O7 O3 O4 O5 O6 O7 O1 O2 S1 S2 S3 S1 S2 S3 S1 S2 S3 S1 S2 S3 SN Sensor ON Operator Dataflow Evaluation Scenario 4 - Load Reduction 40
41. Evaluation Scenario 4 - Load Reduction Spain United Kingdom Germany O1 O2 O8 O9 O3 O4 O5 O6 O7 Sweden O1 O2 O1 O2 O3 O4 O5 O6 O7 O3 O4 O5 O6 O7 O1 O2 S1 S2 S3 S1 S2 S3 S1 S2 S3 S1 S2 S3 SN Sensor ON Operator Dataflow Old dataflow New Dataflow 41
42. Spain United Kingdom Germany O1 O2 O8 O9 O3 O4 O5 O6 O7 Sweden O1 O2 O1 O2 O3 O4 O5 O6 O7 O3 O4 O5 O6 O7 O1 O2 S1 S2 S3 S1 S2 S3 S1 S2 S3 S1 S2 S3 SN Sensor ON Operator Dataflow Evaluation Scenario 5 - Bugfix for an Operator 42
43. Evaluation Baseline (Apache Storm) VTDL-based New topology in one location 54.98 s 54.98 s New topology across four locations 68.10 s 55.95 s Network disruption — 7.53 s Load reduction 54.05 s 3.30 s Bugfix for an operator 55.62 s 5.69 s 43
44. Contribution Summary ‣ Introduced a topology description approach that supports non-functional aspects ‣ Evaluate failure compensation mechanism across different VISP Runtimes ‣ Showed a reduced downtime when using the VTDL-based approach compared to state of the art ones 44
45. Contribution 3 Cost-efficient resource provisioning approaches for SPAs C. Hochreiner, M. Vögler, S. Schulte, and S. Dustdar, Elastic Stream Processing for the Internet of Things, CLOUD 2016. C. Hochreiner, M. Vögler, S. Schulte, S. Dustdar, Cost-efficient enactment of stream processing topologies, PeerJ Computer Science, 2017. 45
46. Provisioning Approaches Time Computational Resources and Data Volume Fixed under-provisioning 46
47. Provisioning Approaches Fixed over-provisioning Time Computational Resources and Data Volume 47
48. Provisioning Approaches Elastic-provisioning Time Computational Resources and Data Volume 48
49. Optimization Model min X p2P pi + u · N + d · N <latexit sha1_base64="1Y8N28G+CXvJS/CnK7ZQG+jHk1A=">AAACInicbZDLSgMxFIYz3q23qks3B4sgCGUqgpdV0Y0rqWBV6JQhk0nbYJIZkjNiGfoubnwVNy68rQQfxrRW0NYfAh//OYeT80epFBZ9/8ObmJyanpmdmy8sLC4trxRX1y5tkhnG6yyRibmOqOVSaF5HgZJfp4ZTFUl+Fd2c9OtXt9xYkegL7Ka8qWhbi5ZgFJ0VFo8AAuR3aFSuhO5BYDMV5ikEQkOtB2koYAcyCFicIJw5jn+4EBZLftkfCMahMoQSGaoWFt+COGGZ4hqZpNY2Kn6KzZwaFEzyXiHILE8pu6Ft3nCoqeK2mQ9u7MGWc2JoJcY9jTBwf0/kVFnbVZHrVBQ7drTWN/+rNTJsHTRzodMMuWbfi1qZBEygHxjEwnCGsuuAMiPcX4F1qKEMXaz9ECqjJ49Dfbd8WPbP90rV42Eac2SDbJJtUiH7pEpOSY3UCSP35JE8kxfvwXvyXr3379YJbzizTv7I+/wCd9yh0g==</latexit> <latexit sha1_base64="1Y8N28G+CXvJS/CnK7ZQG+jHk1A=">AAACInicbZDLSgMxFIYz3q23qks3B4sgCGUqgpdV0Y0rqWBV6JQhk0nbYJIZkjNiGfoubnwVNy68rQQfxrRW0NYfAh//OYeT80epFBZ9/8ObmJyanpmdmy8sLC4trxRX1y5tkhnG6yyRibmOqOVSaF5HgZJfp4ZTFUl+Fd2c9OtXt9xYkegL7Ka8qWhbi5ZgFJ0VFo8AAuR3aFSuhO5BYDMV5ikEQkOtB2koYAcyCFicIJw5jn+4EBZLftkfCMahMoQSGaoWFt+COGGZ4hqZpNY2Kn6KzZwaFEzyXiHILE8pu6Ft3nCoqeK2mQ9u7MGWc2JoJcY9jTBwf0/kVFnbVZHrVBQ7drTWN/+rNTJsHTRzodMMuWbfi1qZBEygHxjEwnCGsuuAMiPcX4F1qKEMXaz9ECqjJ49Dfbd8WPbP90rV42Eac2SDbJJtUiH7pEpOSY3UCSP35JE8kxfvwXvyXr3379YJbzizTv7I+/wCd9yh0g==</latexit> <latexit sha1_base64="1Y8N28G+CXvJS/CnK7ZQG+jHk1A=">AAACInicbZDLSgMxFIYz3q23qks3B4sgCGUqgpdV0Y0rqWBV6JQhk0nbYJIZkjNiGfoubnwVNy68rQQfxrRW0NYfAh//OYeT80epFBZ9/8ObmJyanpmdmy8sLC4trxRX1y5tkhnG6yyRibmOqOVSaF5HgZJfp4ZTFUl+Fd2c9OtXt9xYkegL7Ka8qWhbi5ZgFJ0VFo8AAuR3aFSuhO5BYDMV5ikEQkOtB2koYAcyCFicIJw5jn+4EBZLftkfCMahMoQSGaoWFt+COGGZ4hqZpNY2Kn6KzZwaFEzyXiHILE8pu6Ft3nCoqeK2mQ9u7MGWc2JoJcY9jTBwf0/kVFnbVZHrVBQ7drTWN/+rNTJsHTRzodMMuWbfi1qZBEygHxjEwnCGsuuAMiPcX4F1qKEMXaz9ECqjJ49Dfbd8WPbP90rV42Eac2SDbJJtUiH7pEpOSY3UCSP35JE8kxfvwXvyXr3379YJbzizTv7I+/wCd9yh0g==</latexit> 49
50. Optimization Model Upscaling ‣ 150 items waiting to be processed by an operator ‣ CPU usage of an operator is higher than 90% min X p2P pi + u · N + d · N <latexit sha1_base64="1Y8N28G+CXvJS/CnK7ZQG+jHk1A=">AAACInicbZDLSgMxFIYz3q23qks3B4sgCGUqgpdV0Y0rqWBV6JQhk0nbYJIZkjNiGfoubnwVNy68rQQfxrRW0NYfAh//OYeT80epFBZ9/8ObmJyanpmdmy8sLC4trxRX1y5tkhnG6yyRibmOqOVSaF5HgZJfp4ZTFUl+Fd2c9OtXt9xYkegL7Ka8qWhbi5ZgFJ0VFo8AAuR3aFSuhO5BYDMV5ikEQkOtB2koYAcyCFicIJw5jn+4EBZLftkfCMahMoQSGaoWFt+COGGZ4hqZpNY2Kn6KzZwaFEzyXiHILE8pu6Ft3nCoqeK2mQ9u7MGWc2JoJcY9jTBwf0/kVFnbVZHrVBQ7drTWN/+rNTJsHTRzodMMuWbfi1qZBEygHxjEwnCGsuuAMiPcX4F1qKEMXaz9ECqjJ49Dfbd8WPbP90rV42Eac2SDbJJtUiH7pEpOSY3UCSP35JE8kxfvwXvyXr3379YJbzizTv7I+/wCd9yh0g==</latexit> <latexit sha1_base64="1Y8N28G+CXvJS/CnK7ZQG+jHk1A=">AAACInicbZDLSgMxFIYz3q23qks3B4sgCGUqgpdV0Y0rqWBV6JQhk0nbYJIZkjNiGfoubnwVNy68rQQfxrRW0NYfAh//OYeT80epFBZ9/8ObmJyanpmdmy8sLC4trxRX1y5tkhnG6yyRibmOqOVSaF5HgZJfp4ZTFUl+Fd2c9OtXt9xYkegL7Ka8qWhbi5ZgFJ0VFo8AAuR3aFSuhO5BYDMV5ikEQkOtB2koYAcyCFicIJw5jn+4EBZLftkfCMahMoQSGaoWFt+COGGZ4hqZpNY2Kn6KzZwaFEzyXiHILE8pu6Ft3nCoqeK2mQ9u7MGWc2JoJcY9jTBwf0/kVFnbVZHrVBQ7drTWN/+rNTJsHTRzodMMuWbfi1qZBEygHxjEwnCGsuuAMiPcX4F1qKEMXaz9ECqjJ49Dfbd8WPbP90rV42Eac2SDbJJtUiH7pEpOSY3UCSP35JE8kxfvwXvyXr3379YJbzizTv7I+/wCd9yh0g==</latexit> <latexit sha1_base64="1Y8N28G+CXvJS/CnK7ZQG+jHk1A=">AAACInicbZDLSgMxFIYz3q23qks3B4sgCGUqgpdV0Y0rqWBV6JQhk0nbYJIZkjNiGfoubnwVNy68rQQfxrRW0NYfAh//OYeT80epFBZ9/8ObmJyanpmdmy8sLC4trxRX1y5tkhnG6yyRibmOqOVSaF5HgZJfp4ZTFUl+Fd2c9OtXt9xYkegL7Ka8qWhbi5ZgFJ0VFo8AAuR3aFSuhO5BYDMV5ikEQkOtB2koYAcyCFicIJw5jn+4EBZLftkfCMahMoQSGaoWFt+COGGZ4hqZpNY2Kn6KzZwaFEzyXiHILE8pu6Ft3nCoqeK2mQ9u7MGWc2JoJcY9jTBwf0/kVFnbVZHrVBQ7drTWN/+rNTJsHTRzodMMuWbfi1qZBEygHxjEwnCGsuuAMiPcX4F1qKEMXaz9ECqjJ49Dfbd8WPbP90rV42Eac2SDbJJtUiH7pEpOSY3UCSP35JE8kxfvwXvyXr3379YJbzizTv7I+/wCd9yh0g==</latexit> 50
51. Optimization Model min X p2P pi + u · N + d · N <latexit sha1_base64="1Y8N28G+CXvJS/CnK7ZQG+jHk1A=">AAACInicbZDLSgMxFIYz3q23qks3B4sgCGUqgpdV0Y0rqWBV6JQhk0nbYJIZkjNiGfoubnwVNy68rQQfxrRW0NYfAh//OYeT80epFBZ9/8ObmJyanpmdmy8sLC4trxRX1y5tkhnG6yyRibmOqOVSaF5HgZJfp4ZTFUl+Fd2c9OtXt9xYkegL7Ka8qWhbi5ZgFJ0VFo8AAuR3aFSuhO5BYDMV5ikEQkOtB2koYAcyCFicIJw5jn+4EBZLftkfCMahMoQSGaoWFt+COGGZ4hqZpNY2Kn6KzZwaFEzyXiHILE8pu6Ft3nCoqeK2mQ9u7MGWc2JoJcY9jTBwf0/kVFnbVZHrVBQ7drTWN/+rNTJsHTRzodMMuWbfi1qZBEygHxjEwnCGsuuAMiPcX4F1qKEMXaz9ECqjJ49Dfbd8WPbP90rV42Eac2SDbJJtUiH7pEpOSY3UCSP35JE8kxfvwXvyXr3379YJbzizTv7I+/wCd9yh0g==</latexit> <latexit sha1_base64="1Y8N28G+CXvJS/CnK7ZQG+jHk1A=">AAACInicbZDLSgMxFIYz3q23qks3B4sgCGUqgpdV0Y0rqWBV6JQhk0nbYJIZkjNiGfoubnwVNy68rQQfxrRW0NYfAh//OYeT80epFBZ9/8ObmJyanpmdmy8sLC4trxRX1y5tkhnG6yyRibmOqOVSaF5HgZJfp4ZTFUl+Fd2c9OtXt9xYkegL7Ka8qWhbi5ZgFJ0VFo8AAuR3aFSuhO5BYDMV5ikEQkOtB2koYAcyCFicIJw5jn+4EBZLftkfCMahMoQSGaoWFt+COGGZ4hqZpNY2Kn6KzZwaFEzyXiHILE8pu6Ft3nCoqeK2mQ9u7MGWc2JoJcY9jTBwf0/kVFnbVZHrVBQ7drTWN/+rNTJsHTRzodMMuWbfi1qZBEygHxjEwnCGsuuAMiPcX4F1qKEMXaz9ECqjJ49Dfbd8WPbP90rV42Eac2SDbJJtUiH7pEpOSY3UCSP35JE8kxfvwXvyXr3379YJbzizTv7I+/wCd9yh0g==</latexit> <latexit sha1_base64="1Y8N28G+CXvJS/CnK7ZQG+jHk1A=">AAACInicbZDLSgMxFIYz3q23qks3B4sgCGUqgpdV0Y0rqWBV6JQhk0nbYJIZkjNiGfoubnwVNy68rQQfxrRW0NYfAh//OYeT80epFBZ9/8ObmJyanpmdmy8sLC4trxRX1y5tkhnG6yyRibmOqOVSaF5HgZJfp4ZTFUl+Fd2c9OtXt9xYkegL7Ka8qWhbi5ZgFJ0VFo8AAuR3aFSuhO5BYDMV5ikEQkOtB2koYAcyCFicIJw5jn+4EBZLftkfCMahMoQSGaoWFt+COGGZ4hqZpNY2Kn6KzZwaFEzyXiHILE8pu6Ft3nCoqeK2mQ9u7MGWc2JoJcY9jTBwf0/kVFnbVZHrVBQ7drTWN/+rNTJsHTRzodMMuWbfi1qZBEygHxjEwnCGsuuAMiPcX4F1qKEMXaz9ECqjJ49Dfbd8WPbP90rV42Eac2SDbJJtUiH7pEpOSY3UCSP35JE8kxfvwXvyXr3379YJbzizTv7I+/wCd9yh0g==</latexit> 51 Downscaling ‣ 5 items waiting to be processed by an operator ‣ CPU usage of an operator is less than 10%
52. Evaluation Setup ‣ Each VM hosts one operator instance ‣ BTU for VMs is 60 minutes Distance Speed Average Speed Aggregation Analysis Monitor 52
53. Evaluation Results 0 5 10 15 20 25 0 20 40 60 80 100 120 100 200 300 400 500 600 700 800 900 Number of Operator Instances Incoming Data Items Time in Minutes Under-provisioning Over-provisioning Elastic-provisioning Streaming Data Data arrival pattern is based on day-night cycles of taxi rides within Beijing 53
54. Evaluation Results 0 5 10 15 20 25 0 20 40 60 80 100 120 100 200 300 400 500 600 700 800 900 Number of Operator Instances Incoming Data Items Time in Minutes Under-provisioning Over-provisioning Elastic-provisioning Streaming Data Overall data processing takes the longest due to too little computational resources 54
55. Evaluation Results Data is processed without any delay at high costs 0 5 10 15 20 25 0 20 40 60 80 100 120 100 200 300 400 500 600 700 800 900 Number of Operator Instances Incoming Data Items Time in Minutes Under-provisioning Over-provisioning Elastic-provisioning Streaming Data 55
56. total makespan is the same as for the over-provisioning and 5% faster compared to under-provisioning Evaluation Results 0 5 10 15 20 25 0 20 40 60 80 100 120 100 200 300 400 500 600 700 800 900 Number of Operator Instances Incoming Data Items Time in Minutes Under-provisioning Over-provisioning Elastic-provisioning Streaming Data 56
57. Observations ‣ Threshold-based provisioning can lead to oscillation ‣ Resources may not be used up to their full potential ‣ Threshold-based approaches can only react to changing data volumes which leads to delays ‣ VMs are too coarse grained for resource provisioning 57
58. Optimization Model Min <latexit sha1_base64="Iww38d60nYMRtuU88ICQJ2ruV0o=">AAAB/HicbVDLSgNBEJyNrxhf8XHzMhgET2FXBPUW9OJFiOCaQHYJs5NJMmRmdjPTK8Yl+CtePKh49UO8+TdOHgdNLGgoqrrp7ooSwQ247reTW1hcWl7JrxbW1jc2t4rbO3cmTjVlPo1FrOsRMUxwxXzgIFg90YzISLBa1Lsc+bV7pg2P1S0MEhZK0lG8zSkBKzWLewGwB9Ayu+YKB/1+Slp42CyW3LI7Bp4n3pSU0BTVZvEraMU0lUwBFcSYhucmEGZEA6eCDQtBalhCaI90WMNSRSQzYTa+fogPrdLC7VjbUoDH6u+JjEhjBjKynZJA18x6I/E/r5FC+yzMuEpSYIpOFrVTgSHGoyhwi2tGQQwsIVRzeyumXaIJBRtYwYbgzb48T/zj8nnZvTkpVS6maeTRPjpAR8hDp6iCrlAV+YiiR/SMXtGb8+S8OO/Ox6Q150xndtEfOJ8/+J6VGg==</latexit> <latexit sha1_base64="Iww38d60nYMRtuU88ICQJ2ruV0o=">AAAB/HicbVDLSgNBEJyNrxhf8XHzMhgET2FXBPUW9OJFiOCaQHYJs5NJMmRmdjPTK8Yl+CtePKh49UO8+TdOHgdNLGgoqrrp7ooSwQ247reTW1hcWl7JrxbW1jc2t4rbO3cmTjVlPo1FrOsRMUxwxXzgIFg90YzISLBa1Lsc+bV7pg2P1S0MEhZK0lG8zSkBKzWLewGwB9Ayu+YKB/1+Slp42CyW3LI7Bp4n3pSU0BTVZvEraMU0lUwBFcSYhucmEGZEA6eCDQtBalhCaI90WMNSRSQzYTa+fogPrdLC7VjbUoDH6u+JjEhjBjKynZJA18x6I/E/r5FC+yzMuEpSYIpOFrVTgSHGoyhwi2tGQQwsIVRzeyumXaIJBRtYwYbgzb48T/zj8nnZvTkpVS6maeTRPjpAR8hDp6iCrlAV+YiiR/SMXtGb8+S8OO/Ox6Q150xndtEfOJ8/+J6VGg==</latexit> <latexit sha1_base64="Iww38d60nYMRtuU88ICQJ2ruV0o=">AAAB/HicbVDLSgNBEJyNrxhf8XHzMhgET2FXBPUW9OJFiOCaQHYJs5NJMmRmdjPTK8Yl+CtePKh49UO8+TdOHgdNLGgoqrrp7ooSwQ247reTW1hcWl7JrxbW1jc2t4rbO3cmTjVlPo1FrOsRMUxwxXzgIFg90YzISLBa1Lsc+bV7pg2P1S0MEhZK0lG8zSkBKzWLewGwB9Ayu+YKB/1+Slp42CyW3LI7Bp4n3pSU0BTVZvEraMU0lUwBFcSYhucmEGZEA6eCDQtBalhCaI90WMNSRSQzYTa+fogPrdLC7VjbUoDH6u+JjEhjBjKynZJA18x6I/E/r5FC+yzMuEpSYIpOFrVTgSHGoyhwi2tGQQwsIVRzeyumXaIJBRtYwYbgzb48T/zj8nnZvTkpVS6maeTRPjpAR8hDp6iCrlAV+YiiR/SMXtGb8+S8OO/Ox6Q150xndtEfOJ8/+J6VGg==</latexit> h# · hcost <latexit sha1_base64="Ojz3NR4aHMmkixGq35HRXv1L4zw=">AAAB/HicbVDLSsNAFJ34rPUVHzs3g0FwVRIR1F3RjcsKxhaaWCbTSTN0kgkzN0INxV9x40LFrR/izr9x+lho64ELh3Pu5d57olxwDa77bS0sLi2vrFbWqusbm1vb9s7unZaFosynUkjViohmgmfMBw6CtXLFSBoJ1oz6VyO/+cCU5jK7hUHOwpT0Mh5zSsBIHXs/uQ+cAAe0KwEnnZJKDcOO7bg1dww8T7wpcdAUjY79FXQlLVKWARVE67bn5hCWRAGngg2rQaFZTmif9Fjb0IykTIfl+PohPjJKF8dSmcoAj9XfEyVJtR6kkelMCSR61huJ/3ntAuLzsORZXgDL6GRRXAgMEo+iwF2uGAUxMIRQxc2tmCZEEQomsKoJwZt9eZ74J7WLmntz6tQvp2lU0AE6RMfIQ2eojq5RA/mIokf0jF7Rm/VkvVjv1sekdcGazuyhP7A+fwA7KpSl</latexit> <latexit sha1_base64="Ojz3NR4aHMmkixGq35HRXv1L4zw=">AAAB/HicbVDLSsNAFJ34rPUVHzs3g0FwVRIR1F3RjcsKxhaaWCbTSTN0kgkzN0INxV9x40LFrR/izr9x+lho64ELh3Pu5d57olxwDa77bS0sLi2vrFbWqusbm1vb9s7unZaFosynUkjViohmgmfMBw6CtXLFSBoJ1oz6VyO/+cCU5jK7hUHOwpT0Mh5zSsBIHXs/uQ+cAAe0KwEnnZJKDcOO7bg1dww8T7wpcdAUjY79FXQlLVKWARVE67bn5hCWRAGngg2rQaFZTmif9Fjb0IykTIfl+PohPjJKF8dSmcoAj9XfEyVJtR6kkelMCSR61huJ/3ntAuLzsORZXgDL6GRRXAgMEo+iwF2uGAUxMIRQxc2tmCZEEQomsKoJwZt9eZ74J7WLmntz6tQvp2lU0AE6RMfIQ2eojq5RA/mIokf0jF7Rm/VkvVjv1sekdcGazuyhP7A+fwA7KpSl</latexit> <latexit sha1_base64="Ojz3NR4aHMmkixGq35HRXv1L4zw=">AAAB/HicbVDLSsNAFJ34rPUVHzs3g0FwVRIR1F3RjcsKxhaaWCbTSTN0kgkzN0INxV9x40LFrR/izr9x+lho64ELh3Pu5d57olxwDa77bS0sLi2vrFbWqusbm1vb9s7unZaFosynUkjViohmgmfMBw6CtXLFSBoJ1oz6VyO/+cCU5jK7hUHOwpT0Mh5zSsBIHXs/uQ+cAAe0KwEnnZJKDcOO7bg1dww8T7wpcdAUjY79FXQlLVKWARVE67bn5hCWRAGngg2rQaFZTmif9Fjb0IykTIfl+PohPjJKF8dSmcoAj9XfEyVJtR6kkelMCSR61huJ/3ntAuLzsORZXgDL6GRRXAgMEo+iwF2uGAUxMIRQxc2tmCZEEQomsKoJwZt9eZ74J7WLmntz6tQvp2lU0AE6RMfIQ2eojq5RA/mIokf0jF7Rm/VkvVjv1sekdcGazuyhP7A+fwA7KpSl</latexit> + P h2H h cpu P i2I\itype=o o cpu P h2H h cpu <latexit sha1_base64="FC2LMoTMydIHirzI5Lws8bpZXkI=">AAACUXicdVFBSyMxFH4dXe3W1a3ucS8Py4IglqksqAeh6KV764K1QqcMmTRjg5lkSDJCCfMbBfHiD/Gyh91N2zms1n0Q+Pi+772XfElywY0Nw+dasLb+YWOz/rGx9Wl753Nzd+/aqEJTNqBKKH2TEMMEl2xguRXsJteMZIlgw+Tucq4P75k2XMkrO8vZOCO3kqecEuupuMkPMZqkmlAXmSKL3RQjLrFXopvGjuZFWeIRLiW+kH5gREmOPHbzcSWeo/JmVZnL/44p42YrbIeLwlXQqUALqurHzcdoomiRMWmpIMaMOmFux45oy6lgZSMqDMsJvSO3bOShJBkzY7eIpMRvnplgqrQ/0uKC/bfDkcyYWZZ4Z0bs1LzV5uR72qiw6enYcZkXlkm6XJQWAq3CeSA44ZpRK2YeEKq5vyvSKfEBW/8LDR9C5+2TV8HguH3WDn9+b3UvqjTq8BX24QA6cAJd6EEfBkDhAV7gN/ypPdV+BRAES2tQq3q+wKsKtv4C+FK0xA==</latexit> <latexit sha1_base64="FC2LMoTMydIHirzI5Lws8bpZXkI=">AAACUXicdVFBSyMxFH4dXe3W1a3ucS8Py4IglqksqAeh6KV764K1QqcMmTRjg5lkSDJCCfMbBfHiD/Gyh91N2zms1n0Q+Pi+772XfElywY0Nw+dasLb+YWOz/rGx9Wl753Nzd+/aqEJTNqBKKH2TEMMEl2xguRXsJteMZIlgw+Tucq4P75k2XMkrO8vZOCO3kqecEuupuMkPMZqkmlAXmSKL3RQjLrFXopvGjuZFWeIRLiW+kH5gREmOPHbzcSWeo/JmVZnL/44p42YrbIeLwlXQqUALqurHzcdoomiRMWmpIMaMOmFux45oy6lgZSMqDMsJvSO3bOShJBkzY7eIpMRvnplgqrQ/0uKC/bfDkcyYWZZ4Z0bs1LzV5uR72qiw6enYcZkXlkm6XJQWAq3CeSA44ZpRK2YeEKq5vyvSKfEBW/8LDR9C5+2TV8HguH3WDn9+b3UvqjTq8BX24QA6cAJd6EEfBkDhAV7gN/ypPdV+BRAES2tQq3q+wKsKtv4C+FK0xA==</latexit> <latexit sha1_base64="FC2LMoTMydIHirzI5Lws8bpZXkI=">AAACUXicdVFBSyMxFH4dXe3W1a3ucS8Py4IglqksqAeh6KV764K1QqcMmTRjg5lkSDJCCfMbBfHiD/Gyh91N2zms1n0Q+Pi+772XfElywY0Nw+dasLb+YWOz/rGx9Wl753Nzd+/aqEJTNqBKKH2TEMMEl2xguRXsJteMZIlgw+Tucq4P75k2XMkrO8vZOCO3kqecEuupuMkPMZqkmlAXmSKL3RQjLrFXopvGjuZFWeIRLiW+kH5gREmOPHbzcSWeo/JmVZnL/44p42YrbIeLwlXQqUALqurHzcdoomiRMWmpIMaMOmFux45oy6lgZSMqDMsJvSO3bOShJBkzY7eIpMRvnplgqrQ/0uKC/bfDkcyYWZZ4Z0bs1LzV5uR72qiw6enYcZkXlkm6XJQWAq3CeSA44ZpRK2YeEKq5vyvSKfEBW/8LDR9C5+2TV8HguH3WDn9+b3UvqjTq8BX24QA6cAJd6EEfBkDhAV7gN/ypPdV+BRAES2tQq3q+wKsKtv4C+FK0xA==</latexit> + P h2H h memory P i2I\itype=o o memory P h2H h memory <latexit sha1_base64="dQikx1vGD9GpzBE8abB6+iA0g7o=">AAACWnicfVHLSgMxFM2Mr7a+6mPn5mIRBLFMRVAXguim7hSsCp0yZNKMDeYxJBmhhPlJV7rwV8T0AT7xQuBwzrn3JidpzpmxUfQahDOzc/MLlWptcWl5ZbW+tn5rVKEJ7RDFlb5PsaGcSdqxzHJ6n2uKRcrpXfp4MdLvnqg2TMkbO8xpT+AHyTJGsPVUUtd7EPczjYmLTSESN4CYSWiX4AaJE1QoPSxL2IeJysbqJcQE58ASN5pYwiko71ef/vK/YWVSb0TNaFzwG7SmoIGmdZXUn+O+IoWg0hKOjem2otz2HNaWEU7LWlwYmmPyiB9o10OJBTU9N86mhB3P9CFT2h9pYcx+7XBYGDMUqXcKbAfmpzYi/9K6hc2Oe47JvLBUksmirOBgFYxigT7TlFg+9AATzfxdgQywT9r676j5EFo/n/wbdA6aJ83o+rBxdj5No4K20DbaRS10hM5QG12hDiLoBb0HC0EleAtnwmq4OLGGwbRnA32rcPMDTjG2FA==</latexit> <latexit sha1_base64="dQikx1vGD9GpzBE8abB6+iA0g7o=">AAACWnicfVHLSgMxFM2Mr7a+6mPn5mIRBLFMRVAXguim7hSsCp0yZNKMDeYxJBmhhPlJV7rwV8T0AT7xQuBwzrn3JidpzpmxUfQahDOzc/MLlWptcWl5ZbW+tn5rVKEJ7RDFlb5PsaGcSdqxzHJ6n2uKRcrpXfp4MdLvnqg2TMkbO8xpT+AHyTJGsPVUUtd7EPczjYmLTSESN4CYSWiX4AaJE1QoPSxL2IeJysbqJcQE58ASN5pYwiko71ef/vK/YWVSb0TNaFzwG7SmoIGmdZXUn+O+IoWg0hKOjem2otz2HNaWEU7LWlwYmmPyiB9o10OJBTU9N86mhB3P9CFT2h9pYcx+7XBYGDMUqXcKbAfmpzYi/9K6hc2Oe47JvLBUksmirOBgFYxigT7TlFg+9AATzfxdgQywT9r676j5EFo/n/wbdA6aJ83o+rBxdj5No4K20DbaRS10hM5QG12hDiLoBb0HC0EleAtnwmq4OLGGwbRnA32rcPMDTjG2FA==</latexit> <latexit sha1_base64="dQikx1vGD9GpzBE8abB6+iA0g7o=">AAACWnicfVHLSgMxFM2Mr7a+6mPn5mIRBLFMRVAXguim7hSsCp0yZNKMDeYxJBmhhPlJV7rwV8T0AT7xQuBwzrn3JidpzpmxUfQahDOzc/MLlWptcWl5ZbW+tn5rVKEJ7RDFlb5PsaGcSdqxzHJ6n2uKRcrpXfp4MdLvnqg2TMkbO8xpT+AHyTJGsPVUUtd7EPczjYmLTSESN4CYSWiX4AaJE1QoPSxL2IeJysbqJcQE58ASN5pYwiko71ef/vK/YWVSb0TNaFzwG7SmoIGmdZXUn+O+IoWg0hKOjem2otz2HNaWEU7LWlwYmmPyiB9o10OJBTU9N86mhB3P9CFT2h9pYcx+7XBYGDMUqXcKbAfmpzYi/9K6hc2Oe47JvLBUksmirOBgFYxigT7TlFg+9AATzfxdgQywT9r676j5EFo/n/wbdA6aJ83o+rBxdj5No4K20DbaRS10hM5QG12hDiLoBb0HC0EleAtnwmq4OLGGwbRnA32rcPMDTjG2FA==</latexit> + X v2V v · P <latexit sha1_base64="rWldUUTRw7lvvczrI32Ii0/gU20=">AAACAnicbVDLSgMxFM34rPVVdaebYBEEocyIoO6KblxWcNpCZyiZTKYNzWNIMoUyFNz4K25cqLj1K9z5N6btLLT1QOBwzrnc3BOljGrjut/O0vLK6tp6aaO8ubW9s1vZ229qmSlMfCyZVO0IacKoIL6hhpF2qgjiESOtaHA78VtDojSV4sGMUhJy1BM0oRgZK3Urh2cw0Bnv5kMYUAGbY2gJjqWBjW6l6tbcKeAi8QpSBQVs/iuIJc44EQYzpHXHc1MT5kgZihkZl4NMkxThAeqRjqUCcaLDfHrDGJ5YJYaJVPYJA6fq74kcca1HPLJJjkxfz3sT8T+vk5nkKsypSDNDBJ4tSjIGjYSTQmBMFcGGjSxBWFH7V4j7SCFsbG1lW4I3f/Ii8c9r1zX3/qJavynaKIEjcAxOgQcuQR3cgQbwAQaP4Bm8gjfnyXlx3p2PWXTJKWYOwB84nz9aA5ZA</latexit> <latexit sha1_base64="rWldUUTRw7lvvczrI32Ii0/gU20=">AAACAnicbVDLSgMxFM34rPVVdaebYBEEocyIoO6KblxWcNpCZyiZTKYNzWNIMoUyFNz4K25cqLj1K9z5N6btLLT1QOBwzrnc3BOljGrjut/O0vLK6tp6aaO8ubW9s1vZ229qmSlMfCyZVO0IacKoIL6hhpF2qgjiESOtaHA78VtDojSV4sGMUhJy1BM0oRgZK3Urh2cw0Bnv5kMYUAGbY2gJjqWBjW6l6tbcKeAi8QpSBQVs/iuIJc44EQYzpHXHc1MT5kgZihkZl4NMkxThAeqRjqUCcaLDfHrDGJ5YJYaJVPYJA6fq74kcca1HPLJJjkxfz3sT8T+vk5nkKsypSDNDBJ4tSjIGjYSTQmBMFcGGjSxBWFH7V4j7SCFsbG1lW4I3f/Ii8c9r1zX3/qJavynaKIEjcAxOgQcuQR3cgQbwAQaP4Bm8gjfnyXlx3p2PWXTJKWYOwB84nz9aA5ZA</latexit> <latexit sha1_base64="rWldUUTRw7lvvczrI32Ii0/gU20=">AAACAnicbVDLSgMxFM34rPVVdaebYBEEocyIoO6KblxWcNpCZyiZTKYNzWNIMoUyFNz4K25cqLj1K9z5N6btLLT1QOBwzrnc3BOljGrjut/O0vLK6tp6aaO8ubW9s1vZ229qmSlMfCyZVO0IacKoIL6hhpF2qgjiESOtaHA78VtDojSV4sGMUhJy1BM0oRgZK3Urh2cw0Bnv5kMYUAGbY2gJjqWBjW6l6tbcKeAi8QpSBQVs/iuIJc44EQYzpHXHc1MT5kgZihkZl4NMkxThAeqRjqUCcaLDfHrDGJ5YJYaJVPYJA6fq74kcca1HPLJJjkxfz3sT8T+vk5nkKsypSDNDBJ4tSjIGjYSTQmBMFcGGjSxBWFH7V4j7SCFsbG1lW4I3f/Ii8c9r1zX3/qJavynaKIEjcAxOgQcuQR3cgQbwAQaP4Bm8gjfnyXlx3p2PWXTJKWYOwB84nz9aA5ZA</latexit> 58
59. BTU-based Approach ‣ Scale up, when operators require more resources based on thresholds ‣ Only scale down at the end of a BTU to keep already paid processing capabilities for future data spikes ‣ Use Docker container to employ fine grained resource provisioning 59
60. Downscaling Procedure BTU of VM ends Create a down-scaling plan Release operators Migrate operators Release VM Lease VM for another BTU 60
61. Evaluation Setup ‣ Threshold-based approach serves as the baseline ‣ BTUs: 10 min, 30 min, 60 min Spain United Kindom Germany O1 O2 O8 O9 O3 O4 O5 O6 O7 Sweden O1 O2 O1 O2 O3 O4 O5 O6 O7 O3 O4 O5 O6 O7 O1 O2 S1 S2 S3 S1 S2 S3 S1 S2 S3 S1 S2 S3 61
62. Data Arrival Pattern 0 20 40 60 80 100 120 0 2 4 6 8 Stepwise Pattern Time (Minutes) Manufacturing Machines 0 20 40 60 80 100 120 0 2 4 6 8 2−Level Pattern Time (Minutes) Manufacturing Machines 0 20 40 60 80 100 120 0 2 4 6 8 Random Walk Pattern 1 Time (Minutes) Manufacturing Machines 0 20 40 60 80 100 120 0 2 4 6 8 Random Walk Pattern 2 Time (Minutes) Manufacturing Machines 62
63. Evaluation Results - Threshold-based Baseline 0 20 40 60 80 100 120 0 2 4 6 8 Time (Minutes) Amount of Manufacturing Machines and VMs 0 10 20 30 40 50 60 70 Amount of Operator Instances 0 0 Pattern VMs Operator Instances 63
64. Evaluation Results - BTU 30 0 0 Pattern VMs Operator Instances 0 20 40 60 80 100 120 0 2 4 6 8 Time (Minutes) Amount of Manufacturing Machines and VMs 0 10 20 30 40 50 60 70 Amount of Operator Instances 64
65. Evaluation Metrics Compliance Metrics ‣ Real-time: average data processing time without any load ‣ Near real-time: 2 x the realtime SLO ‣ Relaxed: 5 x the realtime SLO Cost ‣ Resource cost: cost for leasing a VM for 10 min amounts for 1 cost unit ‣ Penalty cost: delaying 10000 data items amounts for 1 cost unit ‣ Total cost = Resource cost + penalty cost 65
66. Evaluation Results - Compliance Threshold- based BTU 10 BTU 30 BTU 60 Real-time Compliance 40 % 48 % 50 % 55 % Near real- time Compliance 68 % 84 % 88 % 93 % Relaxed Compliance 72 % 88 % 91 % 95 % on the results for the other —> mention here the results m walk 1 and rw 2 66
67. Threshold-based BTU-based BTU 10 BTU 30 BTU 60 BTU 10 BTU 30 BTU 60 Resource Cost 66 86 122 82 96 104 Real-time Compliance 157 178 214 169 178 176 Near real-time Compliance 115 135 171 108 155 115 Relaxed Compliance 108 128 165 102 111 112 Evaluation Results - Cost 67
68. Contribution Summary ‣ Introduced, implemented and evaluated a threshold-based provisioning approach ‣ Analyzed the limitations of threshold-based provisioning approaches ‣ Proposed a lazy downscaling approach to improve the SLA compliance, reduce the management overhead and achieve a cost reduction of up to 36% 68
69. Conclusion 69
70. Research Questions Revisited RQ1 How can geographically distributed data streams be processed efficiently? C1 A system design for a holistic stream processing ecosystem in distributed environments 70
71. Research Questions Revisited RQ2 How can SPAs be described to allow a distributed deployment model and respect service level objectives? C2 An approach to define quality of service- and location-specific aspects for SPAs 71
72. Research Questions Revisited C3 Cost-efficient resource provisioning approaches for SPAs RQ3 How can SPAs be optimally executed on computational resources? 72
73. Future Work ‣Optimal deployment of operators across different geographic locations ‣Development of suitable monitoring metrics ‣Complex failure compensation strategies for SPAs ‣Data privacy and data ownership 73
74. Final Notes Publications ‣Journal 4 / 7 (FGCS, IC, PeerJ, TSC … ) ‣Conference, Workshops 9 / 18 (CLOUD, EDOC, SOSE, UCC …) ‣Technical Report, Proceedings 1 / 3 Teaching ‣Advanced Internet Computing ‣Co-Supervision: 2 bachelor theses, 3 master theses, 2 projects in SE & IC 74
75. Acknowledgements ‣Prof. Dr. Schahram Dustdar and Assistant. Prof. Dr.-Ing. Stefan Schulte for guiding me through my PhD ‣Prof. Dr. Harald Gall and Prof. Dr. Gertrude Kappel for serving as reviewers ‣All current and previous members of the DSG for insightful discussions FP7 SIMPLICITY (318201) H2020 CREMA (637066) Innovative Project 75
76. Discussion 76
77. References [1] Akpınar, K., Hua, K. A., & Li, K. ThingStore: a platform for internet-of-things application development and deployment. DEBS 2015.
 [2] Cardellini, V., Grassi, V., Lo Presti, F., & Nardelli, M. Distributed QoS-aware scheduling in Storm. DEBS 2015.
 [3] Gulisano, V., Jimenez-Peris, R., Patino-Martinez, M., Soriente, C., & Valduriez, P. Streamcloud: An elastic and scalable data streaming system. Transactions on Parallel and Distributed Systems, 2012
 [4] Lim, H., Han, Y., & Babu, S. How to Fit when No One Size Fits. CIDR, 2013
 [5] Schneider, S., Andrade, H., Gedik, B., Biem, A., & Wu, K. L. Elastic scaling of data parallel operators in stream processing. IPDPS 2009.
 [6] Gedik, B., Schneider, S., Hirzel, M., & Wu, K. L. Elastic scaling for data stream processing. Transactions on Parallel and Distributed Systems, 2014
 [7] Aniello, L., Baldoni, R., & Querzoni, L. Adaptive online scheduling in storm. DEBS 2013
 [8] Xu, J., Chen, Z., Tang, J., & Su, S. T-storm: Traffic-aware online scheduling in storm. ICDCS 2014
 [9] Xu, L., Peng, B., & Gupta, I. Stela: Enabling stream processing systems to scale-in and scale-out on-demand. IC2E 2016 
 [10] Heinze, T., Roediger, L., Meister, A., Ji, Y., Jerzak, Z., & Fetzer, C. Online parameter optimization for elastic data stream processing. SCC 2015
 [11] Castro Fernandez, R., Migliavacca, M., Kalyvianaki, E., & Pietzuch, P. Integrating scale out and fault tolerance in stream processing using operator state management. SIGMOD 2013 77
78. Backup Slides 78
79. Methodology Based on the design science methodology ‣Identify a challenge ‣Conduct a literature survey ‣Define a research hypotheses and performance indicators ‣Design and implementation of a novel algorithm or system design ‣Evaluation of the research hypotheses based on the research prototype 79
80. Contribution 1 80
81. System Design Infrastructure Data Sources VISP Marketplace Cloud Resources Edge Resources VISP Runtime Stream Processing Application Topology Configuration Operator Replication Infrastructure Management Operator Provisioning Messaging Operator Migration Operator Placement Shared Data Failure Detection Privacy Security Topology Enactment Manufacturing Machines Wearable Devices News Portals Failure Compensation Monitoring Monetization Operator Distribution Topology Builder Operator Registry Runtime Synchronization Stock Market SLA Compliance Operational Layer Communication Layer Infrastructure Layer Cross-cutting Concerns 81
82. Screenshot of the Web-based UI 82
83. Distributed Deployment of VISP Runtimes VISP Runtime Data Source 1 Data Source N • • • VISP Runtime VISP Runtime • • • Topology Configuration Operator Replication Infrastructure Management Monitoring Messaging Computational Resources Virtual Machine 1 Operator 1 Operator N • • • Virtual Machine N Operator 1 Operator N • • • • • • Shared Data Operator Image 1 Operator Image N • • • Operator Repository VISP Runtime VISP Runtime • • • R R 83
84. Contribution 2 84
85. Features of the VTDL Attribute Values Default Value Description type string — Operator logic allowedLocations IP+ — Possible locations poolPreferences string+ general Hardware preferences concreteLocation IP/poolIdentifier* — Preselected location inputFormat string+ — Accepted inputs outputFormat string — Data output format responseTime numerical 2s Response time queueLength numerical 100 Buffered items stateful { true | false } true Operator statefulness replicationAllowed { true | false } false Operator replication protocol {stream | microbatch: <X>items/<X>ms} stream Processing mode compensation { redeploySingle | redeployTopology | deploy<URL} | mailto:<email> | none } none Failure recovery mode 85
86. Topology Update Procedure (S13) Continue processing (S10) Update message infrastructure (S7) Distribute update operations (S6) Initiate update Processing in place / No topology in place No compensation required for failure Processing stopped Compensation required for failure Managing VISP Runtime All involved VISP Runtimes All involved VISP Runtimes synchronized by managing VISP Runtime (S1) Receive new topology (S2) Check integrity for new topology (S3) Topology grounding (S4) Distribute new topology (S5) Check that no update is in place (S8) Validate resource availability (S9) Stop processing of involved operators (S11) Apply operator updates (S12) Check integrity 86
87. Evaluation VTDL Approach Baseline Approach duration (s) interactions duration (s) interactions 1. New topology in one location 54.98 3 54.98 3 2. New topology across four locations 40.95 3 68.10 12 3. Network disruption 7.53 0 — — 4. Load reduction 3.30 3 54.05 8 5. Bugfix for an operator 5.69 3 35.62 4 87
88. Contribution 3 88
89. Deployment of SPAs O5 O4 O3 O2 O1 O6 O9 O7 O8 O5 O4 O3 O2 O1 O6 O9 O7 O8 O2 O1 O6 O5 O4 O3 O2 O1 O6 O9 O7 O8 O2 O1 O6 O1 O1 O3 O5 O4 O6 O9 O7 O8 O2 O1 O6 O1 O2 O1 O3 O5 O4 H1 H2 H1 H2 H3 H1 H2 H3 H4 H1 H2 H3 H4 Stage 1 Stage 2 Stage 3 Stage 4 Data Volume over Time Amount of Manufacturing Machines 89
90. Elastic Resource Provisioning Model 1 min X p2P pi + X p2P piBT U + u · N + d · N <latexit sha1_base64="gPDVtNvhSDVp6sXC9eN/+3LxsS0=">AAACPnicbZBNSwMxEIazflu/qh69DBZBEMpWBPVW9OJJKrgqdMuSzaYaTLJLMiuWZf+ZF/+CN69ePKh49Whaq/g1EHh43xkm88aZFBZ9/94bGR0bn5icmq7MzM7NL1QXl05smhvGA5bK1JzF1HIpNA9QoORnmeFUxZKfxpf7ff/0ihsrUn2MvYx3FD3XoisYRSdF1ROAEPk1GlUooUsIba6iIoNQaGiVkEUCNj7FL60QUbF3HJSl83IIWZIiHDpOPrkSVWt+3R8U/IXGEGpkWK2oehcmKcsV18gktbbd8DPsFNSgYJKXlTC3PKPskp7ztkNNFbedYnB/CWtOSaCbGvc0wkD9PlFQZW1Pxa5TUbywv72++J/XzrG70ymEznLkmn0s6uYSMIV+mJAIwxnKngPKjHB/BXZBDWXoIu+H0Ph98l8INuu7df9oq9bcG6YxRVbIKlknDbJNmuSAtEhAGLkhD+SJPHu33qP34r1+tI54w5ll8qO8t3eoMa0C</latexit> <latexit sha1_base64="gPDVtNvhSDVp6sXC9eN/+3LxsS0=">AAACPnicbZBNSwMxEIazflu/qh69DBZBEMpWBPVW9OJJKrgqdMuSzaYaTLJLMiuWZf+ZF/+CN69ePKh49Whaq/g1EHh43xkm88aZFBZ9/94bGR0bn5icmq7MzM7NL1QXl05smhvGA5bK1JzF1HIpNA9QoORnmeFUxZKfxpf7ff/0ihsrUn2MvYx3FD3XoisYRSdF1ROAEPk1GlUooUsIba6iIoNQaGiVkEUCNj7FL60QUbF3HJSl83IIWZIiHDpOPrkSVWt+3R8U/IXGEGpkWK2oehcmKcsV18gktbbd8DPsFNSgYJKXlTC3PKPskp7ztkNNFbedYnB/CWtOSaCbGvc0wkD9PlFQZW1Pxa5TUbywv72++J/XzrG70ymEznLkmn0s6uYSMIV+mJAIwxnKngPKjHB/BXZBDWXoIu+H0Ph98l8INuu7df9oq9bcG6YxRVbIKlknDbJNmuSAtEhAGLkhD+SJPHu33qP34r1+tI54w5ll8qO8t3eoMa0C</latexit> <latexit sha1_base64="gPDVtNvhSDVp6sXC9eN/+3LxsS0=">AAACPnicbZBNSwMxEIazflu/qh69DBZBEMpWBPVW9OJJKrgqdMuSzaYaTLJLMiuWZf+ZF/+CN69ePKh49Whaq/g1EHh43xkm88aZFBZ9/94bGR0bn5icmq7MzM7NL1QXl05smhvGA5bK1JzF1HIpNA9QoORnmeFUxZKfxpf7ff/0ihsrUn2MvYx3FD3XoisYRSdF1ROAEPk1GlUooUsIba6iIoNQaGiVkEUCNj7FL60QUbF3HJSl83IIWZIiHDpOPrkSVWt+3R8U/IXGEGpkWK2oehcmKcsV18gktbbd8DPsFNSgYJKXlTC3PKPskp7ztkNNFbedYnB/CWtOSaCbGvc0wkD9PlFQZW1Pxa5TUbywv72++J/XzrG70ymEznLkmn0s6uYSMIV+mJAIwxnKngPKjHB/BXZBDWXoIu+H0Ph98l8INuu7df9oq9bcG6YxRVbIKlknDbJNmuSAtEhAGLkhD+SJPHu33qP34r1+tI54w5ll8qO8t3eoMa0C</latexit> pi <latexit sha1_base64="p6RESMUprFwWDqzMCniPeYau4Sw=">AAAB7HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqN6KXjxWMG2hDWWz3bRrN7thdyOU0P/gxYOKV3+QN/+NmzYHbX0w8Hhvhpl5YcKZNq777ZTW1jc2t8rblZ3dvf2D6uFRW8tUEeoTyaXqhlhTzgT1DTOcdhNFcRxy2gknt7nfeaJKMykezDShQYxHgkWMYGOlNkoGDFUG1Zpbd+dAq8QrSA0KtAbVr/5QkjSmwhCOte55bmKCDCvDCKezSj/VNMFkgke0Z6nAMdVBNr92hs6sMkSRVLaEQXP190SGY62ncWg7Y2zGetnLxf+8XmqiqyBjIkkNFWSxKEo5MhLlr6MhU5QYPrUEE8XsrYiMscLE2IDyELzll1eJf1G/rrv3l7XmTZFGGU7gFM7BgwY04Q5a4AOBR3iGV3hzpPPivDsfi9aSU8wcwx84nz+je44L</latexit> <latexit sha1_base64="p6RESMUprFwWDqzMCniPeYau4Sw=">AAAB7HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqN6KXjxWMG2hDWWz3bRrN7thdyOU0P/gxYOKV3+QN/+NmzYHbX0w8Hhvhpl5YcKZNq777ZTW1jc2t8rblZ3dvf2D6uFRW8tUEeoTyaXqhlhTzgT1DTOcdhNFcRxy2gknt7nfeaJKMykezDShQYxHgkWMYGOlNkoGDFUG1Zpbd+dAq8QrSA0KtAbVr/5QkjSmwhCOte55bmKCDCvDCKezSj/VNMFkgke0Z6nAMdVBNr92hs6sMkSRVLaEQXP190SGY62ncWg7Y2zGetnLxf+8XmqiqyBjIkkNFWSxKEo5MhLlr6MhU5QYPrUEE8XsrYiMscLE2IDyELzll1eJf1G/rrv3l7XmTZFGGU7gFM7BgwY04Q5a4AOBR3iGV3hzpPPivDsfi9aSU8wcwx84nz+je44L</latexit> <latexit sha1_base64="p6RESMUprFwWDqzMCniPeYau4Sw=">AAAB7HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqN6KXjxWMG2hDWWz3bRrN7thdyOU0P/gxYOKV3+QN/+NmzYHbX0w8Hhvhpl5YcKZNq777ZTW1jc2t8rblZ3dvf2D6uFRW8tUEeoTyaXqhlhTzgT1DTOcdhNFcRxy2gknt7nfeaJKMykezDShQYxHgkWMYGOlNkoGDFUG1Zpbd+dAq8QrSA0KtAbVr/5QkjSmwhCOte55bmKCDCvDCKezSj/VNMFkgke0Z6nAMdVBNr92hs6sMkSRVLaEQXP190SGY62ncWg7Y2zGetnLxf+8XmqiqyBjIkkNFWSxKEo5MhLlr6MhU5QYPrUEE8XsrYiMscLE2IDyELzll1eJf1G/rrv3l7XmTZFGGU7gFM7BgwY04Q5a4AOBR3iGV3hzpPPivDsfi9aSU8wcwx84nz+je44L</latexit> piBT U <latexit sha1_base64="FmGu3g7XaE19MK2k4P833H+fNoM=">AAAB8XicbVBNS8NAEN3Ur1q/qh69LBbBU0lEUG+lXjxWaGyhDWGz3bRLN7thdyKUkJ/hxYOKV/+NN/+N2zYHrT4YeLw3w8y8KBXcgOt+OZW19Y3Nrep2bWd3b/+gfnj0YFSmKfOpEkr3I2KY4JL5wEGwfqoZSSLBetH0du73Hpk2XMkuzFIWJGQsecwpASsN0jDnYd7u+kUR1htu010A/yVeSRqoRCesfw5HimYJk0AFMWbguSkEOdHAqWBFbZgZlhI6JWM2sFSShJkgX5xc4DOrjHCstC0JeKH+nMhJYswsiWxnQmBiVr25+J83yCC+DnIu0wyYpMtFcSYwKDz/H4+4ZhTEzBJCNbe3YjohmlCwKdVsCN7qy3+Jf9G8abr3l41Wu0yjik7QKTpHHrpCLXSHOshHFCn0hF7QqwPOs/PmvC9bK045c4x+wfn4Bt8WkS0=</latexit> <latexit sha1_base64="FmGu3g7XaE19MK2k4P833H+fNoM=">AAAB8XicbVBNS8NAEN3Ur1q/qh69LBbBU0lEUG+lXjxWaGyhDWGz3bRLN7thdyKUkJ/hxYOKV/+NN/+N2zYHrT4YeLw3w8y8KBXcgOt+OZW19Y3Nrep2bWd3b/+gfnj0YFSmKfOpEkr3I2KY4JL5wEGwfqoZSSLBetH0du73Hpk2XMkuzFIWJGQsecwpASsN0jDnYd7u+kUR1htu010A/yVeSRqoRCesfw5HimYJk0AFMWbguSkEOdHAqWBFbZgZlhI6JWM2sFSShJkgX5xc4DOrjHCstC0JeKH+nMhJYswsiWxnQmBiVr25+J83yCC+DnIu0wyYpMtFcSYwKDz/H4+4ZhTEzBJCNbe3YjohmlCwKdVsCN7qy3+Jf9G8abr3l41Wu0yjik7QKTpHHrpCLXSHOshHFCn0hF7QqwPOs/PmvC9bK045c4x+wfn4Bt8WkS0=</latexit> <latexit sha1_base64="FmGu3g7XaE19MK2k4P833H+fNoM=">AAAB8XicbVBNS8NAEN3Ur1q/qh69LBbBU0lEUG+lXjxWaGyhDWGz3bRLN7thdyKUkJ/hxYOKV/+NN/+N2zYHrT4YeLw3w8y8KBXcgOt+OZW19Y3Nrep2bWd3b/+gfnj0YFSmKfOpEkr3I2KY4JL5wEGwfqoZSSLBetH0du73Hpk2XMkuzFIWJGQsecwpASsN0jDnYd7u+kUR1htu010A/yVeSRqoRCesfw5HimYJk0AFMWbguSkEOdHAqWBFbZgZlhI6JWM2sFSShJkgX5xc4DOrjHCstC0JeKH+nMhJYswsiWxnQmBiVr25+J83yCC+DnIu0wyYpMtFcSYwKDz/H4+4ZhTEzBJCNbe3YjohmlCwKdVsCN7qy3+Jf9G8abr3l41Wu0yjik7QKTpHHrpCLXSHOshHFCn0hF7QqwPOs/PmvC9bK045c4x+wfn4Bt8WkS0=</latexit> u <latexit sha1_base64="kF4y8yL2rgRNwd6KnxzHbkHreI8=">AAAB53icbVBNS8NAEJ34WetX1aOXxSJ4KokI6q3oxWMLxhbaUDbbSbt2swm7G6GE/gIvHlS8+pe8+W/ctjlo64OBx3szzMwLU8G1cd1vZ2V1bX1js7RV3t7Z3duvHBw+6CRTDH2WiES1Q6pRcIm+4UZgO1VI41BgKxzdTv3WEyrNE3lvxikGMR1IHnFGjZWaWa9SdWvuDGSZeAWpQoFGr/LV7Scsi1EaJqjWHc9NTZBTZTgTOCl3M40pZSM6wI6lksaog3x26IScWqVPokTZkobM1N8TOY21Hseh7YypGepFbyr+53UyE10FOZdpZlCy+aIoE8QkZPo16XOFzIixJZQpbm8lbEgVZcZmU7YheIsvLxP/vHZdc5sX1fpNkUYJjuEEzsCDS6jDHTTABwYIz/AKb86j8+K8Ox/z1hWnmDmCP3A+fwBPxIzM</latexit> <latexit sha1_base64="kF4y8yL2rgRNwd6KnxzHbkHreI8=">AAAB53icbVBNS8NAEJ34WetX1aOXxSJ4KokI6q3oxWMLxhbaUDbbSbt2swm7G6GE/gIvHlS8+pe8+W/ctjlo64OBx3szzMwLU8G1cd1vZ2V1bX1js7RV3t7Z3duvHBw+6CRTDH2WiES1Q6pRcIm+4UZgO1VI41BgKxzdTv3WEyrNE3lvxikGMR1IHnFGjZWaWa9SdWvuDGSZeAWpQoFGr/LV7Scsi1EaJqjWHc9NTZBTZTgTOCl3M40pZSM6wI6lksaog3x26IScWqVPokTZkobM1N8TOY21Hseh7YypGepFbyr+53UyE10FOZdpZlCy+aIoE8QkZPo16XOFzIixJZQpbm8lbEgVZcZmU7YheIsvLxP/vHZdc5sX1fpNkUYJjuEEzsCDS6jDHTTABwYIz/AKb86j8+K8Ox/z1hWnmDmCP3A+fwBPxIzM</latexit> <latexit sha1_base64="kF4y8yL2rgRNwd6KnxzHbkHreI8=">AAAB53icbVBNS8NAEJ34WetX1aOXxSJ4KokI6q3oxWMLxhbaUDbbSbt2swm7G6GE/gIvHlS8+pe8+W/ctjlo64OBx3szzMwLU8G1cd1vZ2V1bX1js7RV3t7Z3duvHBw+6CRTDH2WiES1Q6pRcIm+4UZgO1VI41BgKxzdTv3WEyrNE3lvxikGMR1IHnFGjZWaWa9SdWvuDGSZeAWpQoFGr/LV7Scsi1EaJqjWHc9NTZBTZTgTOCl3M40pZSM6wI6lksaog3x26IScWqVPokTZkobM1N8TOY21Hseh7YypGepFbyr+53UyE10FOZdpZlCy+aIoE8QkZPo16XOFzIixJZQpbm8lbEgVZcZmU7YheIsvLxP/vHZdc5sX1fpNkUYJjuEEzsCDS6jDHTTABwYIz/AKb86j8+K8Ox/z1hWnmDmCP3A+fwBPxIzM</latexit> d <latexit sha1_base64="493iywG9k3VTthaAR+vadVA7SLQ=">AAAB53icbVBNS8NAEJ34WetX1aOXxSJ4KokI6q3oxWMLxhbaUDabSbt2swm7G6GU/gIvHlS8+pe8+W/ctjlo64OBx3szzMwLM8G1cd1vZ2V1bX1js7RV3t7Z3duvHBw+6DRXDH2WilS1Q6pRcIm+4UZgO1NIk1BgKxzeTv3WEyrNU3lvRhkGCe1LHnNGjZWaUa9SdWvuDGSZeAWpQoFGr/LVjVKWJygNE1TrjudmJhhTZTgTOCl3c40ZZUPax46lkiaog/Hs0Ak5tUpE4lTZkobM1N8TY5poPUpC25lQM9CL3lT8z+vkJr4KxlxmuUHJ5oviXBCTkunXJOIKmREjSyhT3N5K2IAqyozNpmxD8BZfXib+ee265jYvqvWbIo0SHMMJnIEHl1CHO2iADwwQnuEV3pxH58V5dz7mrStOMXMEf+B8/gA2EYy7</latexit> <latexit sha1_base64="493iywG9k3VTthaAR+vadVA7SLQ=">AAAB53icbVBNS8NAEJ34WetX1aOXxSJ4KokI6q3oxWMLxhbaUDabSbt2swm7G6GU/gIvHlS8+pe8+W/ctjlo64OBx3szzMwLM8G1cd1vZ2V1bX1js7RV3t7Z3duvHBw+6DRXDH2WilS1Q6pRcIm+4UZgO1NIk1BgKxzeTv3WEyrNU3lvRhkGCe1LHnNGjZWaUa9SdWvuDGSZeAWpQoFGr/LVjVKWJygNE1TrjudmJhhTZTgTOCl3c40ZZUPax46lkiaog/Hs0Ak5tUpE4lTZkobM1N8TY5poPUpC25lQM9CL3lT8z+vkJr4KxlxmuUHJ5oviXBCTkunXJOIKmREjSyhT3N5K2IAqyozNpmxD8BZfXib+ee265jYvqvWbIo0SHMMJnIEHl1CHO2iADwwQnuEV3pxH58V5dz7mrStOMXMEf+B8/gA2EYy7</latexit> <latexit sha1_base64="493iywG9k3VTthaAR+vadVA7SLQ=">AAAB53icbVBNS8NAEJ34WetX1aOXxSJ4KokI6q3oxWMLxhbaUDabSbt2swm7G6GU/gIvHlS8+pe8+W/ctjlo64OBx3szzMwLM8G1cd1vZ2V1bX1js7RV3t7Z3duvHBw+6DRXDH2WilS1Q6pRcIm+4UZgO1NIk1BgKxzeTv3WEyrNU3lvRhkGCe1LHnNGjZWaUa9SdWvuDGSZeAWpQoFGr/LVjVKWJygNE1TrjudmJhhTZTgTOCl3c40ZZUPax46lkiaog/Hs0Ak5tUpE4lTZkobM1N8TY5poPUpC25lQM9CL3lT8z+vkJr4KxlxmuUHJ5oviXBCTkunXJOIKmREjSyhT3N5K2IAqyozNpmxD8BZfXib+ee265jYvqvWbIo0SHMMJnIEHl1CHO2iADwwQnuEV3pxH58V5dz7mrStOMXMEf+B8/gA2EYy7</latexit> Concrete Operator Instance Remaining BTU for the VM of the concrete Operator Instance Upscaling decision variable Downscaling decision variable 90
91. Resource Setup Under-Provisioning Over-Provisioning Speed 5 8 Average Speed 6 10 Aggregation 2 3 Distance 1 1 Analysis 1 1 Monitor 1 1 91
92. Threshold-based Evaluation Elastic Provisioning Over-Provisioning Under-Provisioning Number of Total Rides 75 75 75 Number of Location Information Items 50742 50742 50742 Cost for Operator Instances 2160.66 2664 1856 (σ = 13.61) (σ = 0.00) (σ = 0.00) Total Makespan in Seconds 6653 6655 6975 (σ = 9.60) (σ = 0.00) (σ = 1.00) Average Duration for the 77 35 355 Report Generation in Seconds (σ = 10.69) (σ = 0.00) (σ = 0.57) Total Number of Delays 21 0 75 (σ = 5.29) (σ = 0.00) (σ = 0.00) SLA Adherence in % 28 % 100 % 0 % (σ = 7.40) (σ = 0.00) (σ= 0.00) 92
93. Upscaling Procedure Upscaling for a specific operator triggered Enough resources available Not enough resources available Best host selected Not enough resources available Assess utility for hosts Try scaledown for other operator Scaledown successful Scaledown not successful Lease another host Select best host Sufficient processing capabilities for specific operator Add instance for a specific operator 93
94. Downscaling Procedure BTU of host ends Downscaling plan exists Host leased for another BTU Still running Operator Instances Still running Operator Instances No running Operator Instances Host released Simulate operator downscaling and migration Select best host Prolong leasing Simulation successful Try scaledown of Operator Instances Scaledown not successful Scaledown successful Try migration of Operator Instances Migration is not successful Migration is successful Release host 94
95. Upscaling Algorithm 95
96. VM Selection Algorithm 96
97. Operator Selection Algorithm 97
98. Sensor Configuration Emission Rate / min Size (Bytes) Availability Sensor (S1) 5 95 Production Data (S2) 1 12500 Temperature Sensor (S33) 10 90 98
99. Related Work on Resource Elasticity ‣Resource elasticity for monolithic SPAs Lim et al.[4] ‣Resource elasticity for individual operators Schneider et al. [5][6] ‣Placement of operators Aneillo et al. [7], Xu et al. [8] ‣Minimization of reconfigurations for the SPE Xu et al. [9] ‣Threshold-based approaches ‣Update of thresholds at runtime Heinze et al. [10] ‣Consideration of stateful operators Castro Fernandez et al. [11] 99
100. Operator Configuration Processing Duration (ms) CPU Shares Memory (MB) Storage (MB) State Outgoing Ratio O1 600 131 524 68 x 50:1 O2 600 65 440 68 x 100:1 O3 1500 660 452 89 1:3 O4 750 100 430 68 1:1 O5 750 83 502 68 x 1:1 O6 750 77 527 68 x 1:1 O7 700 46 464 68 x 3:1 O8 1300 47 452 70 x 300:1 O9 500 74 466 68 1:0 100
101. Evaluation Results - Compliance (2-level) Threshold BTU 10 BTU 30 BTU 60 Real-time Compliance 40 % (σ = 2%) 48 % (σ = 1%) 50 % (σ = 1%) 55 % (σ = 1%) Near real- time Compliance 68 % (σ = 2%) 84 % (σ = 2%) 88 % (σ = 0%) 93 % (σ = 1%) Relaxed Compliance 72 % (σ = 2%) 88 % (σ = 2%) 91 % (σ = 0%) 95 % (σ = 1%) 101
102. Evaluation Results - Compliance (Stepwise) Threshold BTU 10 BTU 30 BTU 60 Real-time Compliance 40 % (σ = 1%) 49 % (σ = 1%) 52 % (σ = 1%) 53 % (σ = 1%) Near real- time Compliance 67 % (σ = 2%) 85 % (σ = 2%) 90 % (σ = 1%) 93 % (σ = 1%) Relaxed Compliance 71 % (σ = 1%) 89 % (σ = 1%) 93 % (σ = 1%) 95 % (σ = 1%) 102
103. Evaluation Results - Compliance (Random Walk 1) Threshold BTU 10 BTU 30 BTU 60 Real-time Compliance 39 % (σ = 0%) 49 % (σ = 1%) 52 % (σ = 2%) 54 % (σ = 0%) Near real- time Compliance 66 % (σ = 1%) 85 % (σ = 2%) 90 % (σ = 3%) 93 % (σ = 0%) Relaxed Compliance 71 % (σ = 1%) 89 % (σ = 2%) 93 % (σ = 2%) 95 % (σ = 1%) 103
104. Evaluation Results - Compliance (Random Walk 2) Threshold BTU 10 BTU 30 BTU 60 Real-time Compliance 41 % (σ = 0%) 49 % (σ = 2%) 51 % (σ = 1%) 53 % (σ = 0%) Near real- time Compliance 70 % (σ = 1%) 87 % (σ = 2%) 90 % (σ = 1%) 92 % (σ = 1%) Relaxed Compliance 75 % (σ = 1%) 90 % (σ = 2%) 94 % (σ = 1%) 95 % (σ = 0%) 104
105. Evaluation Results - Cost (2-level) Threshold-based BTU-based BTU 10 BTU 30 BTU 60 BTU 10 BTU 30 BTU 60 Resource Cost 66 (σ = 0.00) 86 (σ = 1.73) 122 (σ = 3.46) 82 (σ = 5.13) 96 (σ = 7.94) 104 (σ = 6.93) Real-time Compliance 157 (σ = 2.64) 178 (σ = 4.17) 214 (σ = 4.16) 169 (σ = 6.83) 178 (σ = 6.82) 176 (σ = 4.77) Near real-time Compliance 115 (σ = 3.21) 135 (σ = 4.19) 171 (σ = 2.94) 108 (σ = 6.74) 155 (σ = 8.18) 115 (σ = 6.28) Relaxed Compliance 108 (σ = 2.40) 128 (σ = 3.37) 165 (σ = 2.59) 102 (σ = 6.74) 111 (σ = 8.18) 112 (σ = 6.28) 105
106. Evaluation Results - Cost (Stepwise) Threshold-based BTU-based BTU 10 BTU 30 BTU 60 BTU 10 BTU 30 BTU 60 Resource Cost 58 (σ = 1.73) 79 (σ = 4.58) 120 (σ = 6.00) 72 (σ = 3.79) 92 (σ = 1.73) 98 (σ = 3.84) Real-time Compliance 152 (σ = 1.95) 173 (σ = 4.93) 214 (σ = 6.45) 159 (σ = 0.82) 173 (σ = 0.68) 175 (σ = 4.25) Near real-time Compliance 110 (σ = 1.77) 131 (σ = 4.73) 172 (σ = 6.35) 97 (σ = 0.39) 108 (σ = 0.50) 109 (σ = 3.17) Relaxed Compliance 103 (σ = 1.57) 124 (σ = 1.57) 165 (σ = 6.12) 91 (σ = 1.84) 103 (σ = 1.04) 105 (σ = 2.91) 106
107. Evaluation Results - Cost (Random Walk 1) Threshold-based BTU-based BTU 10 BTU 30 BTU 60 BTU 10 BTU 30 BTU 60 Resource Cost 61 (σ = 1.53) 86 (σ = 1.73) 128 (σ = 6.93) 69 (σ = 3.51) 95 (σ = 1.73) 110 (σ = 3.46) Real-time Compliance 159 (σ = 1.54) 183 (σ = 1.40) 225 (σ = 6.56) 158 (σ = 4.99) 177 (σ = 4.70) 186 (σ = 3.40) Near real-time Compliance 116 (σ = 1.14) 140 (σ = 1.77) 182 (σ = 6.88) 94 (σ = 6.14) 111 (σ = 5.65) 122 (σ = 2.89) Relaxed Compliance 108 (σ = 1.47) 132 (σ = 2.14) 174 (σ = 7.25) 88 (σ = 5.86) 107 (σ = 4.69) 117 (σ = 3.16) 107
108. Evaluation Results - Cost (Random Walk 2) Threshold-based BTU-based BTU 10 BTU 30 BTU 60 BTU 10 BTU 30 BTU 60 Resource Cost 60 (σ = 4.04) 82 (σ = 9.17) 118 (σ = 9.17) 74 (σ = 6.08) 90 (σ = 0.00) 106 (σ = 3.46) Real-time Compliance 164 (σ = 3.84) 187 (σ = 9.88) 223 (σ = 9.88) 173 (σ = 5.35) 184 (σ = 0.99) 196 (σ = 3.33) Near real-time Compliance 113 (σ = 3.47) 136 (σ = 9.93) 172 (σ = 9.93) 100 (σ = 6.65) 109 (σ = 2.47) 121 (σ = 3.59) Relaxed Compliance 104 (σ = 2.49) 127 (σ = 8.29) 163 (σ = 8.29) 93 (σ = 7.15) 102 (σ = 1.61) 116 (σ = 3.48) 108
109. Evaluation Results - Threshold-based Baseline (2-Level) 0 20 40 60 80 100 120 0 2 4 6 8 Time (Minutes) Amount of Manufacturing Machines and VMs 0 10 20 30 40 50 60 70 Amount of Operator Instances 0 0 Pattern VMs Operator Instances 109
110. Evaluation Results - Threshold-based Baseline (Stepwise) 0 0 Pattern VMs Operator Instances 110 0 20 40 60 80 100 120 0 2 4 6 8 Time (Minutes) Amount of Manufacturing Machines and VMs 0 10 20 30 40 50 60 70 Amount of Operator Instances
111. Evaluation Results - Threshold-based Baseline (Random Walk 1) 0 0 Pattern VMs Operator Instances 111 0 20 40 60 80 100 120 0 2 4 6 8 Time (Minutes) Amount of Manufacturing Machines and VMs 0 10 20 30 40 50 60 70 Amount of Operator Instances
112. Evaluation Results - Threshold-based Baseline (Random Walk 2) 0 0 Pattern VMs Operator Instances 112 0 20 40 60 80 100 120 0 2 4 6 8 Time (Minutes) Amount of Manufacturing Machines and VMs 0 10 20 30 40 50 60 70 Amount of Operator Instances
113. Evaluation Results - BTU 10 (2-Level) 0 0 Pattern VMs Operator Instances 113 0 20 40 60 80 100 120 0 2 4 6 8 Time (Minutes) Amount of Manufacturing Machines and VMs 0 10 20 30 40 50 60 70 Amount of Operator Instances
114. Evaluation Results - BTU 30 (2-Level) 0 0 Pattern VMs Operator Instances 0 20 40 60 80 100 120 0 2 4 6 8 Time (Minutes) Amount of Manufacturing Machines and VMs 0 10 20 30 40 50 60 70 Amount of Operator Instances 114
115. Evaluation Results - BTU 60 (2-Level) 0 0 Pattern VMs Operator Instances 115 0 20 40 60 80 100 120 0 2 4 6 8 Time (Minutes) Amount of Manufacturing Machines and VMs 0 10 20 30 40 50 60 70 Amount of Operator Instances
116. Evaluation Results - BTU 30 (Stepwise) 0 0 Pattern VMs Operator Instances 116 0 20 40 60 80 100 120 0 2 4 6 8 Time (Minutes) Amount of Manufacturing Machines and VMs 0 10 20 30 40 50 60 70 Amount of Operator Instances
117. Evaluation Results - BTU 30 (Random Walk 1) 0 0 Pattern VMs Operator Instances 117 0 20 40 60 80 100 120 0 2 4 6 8 Time (Minutes) Amount of Manufacturing Machines and VMs 0 10 20 30 40 50 60 70 Amount of Operator Instances
118. Evaluation Results - BTU 30 (Random Walk 2) 0 0 Pattern VMs Operator Instances 118 0 20 40 60 80 100 120 0 2 4 6 8 Time (Minutes) Amount of Manufacturing Machines and VMs 0 10 20 30 40 50 60 70 Amount of Operator Instances
119. Contribution Overview P1: C. Hochreiner, M. Vögler, S. Schulte, S. Dustdar, "Cost-efficient enactment of stream processing topologies," PeerJ Computer Science. P2: C. Hochreiner, S. Schulte, S. Dustdar, and F. Lecue, "Elastic Stream Processing for Distributed Environments," IEEE Internet Computing. P3: C. Hochreiner, M. Nardelli, B. Knasmueller, S. Schulte, and S. Dustdar, “VTDL: A Notation for Data Stream Processing Applications," SOSE 2018. B2: R. Ganaus, Deployment Techniques for Dynamic Application Scenarios. P4: C. Hochreiner, M. Vögler, P. Waibel, and S. Dustdar, "VISP: An Ecosystem for Elastic Data Stream Processing for the Internet of Things,” EDOC 2016. P5: C. Hochreiner, M. Vögler, S. Schulte, and S. Dustdar, "Elastic Stream Processing for the Internet of Things," CLOUD 2016. P6: C. Hochreiner, M. Vögler, J. M. Schleicher, C. Inzinger, S. Schulte, and S. Dustdar, "Nomadic Applications Traveling in the Fog," CN4IoT 2017. P7: C. Hochreiner, "VISP Testbed - A Toolkit for Modeling and Evaluating Resource Provisioning Algorithms for Stream Processing Applications," ZEUS 2017. P8: M. Borkowski, C. Hochreiner, and S. Schulte, "Moderated Resource Elasticity for Stream Processing Applications," Auto-DaSP 2017. P9: M. Nardelli, C. Hochreiner, and S. Schulte, "Elastic Provisioning of Virtual Machines for Container Deployment," ACPROSS 2017. M1: T. Biasion, A Network-based Business Model Framework for the Internet of Things. M2: T. Hiessl, Optimizing the Placement of Stream Processing Operators in the Fog. M3: B. Knasmueller, On Fault Tolerance in Stream Processing Systems. M4: F. Rinker, A Monitoring Framework for Stream Processing Applications in Heterogenous Environments. B1: A. Carrasco, Scalability for Data Stream Processing Frameworks. Infrastructure Data Sources VISP Marketplace Cloud Resources Edge Resources VISP Runtime Stream Processing Application Topology Configuration Operator Replication Infrastructure Management Operator Provisioning Messaging Operator Migration Operator Placement Shared Data Failure Detection Privacy Security Topology Enactment Manufacturing Machines Wearable Devices News Portals Failure Compensation Monitoring Monetization Operator Distribution Topology Builder Operator Registry Runtime Synchronization Stock Market SLA Compliance Operational Layer Communication Layer Infrastructure Layer Cross-cutting Concerns P2 P3 P4 P6 P1 P5 P7 P8 P9 M1 M2 M3 M4 B1 B2 119