(Mis-)Adventures in Performance Engineering

Transcript

1. (Mis-)Adventures in Software Performance Engineering Dr. Philipp Leitner philipp.leitner@chalmers.se @xLeitix
2. Chalmers 2 About Me
3. Chalmers 2 About Me PhD (2011) TU Vienna
4. Chalmers 2 About Me PhD (2011) TU Vienna 2014 - 2017 University of Zurich
5. Chalmers 2 About Me PhD (2011) TU Vienna 2014 - 2017 University of Zurich 2017+ Chalmers
6. Chalmers 3 Team
7. Chalmers 3 Team UZH
8. Chalmers 3 Team UZH Chalmers / GU
9. Chalmers 4 About Software Performance
10. Chalmers 5 Different Views
11. Chalmers 5 Different Views
12. Chalmers 5 Different Views
13. Chalmers 5 Different Views
14. Chalmers 5 Different Views
15. Chalmers 6 Performance Matters Source: http://radar.oreilly.com/2009/06/bing-and-google-agree-slow-pag.html “(…) Amazon found every 100 ms response time cost them 1% in sales (…)”
16. Chalmers 7 Performance Matters
17. Chalmers 8 Factors of Influence
18. Chalmers 8 Factors of Influence Code
19. Chalmers 8 Factors of Influence Code Environment
20. Chalmers 8 Factors of Influence Code Environment Users
21. Chalmers 9 Code Environment Users Exploratory Studies Benchmarking Systems
22. Chalmers 9 Code Environment Users Exploratory Studies Benchmarking Systems ICPE’17 (under submission) FSE’15
23. Chalmers 9 Code Environment Users Exploratory Studies Benchmarking Systems ICPE’17 (under submission) FSE’15 TOIT’16
24. Chalmers 9 Code Environment Users Exploratory Studies Benchmarking Systems ICPE’17 (under submission) FSE’15 TOIT’16 Onward’15 Middleware’16
25. Chalmers 10 Code Environment Users Exploratory Studies Benchmarking Systems ICPE’17 (under submission) FSE’15 TOIT’16 Onward’15 Middleware’16
26. Chalmers 11 Cloud Development Interview Study ESEC/FSE 2015 “The Making of Cloud Applications” Exploratory study on how developers build cloud applications 25 semi-structured interviews (2 rounds) 294 survey responses Themes: Architecture of cloud apps Managing performance and costs Automation and tooling Changed processes and culture
27. Chalmers 12 Performance Dashboards in Practice Adapted from https://xkcd.com/1423/ Nah, I rather go by intuition? Do you look at any metrics? ESEC/FSE 2015 “The Making of Cloud Applications”
28. Chalmers 13 Feedback-Driven Development
29. Chalmers 13 Feedback-Driven Development
30. Chalmers 13 Feedback-Driven Development Deployment
31. Chalmers 13 Feedback-Driven Development Deployment VM1 VM2 Supplier Service User Interface VM Purchase Or Service
32. Chalmers 13 Feedback-Driven Development Deployment ....... [26/06/2015:21205.0], responseTime, "CustomerService", 204 [26/06/2015:21215.0], responseTime, "CustomerService", 169 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ VM1 VM2 Supplier Service User Interface VM Purchase Or Service observe
33. Chalmers 13 Feedback-Driven Development Deployment ....... [26/06/2015:21205.0], responseTime, "CustomerService", 204 [26/06/2015:21215.0], responseTime, "CustomerService", 169 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ readConnecti on getConnectio ns ids connectionPo ol readConnecti on getConnectio ns ids connectionPo ol readConnection getConnections ids connectionPool VM1 VM2 Supplier Service User Interface VM Purchase Or Service observe
34. Chalmers 14 PerformanceHat OSS Eclipse plug-in that implements those ideas Github: http://sealuzh.github.io/PerformanceHat/
35. Chalmers 14 PerformanceHat OSS Eclipse plug-in that implements those ideas Github: http://sealuzh.github.io/PerformanceHat/ Covered by Adrian Colyer “The Morning Paper”
36. Chalmers 14 PerformanceHat OSS Eclipse plug-in that implements those ideas Github: http://sealuzh.github.io/PerformanceHat/ Covered by Adrian Colyer “The Morning Paper” Presented at WebPerfDays, Velocity, …
37. Chalmers 14 PerformanceHat OSS Eclipse plug-in that implements those ideas Github: http://sealuzh.github.io/PerformanceHat/ Covered by Adrian Colyer “The Morning Paper” Presented at WebPerfDays, Velocity, … Feature requests for Visual Studio, IntelliJ, …
38. Chalmers 15 Code Environment Users Exploratory Studies Benchmarking Systems ICPE’17 (under submission) FSE’15 TOIT’16 Onward’15 Middleware’16
39. Chalmers 16 Capacity Planning in the Cloud is hard
40. Chalmers 16 Capacity Planning in the Cloud is hard
41. Chalmers 16 Capacity Planning in the Cloud is hard
42. Chalmers 17 What cloud provider should I choose? Should I go for many small or few large instances? General-purpose or *-optimized? Pay for better IOPS or not? ……………
43. Chalmers 17 What cloud provider should I choose? Should I go for many small or few large instances? General-purpose or *-optimized? Pay for better IOPS or not? …………… ➡ Need for Benchmarking
44. Chalmers 18 Basic Approach CCGrid 2017 “An Approach and Case Study of Cloud Instance Type Selection for Multi-Tier Web Applications” Benchmark Manager Provider API results provision Instance start benchmark destroy
45. Chalmers 18 Basic Approach CCGrid 2017 “An Approach and Case Study of Cloud Instance Type Selection for Multi-Tier Web Applications” Benchmark Manager Provider API results provision Instance start benchmark destroy Repeat X times
46. Chalmers 18 Basic Approach CCGrid 2017 “An Approach and Case Study of Cloud Instance Type Selection for Multi-Tier Web Applications” Benchmark Manager Provider API results provision Instance start benchmark destroy For many benchmarks, providers, instance types, configurations Repeat X times
47. Chalmers 19 Example Results TOIT 2016 “Patterns in the Chaos - A Study of Performance Variation and Predictability in Public IaaS Clouds” 20 30 40 0 20 40 60 Measurement Runtime [h] IO Bandwidth [Mb/s] Instance 9097 Instance 14704
48. Chalmers 20 Example Results TOIT 2016 “Patterns in the Chaos - A Study of Performance Variation and Predictability in Public IaaS Clouds”
49. Chalmers 21 Key Takeaways
50. Chalmers 21 Key Takeaways
51. Chalmers 21 Key Takeaways 20 30 40 0 20 40 60 Measurement Runtime [h] IO Bandwidth [Mb/s] Instance 9097 Instance 14704
52. Chalmers 21 Key Takeaways 20 30 40 0 20 40 60 Measurement Runtime [h] IO Bandwidth [Mb/s] Instance 9097 Instance 14704