[DevOpsCon] Developer Targeted Performance Analytics

Slides on the session on "Developer Targeted Performance Analytics" at DevOpsCon'17 in Berlin, Germany

Transcript

1. Developer Targeted Performance Analytics Supporting Software Development Decisions with Runtime Information Jürgen Cito 
 DevOpsCon’17, Berlin, Germany Source: https://flic.kr/p/bXf4vw @citostyle Source: https://flic.kr/p/ebdWmN by David
2. 2 Distributed Architectures Multiple Technologies Infrastructure at Scale Scattered Information Figure taken from Chen et al. A provider-side viewof web search response time. 2013. Monitoring & Analysis of Complex (Cloud) Systems
3. Code Artifacts Deployment ....... [26/06/2015:21205.0], responseTime, "CustomerService", 204 [26/06/2015:21215.0], responseTime, "CustomerService", 169 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ Operations Data observe Context Cloud Infrastructure VM1 VM2 Supplier Service User Interface Purch S
4. Scenario
 (Overview) Software Engineer VoIP/Collaboration App
 (Industrial Use Case) Adapted from https://xkcd.com/228/
5. Scenario
 (Change Request)
6. Scenario
 (Code Change)
7. Scenario
 (Check-List) Change… …passes functional tests …passes non-functional (performance) tests …goes through Continuous Integration …gets deployed in production …leads to severe performance degradation
8. Scenario
 (What was the Problem?) > Login method for 1 users/team included a service call 
 to 3rd party provider to load profile pictures
 
 > Login was not able to scale gracefully to multiple teams > Tests failed to simulate the situation
9. Problem & Conjecture Data combinatorics of configuration in production environments are different to what profilers and tests can simulate either locally or in staging environments. Especially in the cloud, scalable infrastructure requires approaches to leverage information gathered at production runtime and provide feedback to software engineers during development.
10. Developer Targeted Performance Analytics What tools and data are being leveraged for decision making in software development for the cloud? How can we use runtime information to support data-driven decision making for engineers during software development guides
11. What tools and data are being leveraged for decision making in software development for the cloud?
12. Study on Software Development for the Cloud “The Making of Cloud Applications” - FSE’15 Adapted from https://xkcd.com/1423/ Interview with 25 software developers that deploy in the cloud Survey with 294 responses Developer Me
13. Study on Software Development for the Cloud “The Making of Cloud Applications” - FSE’15
14. Study on Software Development for the Cloud “The Making of Cloud Applications” - FSE’15 62% say more metrics are available in the cloud & 84% say they look at performance metrics on a regular basis
15. Study on Software Development for the Cloud “The Making of Cloud Applications” - FSE’15
16. Study on Software Development for the Cloud “The Making of Cloud Applications” - FSE’15 Adapted from https://xkcd.com/1423/ Topic: Solving problems that have been detected in production Nah, I rather go by intuition? Do you look at any metrics?
17. Log Overload
18. Log Overload Diarrhea
19. State-of-the-Art SaaS tools Open Source
20. How can we use runtime information to support data-driven decision making for engineers during software development?
21. Developer Targeted Performance Analytics integrating metrics into daily developer workflows prevent bad things from happening bring metrics into context
22. Conceptual Overview
 “Runtime Metric Meets Developer” - SPLASH Onward’15 Paper: https://peerj.com/preprints/985/
 Blog Post on #themorningpaper
 https://blog.acolyer.org/2015/11/10/runtime-metric-meets-developer-building-better-cloud-applications-using-feedback/
23. Code Artifacts Deployment ....... [26/06/2015:21205.0], responseTime, "CustomerService", 204 [26/06/2015:21215.0], responseTime, "CustomerService", 169 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ Operations Data observe readConnecti on getConnectio ns ids connectionPo ol readConnecti on getConnectio ns ids connectionPo ol readConnection getConnections ids connectionPool Runtime Metric Annotated AST Conceptual Overview
 Cloud Infrastructure VM1 VM2 Supplier Service User Interface Purch S “Runtime Metric Meets Developer” - SPLASH Onward’15
24. Conceptual Overview
 “Runtime Metric Meets Developer” - SPLASH Onward’15 Abstract Syntax Tree (AST) IDE Distributed Runtime Traces Feedback Mapping Runtime Metric Annotated AST Performance Augmented Source Code ? Prediction through Impact Analysis ....... [26/06/2015:21205.0], responseTime, “showConnections, 204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21205.0], responseTime, “showConnections, 204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35 [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216] ........
25. readConnections connections getConnections getImage setConnectionImage setConnectionStatus ....... [26/06/2015:21205.0], responseTime, “showConnections, 204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35 [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216] ........ Feedback Specification
26. Industrial Case Study: SAP HANA WebIDE
 Performance Spotter
27. Industrial Case Study: SAP HANA WebIDE
 Performance Spotter
28. New Code overallRating readConnection Code Change overallRating readConnection size: suppliers getSuppliers Loop:suppliers getPurchaseRating ?
29. overallRating readConnection size: suppliers getSuppliers Loop:suppliers getPurchaseRating [Statistical Inference] New Code [Feedback Propagation] Predicted Entity
30. Live Performance Prediction (PerformanceHat)
31. PoC Implementation of the approach as an Eclipse plugin > Performance Awareness > Contextualization 31 PerformanceHat
 (Industrial Case Study: Cloudmore)
32. PerformanceHat
 (Industrial Case Study: Cloudmore) > Instant performance feedback prediction on code changes
33. Does it scale?
34. Matching information to source code is part of the build process in the IDE Scaling Runtime Performance Matching Matching information to source code is part of the daily developer workflow It cannot be slow! 34
35. Scaling Runtime Performance Matching Component Analysis: Which of the components could 
 impede the development workflow? Every build needs to:
 > Identify and extract relevant AST nodes > Retrieve information for each relevant node > Execute inference for new nodes with unknown properties > Propagate predicted entity 35
36. Scaling Runtime Performance Matching Component Analysis: Which of the components could 
 impede the development workflow? Analysis needs to take into account different scopes
 > Single File Builds > Incremental Builds > Full Project Builds 36
37. Scaling Runtime Performance Matching Component Analysis: Which of the components could 
 impede the development workflow? Potentially problematic 37
38. Scaling Runtime Performance Matching Revamped architecture to enable scalability Local Workstation Local Feedback Handler Local IDE Cache Datastore HTTP Infrastructure 1 Deployed Feedback Handler Deployed System Transform IDE Infrastructure 2 Deployed Feedback Handler Deployed System Transform Infrastructure n Deployed Feedback Handler Deployed System Transform …. HTTP Stream (Apache Kafka Broker) HTTP Specification Function Inference Function Registered Filters
39. Scaling Runtime Performance Matching Component Analysis: Measurements Agilefant Full EU Project Full Story Single File Controller Single File Cache / Cold Cache / Warm No Cache / Cold No Cache / Warm 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 Build Time [%] Scenario Prediction Marking Fetching Other Attaching
40. Does it help developers?
41. Controlled User Study 
 Overview “Classical” software engineering user study: > Between Subject Study
 > 20 software engineers (min. 1 year of experience) > Relevant study subject: Agilefant > 4 Tasks (both related and not related to performance bugs) Control Group (Kibana) Treatment Group (PerformanceHat) 41
42. Controlled User Study 
 Hypotheses & Metrics H01: Given a maintenance task that would introduce a performance bug, software engineers using PerformanceHat are faster in detecting the performance bug
 H02: Given a maintenance task that would introduce a performance bug, software engineers using PerformanceHat are faster in finding the root cause of the performance bug
 H03: Given a maintenance task that is not relevant to performance, software engineers using PerformanceHat are not slower than the control group in solving the task [Metric: First Encounter (FE)] [Metric: Root-Cause Analysis (RCA)] [Metric: Development Time] 42
43. Controlled User Study 
 Results 43 100 200 300 400 500 Time (in seconds) T1 (Total) T2 (Total) T2 (FE) T2 (RCA) T3 (Total) T4 (Total) T4 (FE) T4 (RCA) Control Treatment
44. @citostyle Jürgen Cito <cito@ifi.uzh.ch> Developer Targeted Performance Analytics leverages runtime data by matching performance metrics to source code artefacts to support decision-making during software development For which (quality) attributes does it make sense 
 to correlate with source code artefacts? How much feedback is too much feedback? http://sealuzh.github.io/PerformanceHat/