[Velocity] Can we make developers care about operations?

Slides on the session on "Can we make developers care about operations?" at Velocity'17 in London, UK.

Transcript

1. Can we make developers care about operations? Extending Developers’ Mental Models by Runtime Dimensions 
 Velocity’17, London, United Kingdom Source: https://flic.kr/p/bXf4vw @citostyle Jürgen Cito Source: Jose Maria Cuellar | https://flic.kr/p/gEThpX
2. Feel concern or interest; attach importance to something. “to care” (Verb) https://en.oxforddictionaries.com/definition/care [often with a negative] “They don’t care about [operations]”
3. Developer’s concern: Code
4. Code Artifacts Deployment ....... [26/06/2015:21205.0], responseTime, "CustomerService", 204 [26/06/2015:21215.0], responseTime, "CustomerService", 169 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ Operations Data Observation Instrumentation Cloud Infrastructure VM1 VM2 Supplier Service User Interface Purch S Code is not the whole story
5. Developer’s concern: Code Mental Model when starting out
6. Developer’s concern: Code Mental models evolve when learning to program Programming Lense
7. Code Artifacts Deployment ....... [26/06/2015:21205.0], responseTime, "CustomerService", 204 [26/06/2015:21215.0], responseTime, "CustomerService", 169 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ Operations Data Observation Instrumentation Cloud Infrastructure VM1 VM2 Supplier Service User Interface Purch S Merging two worlds Magic
8. Developer’s concern: Code Idea: Add runtime dimension to mental models Programming Lense Runtime Lense
9. How are developers solving issues that happen in production? How can we make developers care about operations? [Can we use information from runtime to support data-driven decision making for developers] guides Developer Targeted Runtime Analytics
10. How are developers solving issues that happen in production?
11. Interview with 25 software developers that deploy in the cloud Survey with 294 responses Developer Me Study: Software Development in the Cloud “The Making of Cloud Applications” in Foundations of Software Engineering (FSE’15)
12. Study: Software Development in the Cloud “The Making of Cloud Applications” in Foundations of Software Engineering (FSE’15)
13. 62% say more metrics are available in the cloud & 84% say they look at performance metrics on a regular basis Study: Software Development in the Cloud “The Making of Cloud Applications” in Foundations of Software Engineering (FSE’15)
14. Study: Software Development in the Cloud “The Making of Cloud Applications” in Foundations of Software Engineering (FSE’15)
15. Adapted from https://xkcd.com/1423/ Topic: Solving problems that have been detected in production Nah, I rather go by intuition? Do you look at any metrics? Study: Software Development in the Cloud “The Making of Cloud Applications” in Foundations of Software Engineering (FSE’15)
16. Log Overload
17. Log Overload Diarrhea
18. State-of-the-Art SaaS tools Open Source
19. Can we use information from runtime to support data-driven decision making for developers?
20. Developer Targeted Runtime Analytics integrating metrics into daily developer workflows prevent bad things from happening bring metrics into context
21. Conceptual Overview “Runtime Metric Meets Developer” in SPLASH Onward’15 
 Blog Post on #themorningpaper
 https://blog.acolyer.org/2015/11/10/runtime-metric-meets-developer-building-better-cloud-applications-using-feedback/
22. Abstract Syntax Tree (AST) IDE Distributed Runtime Traces Feedback Mapping Runtime Metric Annotated AST Performance Augmented Source Code ? Prediction through Impact Analysis ....... [26/06/2015:21205.0], responseTime, “showConnections, 204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21205.0], responseTime, “showConnections, 204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35 [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216] ........ Conceptual Overview “Runtime Metric Meets Developer” in SPLASH Onward’15
23. Abstract Syntax Tree (AST) IDE Distributed Runtime Traces Feedback Mapping Runtime Metric Annotated AST Performance Augmented Source Code ? Prediction through Impact Analysis ....... [26/06/2015:21205.0], responseTime, “showConnections, 204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21205.0], responseTime, “showConnections, 204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35 [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216] ........ Specification (Feedback Mapping) Trace Filtering ....... [26/06/2015:21205.0], responseTime, "CustomerService", 204 [26/06/2015:21215.0], responseTime, "CustomerService", 169 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21205.0], responseTime, "CustomerService", 204 [26/06/2015:21215.0], responseTime, "CustomerService", 169 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ Inference on Code Changes Visualization (Presentation) Abstract Syntax Tree (AST) IDE Distributed Runtime Traces Feedback Mapping Runtime Metric Annotated AST Performance Augmented Source Code ? Prediction through Impact Analysis ....... [26/06/2015:21205.0], responseTime, “showConnections, 204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21205.0], responseTime, “showConnections, 204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35 [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216] ........ Abstract Theoretical Framework “Runtime Metric Meets Developer” in SPLASH Onward’15
24. Abstract Syntax Tree (AST) IDE Distributed Runtime Traces Feedback Mapping Runtime Metric Annotated AST Performance Augmented Source Code ? Prediction through Impact Analysis ....... [26/06/2015:21205.0], responseTime, “showConnections, 204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21205.0], responseTime, “showConnections, 204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35 [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216] ........ Specification (Feedback Mapping) Trace Filtering ....... [26/06/2015:21205.0], responseTime, "CustomerService", 204 [26/06/2015:21215.0], responseTime, "CustomerService", 169 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21205.0], responseTime, "CustomerService", 204 [26/06/2015:21215.0], responseTime, "CustomerService", 169 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ Inference on Code Changes Visualization (Presentation)
25. readConnections connections getConnections getImage setConnectionImage setConnectionStatus ....... [26/06/2015:21205.0], responseTime, “showConnections, 204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35 [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216] ........ Specification/Mapping
26. Distributed Runtime Traces ....... [26/06/2015:21205.0], responseTime, “showConnections, 204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21205.0], responseTime, “showConnections, 204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35 [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216] ........ Specification (Feedback Mapping) Inference on Code Changes Visualization (Presentation) Trace Filtering ....... [26/06/2015:21205.0], responseTime, "CustomerService", 204 [26/06/2015:21215.0], responseTime, "CustomerService", 169 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21205.0], responseTime, "CustomerService", 204 [26/06/2015:21215.0], responseTime, "CustomerService", 169 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........
27. Enabling Filters on runtime traces can support different debugging and code comprehension use cases Possible Filters: > Region > Data Centres > Correlation/Session IDs > …
28. Distributed Runtime Traces ....... [26/06/2015:21205.0], responseTime, “showConnections, 204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21205.0], responseTime, “showConnections, 204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35 [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216] ........ Specification (Feedback Mapping) Trace Filtering ....... [26/06/2015:21205.0], responseTime, "CustomerService", 204 [26/06/2015:21215.0], responseTime, "CustomerService", 169 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21205.0], responseTime, "CustomerService", 204 [26/06/2015:21215.0], responseTime, "CustomerService", 169 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ Visualization (Presentation) Inference on Code Changes
29. New Code overallRating readConnection Code Change overallRating readConnection size: suppliers getSuppliers Loop:suppliers getPurchaseRating ?
30. overallRating readConnection size: suppliers getSuppliers Loop:suppliers getPurchaseRating [Statistical Inference] New Code [Feedback Propagation] Predicted Entity
31. Distributed Runtime Traces ....... [26/06/2015:21205.0], responseTime, “showConnections, 204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21205.0], responseTime, “showConnections, 204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35 [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216] ........ Specification (Feedback Mapping) Trace Filtering ....... [26/06/2015:21205.0], responseTime, "CustomerService", 204 [26/06/2015:21215.0], responseTime, "CustomerService", 169 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21205.0], responseTime, "CustomerService", 204 [26/06/2015:21215.0], responseTime, "CustomerService", 169 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ Inference on Code Changes Visualization (Presentation) Abstract Syntax Tree (AST) IDE Distributed Runtime Traces Feedback Mapping Runtime Metric Annotated AST Performance Augmented Source Code ? Prediction through Impact Analysis ....... [26/06/2015:21205.0], responseTime, “showConnections, 204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21205.0], responseTime, “showConnections, 204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35 [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216] ........
32. eedback Mapping Metric Annotated AST Performance Augmented Source Code ........ Feedback Mapping Runtime Metric Annotated AS ? Prediction through Impact Analysis
33. Implementation of the framework as an Eclipse plugin > Performance Awareness > Contextualization 33 PerformanceHat (Proof-of-concept)
34. > Instant performance feedback prediction on code changes PerformanceHat (Industrial Case Study: Cloudmore)
35. First Transfer: SAP HANA WebIDE
 Performance Spotter
36. First Transfer: SAP HANA WebIDE
 Performance Spotter
37. Does it scale?
38. Matching information to source code is part of the build process in the IDE Scaling Runtime Performance Matching Matching information to source code is part of the daily developer workflow It cannot be slow! 38
39. Component Analysis: Which of the components could 
 impede the development workflow? Every build needs to:
 > Identify and extract relevant AST nodes > Retrieve information for each relevant node > Execute inference for new nodes with unknown properties > Propagate predicted entity 39
40. Component Analysis: Which of the components could 
 impede the development workflow? Analysis needs to take into account different scopes
 > Single File Builds > Incremental Builds > Full Project Builds 40
41. Component Analysis: Which of the components could 
 impede the development workflow? Potentially problematic 41
42. Revamped architecture to enable scalability Local Workstation Local Feedback Handler Local IDE Cache Datastore HTTP Infrastructure 1 Deployed Feedback Handler Deployed System Transform IDE Infrastructure 2 Deployed Feedback Handler Deployed System Transform Infrastructure n Deployed Feedback Handler Deployed System Transform …. HTTP Stream (Apache Kafka Broker) HTTP Specification Function Inference Function Registered Filters
43. Scaling Runtime Performance Matching Component Analysis: Measurements Agilefant Full EU Project Full Story Single File Controller Single File Cache / Cold Cache / Warm No Cache / Cold No Cache / Warm 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 Build Time [%] Scenario Prediction Marking Fetching Other Attaching
44. Does it help developers?
45. “Classical” software engineering user study: > Between Subject Study
 > 20 software engineers (min. 1 year of experience) > Relevant study subject: Agilefant > 4 Tasks (both related and not related to performance bugs) Control Group (Kibana) Treatment Group (PerformanceHat) 45 Controlled User Study Overview
46. H01: Given a maintenance task that would introduce a performance bug, software engineers using PerformanceHat are faster in detecting the performance bug
 H02: Given a maintenance task that would introduce a performance bug, software engineers using PerformanceHat are faster in finding the root cause of the performance bug
 H03: Given a maintenance task that is not relevant to performance, software engineers using PerformanceHat are not slower than the control group in solving the task [Metric: First Encounter (FE)] [Metric: Root-Cause Analysis (RCA)] [Metric: Development Time] 46 Controlled User Study Hypothesis and Metrics
47. 47 100 200 300 400 500 Time (in seconds) T1 (Total) T2 (Total) T2 (FE) T2 (RCA) T3 (Total) T4 (Total) T4 (FE) T4 (RCA) Control Treatment Controlled User Study Results
48. @citostyle Jürgen Cito <cito@ifi.uzh.ch> Developer Targeted Runtime Analytics leverages runtime data by matching traces to source code artefacts to support data-driven decision-making during software development For which runtime aspects does it make sense 
 to correlate with source code artefacts? How much feedback is too much feedback? http://sealuzh.github.io/PerformanceHat/ https://speakerdeck.com/citostyle Slides juergencito.com