[Velocity] Can we make developers care about operations?

Slides on the session on "Can we make developers care about operations?" at Velocity'17 in London, UK.

Transcript

1. Can we make developers care about operations?
   Extending Developers’ Mental Models by Runtime Dimensions
   

   Velocity’17, London, United Kingdom
   Source: https://flic.kr/p/bXf4vw
   @citostyle
   Jürgen Cito
   Source: Jose Maria Cuellar | https://flic.kr/p/gEThpX
   

   View Slide

2. Feel concern or interest; attach importance to something.
   “to care” (Verb)
   https://en.oxforddictionaries.com/definition/care
   [often with a negative]
   “They don’t care about [operations]”
   

   View Slide

3. Developer’s concern: Code
   

   View Slide

4. Code Artifacts
   Deployment
   .......
   [26/06/2015:21205.0], responseTime, "CustomerService", 204
   [26/06/2015:21215.0], responseTime, "CustomerService", 169
   [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM2", 0.73
   [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM1", 0.69
   [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35
   [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216]
   ........
   Operations Data
   Observation
   Instrumentation
   Cloud Infrastructure
   VM1
   VM2
   Supplier
   Service
   User
   Interface
   Purch
   S
   Code is not the whole story
   

   View Slide

5. Developer’s concern: Code
   Mental Model when starting out
   

   View Slide

6. Developer’s concern: Code
   Mental models evolve when learning to program
   Programming
   Lense
   

   View Slide

7. Code Artifacts
   Deployment
   .......
   [26/06/2015:21205.0], responseTime, "CustomerService", 204
   [26/06/2015:21215.0], responseTime, "CustomerService", 169
   [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM2", 0.73
   [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM1", 0.69
   [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35
   [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216]
   ........
   Operations Data
   Observation
   Instrumentation
   Cloud Infrastructure
   VM1
   VM2
   Supplier
   Service
   User
   Interface
   Purch
   S
   Merging two worlds
   Magic
   

   View Slide

8. Developer’s concern: Code
   Idea: Add runtime dimension to mental models
   Programming
   Lense
   Runtime
   Lense
   

   View Slide

9. How are developers solving issues that happen in production?
   How can we make developers care about operations?
   [Can we use information from runtime to support
   data-driven decision making for developers]
   guides
   Developer Targeted Runtime Analytics
   

   View Slide

10. How are developers solving issues
    that happen in production?
    

    View Slide

11. Interview with 25 software developers that deploy in the cloud
    Survey with 294 responses Developer
    Me
    Study: Software Development in the Cloud
    “The Making of Cloud Applications” in Foundations of Software Engineering (FSE’15)
    

    View Slide

12. Study: Software Development in the Cloud
    “The Making of Cloud Applications” in Foundations of Software Engineering (FSE’15)
    

    View Slide

13. 62% say more metrics are available in the cloud
    &
    84% say they look at performance metrics
    on a regular basis
    Study: Software Development in the Cloud
    “The Making of Cloud Applications” in Foundations of Software Engineering (FSE’15)
    

    View Slide

14. Study: Software Development in the Cloud
    “The Making of Cloud Applications” in Foundations of Software Engineering (FSE’15)
    

    View Slide

15. Adapted from https://xkcd.com/1423/
    Topic: Solving problems that have been detected in production
    Nah, I rather go
    by intuition?
    Do you look at
    any metrics?
    Study: Software Development in the Cloud
    “The Making of Cloud Applications” in Foundations of Software Engineering (FSE’15)
    

    View Slide

16. Log Overload
    

    View Slide

17. Log Overload Diarrhea
    

    View Slide

18. State-of-the-Art
    SaaS tools
    Open Source
    

    View Slide

19. Can we use information from runtime to support
    data-driven decision making for developers?
    

    View Slide

20. Developer Targeted
    Runtime Analytics
    integrating metrics into
    daily developer workflows
    prevent bad things
    from happening
    bring metrics
    into context
    

    View Slide

21. Conceptual Overview
    “Runtime Metric Meets Developer” in SPLASH Onward’15
    

    Blog Post on #themorningpaper

    https://blog.acolyer.org/2015/11/10/runtime-metric-meets-developer-building-better-cloud-applications-using-feedback/
    

    View Slide

22. Abstract Syntax Tree (AST)
    IDE Distributed Runtime Traces
    Feedback Mapping
    Runtime Metric Annotated AST
    Performance Augmented
    Source Code
    ?
    Prediction through
    Impact Analysis
    .......
    [26/06/2015:21205.0], responseTime, “showConnections, 204
    [26/06/2015:21215.0], responseTime, “setConnectionImage, 169
    [26/06/2015:21216.0], responseTime, “PaymentService”, 79
    [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69
    [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35
    [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216]
    ........
    .......
    [26/06/2015:21205.0], responseTime, “showConnections, 204
    [26/06/2015:21215.0], responseTime, “setConnectionImage, 169
    [26/06/2015:21216.0], responseTime, “PaymentService”, 79
    [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69
    [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35
    [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216]
    ........
    .......
    [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73
    [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69
    [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35
    [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216]
    ........
    Conceptual Overview
    “Runtime Metric Meets Developer” in SPLASH Onward’15
    

    View Slide

23. Abstract Syntax Tree (AST)
    IDE Distributed Runtime Traces
    Feedback Mapping
    Runtime Metric Annotated AST
    Performance Augmented
    Source Code
    ?
    Prediction through
    Impact Analysis
    .......
    [26/06/2015:21205.0], responseTime, “showConnections, 204
    [26/06/2015:21215.0], responseTime, “setConnectionImage, 169
    [26/06/2015:21216.0], responseTime, “PaymentService”, 79
    [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69
    [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35
    [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216]
    ........
    .......
    [26/06/2015:21205.0], responseTime, “showConnections, 204
    [26/06/2015:21215.0], responseTime, “setConnectionImage, 169
    [26/06/2015:21216.0], responseTime, “PaymentService”, 79
    [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69
    [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35
    [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216]
    ........
    .......
    [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73
    [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69
    [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35
    [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216]
    ........
    Specification
    (Feedback Mapping)
    Trace Filtering
    .......
    [26/06/2015:21205.0], responseTime, "CustomerService", 204
    [26/06/2015:21215.0], responseTime, "CustomerService", 169
    [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM2", 0.73
    [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM1", 0.69
    [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35
    [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216]
    ........
    .......
    [26/06/2015:21205.0], responseTime, "CustomerService", 204
    [26/06/2015:21215.0], responseTime, "CustomerService", 169
    [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM2", 0.73
    [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM1", 0.69
    [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35
    [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216]
    ........
    Inference on
    Code Changes
    Visualization
    (Presentation)
    Abstract Syntax Tree (AST)
    IDE Distributed Runtime Traces
    Feedback Mapping
    Runtime Metric Annotated AST
    Performance Augmented
    Source Code
    ?
    Prediction through
    Impact Analysis
    .......
    [26/06/2015:21205.0], responseTime, “showConnections, 204
    [26/06/2015:21215.0], responseTime, “setConnectionImage, 169
    [26/06/2015:21216.0], responseTime, “PaymentService”, 79
    [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69
    [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35
    [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216]
    ........
    .......
    [26/06/2015:21205.0], responseTime, “showConnections, 204
    [26/06/2015:21215.0], responseTime, “setConnectionImage, 169
    [26/06/2015:21216.0], responseTime, “PaymentService”, 79
    [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69
    [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35
    [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216]
    ........
    .......
    [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73
    [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69
    [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35
    [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216]
    ........
    Abstract Theoretical Framework
    “Runtime Metric Meets Developer” in SPLASH Onward’15
    

    View Slide

24. Abstract Syntax Tree (AST)
    IDE Distributed Runtime Traces
    Feedback Mapping
    Runtime Metric Annotated AST
    Performance Augmented
    Source Code
    ?
    Prediction through
    Impact Analysis
    .......
    [26/06/2015:21205.0], responseTime, “showConnections, 204
    [26/06/2015:21215.0], responseTime, “setConnectionImage, 169
    [26/06/2015:21216.0], responseTime, “PaymentService”, 79
    [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69
    [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35
    [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216]
    ........
    .......
    [26/06/2015:21205.0], responseTime, “showConnections, 204
    [26/06/2015:21215.0], responseTime, “setConnectionImage, 169
    [26/06/2015:21216.0], responseTime, “PaymentService”, 79
    [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69
    [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35
    [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216]
    ........
    .......
    [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73
    [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69
    [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35
    [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216]
    ........
    Specification
    (Feedback Mapping)
    Trace Filtering
    .......
    [26/06/2015:21205.0], responseTime, "CustomerService", 204
    [26/06/2015:21215.0], responseTime, "CustomerService", 169
    [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM2", 0.73
    [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM1", 0.69
    [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35
    [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216]
    ........
    .......
    [26/06/2015:21205.0], responseTime, "CustomerService", 204
    [26/06/2015:21215.0], responseTime, "CustomerService", 169
    [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM2", 0.73
    [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM1", 0.69
    [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35
    [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216]
    ........
    Inference on
    Code Changes
    Visualization
    (Presentation)
    

    View Slide

25. readConnections
    connections
    getConnections
    getImage
    setConnectionImage
    setConnectionStatus
    .......
    [26/06/2015:21205.0], responseTime, “showConnections, 204
    [26/06/2015:21215.0], responseTime, “setConnectionImage, 169
    [26/06/2015:21216.0], responseTime, “PaymentService”, 79
    [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69
    [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35
    [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216]
    ........
    .......
    [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73
    [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69
    [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35
    [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216]
    ........
    Specification/Mapping
    

    View Slide

26. Distributed Runtime Traces
    .......
    [26/06/2015:21205.0], responseTime, “showConnections, 204
    [26/06/2015:21215.0], responseTime, “setConnectionImage, 169
    [26/06/2015:21216.0], responseTime, “PaymentService”, 79
    [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69
    [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35
    [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216]
    ........
    .......
    [26/06/2015:21205.0], responseTime, “showConnections, 204
    [26/06/2015:21215.0], responseTime, “setConnectionImage, 169
    [26/06/2015:21216.0], responseTime, “PaymentService”, 79
    [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69
    [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35
    [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216]
    ........
    .......
    [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73
    [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69
    [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35
    [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216]
    ........
    Specification
    (Feedback Mapping)
    Inference on
    Code Changes
    Visualization
    (Presentation)
    Trace Filtering
    .......
    [26/06/2015:21205.0], responseTime, "CustomerService", 204
    [26/06/2015:21215.0], responseTime, "CustomerService", 169
    [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM2", 0.73
    [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM1", 0.69
    [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35
    [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216]
    ........
    .......
    [26/06/2015:21205.0], responseTime, "CustomerService", 204
    [26/06/2015:21215.0], responseTime, "CustomerService", 169
    [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM2", 0.73
    [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM1", 0.69
    [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35
    [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216]
    ........
    

    View Slide

27. Enabling Filters on runtime traces can support different
    debugging and code comprehension use cases
    Possible Filters:
    > Region
    > Data Centres
    > Correlation/Session IDs
    > …
    

    View Slide

28. Distributed Runtime Traces
    .......
    [26/06/2015:21205.0], responseTime, “showConnections, 204
    [26/06/2015:21215.0], responseTime, “setConnectionImage, 169
    [26/06/2015:21216.0], responseTime, “PaymentService”, 79
    [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69
    [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35
    [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216]
    ........
    .......
    [26/06/2015:21205.0], responseTime, “showConnections, 204
    [26/06/2015:21215.0], responseTime, “setConnectionImage, 169
    [26/06/2015:21216.0], responseTime, “PaymentService”, 79
    [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69
    [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35
    [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216]
    ........
    .......
    [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73
    [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69
    [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35
    [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216]
    ........
    Specification
    (Feedback Mapping)
    Trace Filtering
    .......
    [26/06/2015:21205.0], responseTime, "CustomerService", 204
    [26/06/2015:21215.0], responseTime, "CustomerService", 169
    [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM2", 0.73
    [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM1", 0.69
    [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35
    [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216]
    ........
    .......
    [26/06/2015:21205.0], responseTime, "CustomerService", 204
    [26/06/2015:21215.0], responseTime, "CustomerService", 169
    [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM2", 0.73
    [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM1", 0.69
    [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35
    [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216]
    ........
    Visualization
    (Presentation)
    Inference on
    Code Changes
    

    View Slide

29. New
    Code
    overallRating
    readConnection
    Code
    Change
    overallRating
    readConnection
    size:
    suppliers
    getSuppliers Loop:suppliers
    getPurchaseRating
    ?
    

    View Slide

30. overallRating
    readConnection
    size:
    suppliers
    getSuppliers Loop:suppliers
    getPurchaseRating
    [Statistical Inference]
    New
    Code
    [Feedback Propagation]
    Predicted Entity
    

    View Slide

31. Distributed Runtime Traces
    .......
    [26/06/2015:21205.0], responseTime, “showConnections, 204
    [26/06/2015:21215.0], responseTime, “setConnectionImage, 169
    [26/06/2015:21216.0], responseTime, “PaymentService”, 79
    [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69
    [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35
    [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216]
    ........
    .......
    [26/06/2015:21205.0], responseTime, “showConnections, 204
    [26/06/2015:21215.0], responseTime, “setConnectionImage, 169
    [26/06/2015:21216.0], responseTime, “PaymentService”, 79
    [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69
    [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35
    [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216]
    ........
    .......
    [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73
    [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69
    [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35
    [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216]
    ........
    Specification
    (Feedback Mapping)
    Trace Filtering
    .......
    [26/06/2015:21205.0], responseTime, "CustomerService", 204
    [26/06/2015:21215.0], responseTime, "CustomerService", 169
    [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM2", 0.73
    [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM1", 0.69
    [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35
    [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216]
    ........
    .......
    [26/06/2015:21205.0], responseTime, "CustomerService", 204
    [26/06/2015:21215.0], responseTime, "CustomerService", 169
    [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM2", 0.73
    [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM1", 0.69
    [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35
    [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216]
    ........
    Inference on
    Code Changes
    Visualization
    (Presentation)
    Abstract Syntax Tree (AST)
    IDE Distributed Runtime Traces
    Feedback Mapping
    Runtime Metric Annotated AST
    Performance Augmented
    Source Code
    ?
    Prediction through
    Impact Analysis
    .......
    [26/06/2015:21205.0], responseTime, “showConnections, 204
    [26/06/2015:21215.0], responseTime, “setConnectionImage, 169
    [26/06/2015:21216.0], responseTime, “PaymentService”, 79
    [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69
    [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35
    [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216]
    ........
    .......
    [26/06/2015:21205.0], responseTime, “showConnections, 204
    [26/06/2015:21215.0], responseTime, “setConnectionImage, 169
    [26/06/2015:21216.0], responseTime, “PaymentService”, 79
    [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69
    [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35
    [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216]
    ........
    .......
    [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73
    [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69
    [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35
    [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216]
    ........
    

    View Slide

32. eedback Mapping
    Metric Annotated AST
    Performance Augmented
    Source Code
    ........
    Feedback Mapping
    Runtime Metric Annotated AS
    ?
    Prediction through
    Impact Analysis
    

    View Slide

33. Implementation of the framework as an Eclipse plugin
    > Performance Awareness
    > Contextualization
    33
    PerformanceHat
    (Proof-of-concept)
    

    View Slide

34. > Instant performance feedback prediction on code changes
    PerformanceHat
    (Industrial Case Study: Cloudmore)
    

    View Slide

35. First Transfer: SAP HANA WebIDE

    Performance Spotter
    

    View Slide

36. First Transfer: SAP HANA WebIDE

    Performance Spotter
    

    View Slide

37. Does it scale?
    

    View Slide

38. Matching information to source code is part
    of the build process in the IDE
    Scaling Runtime Performance Matching
    Matching information to source code is part
    of the daily developer workflow
    It cannot be slow!
    38
    

    View Slide

39. Component Analysis: Which of the components could 

    impede the development workflow?
    Every build needs to:

    > Identify and extract relevant AST nodes
    > Retrieve information for each relevant node
    > Execute inference for new nodes with unknown properties
    > Propagate predicted entity
    39
    

    View Slide

40. Component Analysis: Which of the components could 

    impede the development workflow?
    Analysis needs to take into account different scopes

    > Single File Builds
    > Incremental Builds
    > Full Project Builds
    40
    

    View Slide

41. Component Analysis: Which of the components could 

    impede the development workflow?
    Potentially problematic
    41
    

    View Slide

42. Revamped architecture to enable scalability
    Local Workstation
    Local Feedback
    Handler
    Local IDE
    Cache
    Datastore
    HTTP
    Infrastructure 1
    Deployed
    Feedback
    Handler
    Deployed
    System
    Transform
    IDE
    Infrastructure 2
    Deployed
    Feedback
    Handler
    Deployed
    System
    Transform
    Infrastructure n
    Deployed
    Feedback
    Handler
    Deployed
    System
    Transform
    ….
    HTTP
    Stream
    (Apache
    Kafka
    Broker)
    HTTP
    Specification
    Function
    Inference
    Function
    Registered
    Filters
    

    View Slide

43. Scaling Runtime Performance Matching
    Component Analysis: Measurements
    Agilefant Full EU Project Full Story Single File Controller Single File
    Cache / Cold
    Cache / Warm
    No Cache / Cold
    No Cache / Warm
    0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00
    Build Time [%]
    Scenario
    Prediction
    Marking
    Fetching
    Other
    Attaching
    

    View Slide

44. Does it help developers?
    

    View Slide

45. “Classical” software engineering user study:
    > Between Subject Study

    > 20 software engineers (min. 1 year of experience)
    > Relevant study subject: Agilefant
    > 4 Tasks (both related and not related to performance bugs)
    Control Group (Kibana) Treatment Group (PerformanceHat)
    45
    Controlled User Study
    Overview
    

    View Slide

46. H01: Given a maintenance task that would introduce a
    performance bug, software engineers using PerformanceHat
    are faster in detecting the performance bug

    H02: Given a maintenance task that would introduce a
    performance bug, software engineers using PerformanceHat
    are faster in finding the root cause of the performance bug

    H03: Given a maintenance task that is not relevant to
    performance, software engineers using PerformanceHat are
    not slower than the control group in solving the task
    [Metric: First Encounter (FE)]
    [Metric: Root-Cause Analysis (RCA)]
    [Metric: Development Time]
    46
    Controlled User Study
    Hypothesis and Metrics
    

    View Slide

47. 47
    100 200 300 400 500
    Time (in seconds)
    T1 (Total) T2 (Total) T2 (FE) T2 (RCA) T3 (Total) T4 (Total) T4 (FE) T4 (RCA)
    Control
    Treatment
    Controlled User Study
    Results
    

    View Slide

48. @citostyle
    Jürgen Cito
    Developer Targeted Runtime Analytics leverages
    runtime data by matching traces to source code
    artefacts to support data-driven decision-making
    during software development
    For which runtime aspects does it make sense 

    to correlate with source code artefacts?
    How much feedback is too much feedback?
    http://sealuzh.github.io/PerformanceHat/
    https://speakerdeck.com/citostyle
    Slides
    juergencito.com
    

    View Slide