Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Runtime Metric Meets Developer - Building Better Cloud Applications using Feedback

559da0ff5e64b92aaa5ae354236d1329?s=47 Jürgen Cito
October 28, 2015

Runtime Metric Meets Developer - Building Better Cloud Applications using Feedback

Conference talk at SPLASH'15 Onward! on our paper "Runtime Metric Meets Developer"

Video of the conference talk: https://www.youtube.com/watch?v=mzHoNcP6fGA
Preprint of the paper: https://peerj.com/preprints/985/

559da0ff5e64b92aaa5ae354236d1329?s=128

Jürgen Cito

October 28, 2015
Tweet

More Decks by Jürgen Cito

Other Decks in Research

Transcript

  1. Runtime Metric Meets Developer Building Better Cloud Applications Using Feedback

    Jürgen Cito, Philipp Leitner, Harald C. Gall (University of Zurich) Aryan Dadashi, Anne Keller, Andreas Roth (SAP SE) SPLASH Pittsburgh 2015 - Onward
  2. None
  3. Continuous Delivery

  4. Continuous Delivery High velocity of change, push of smaller changes

    more frequently
  5. Continuous Delivery High velocity of change, push of smaller changes

    more frequently ~50-60 releases per day
  6. Continuous Delivery High velocity of change, push of smaller changes

    more frequently ~50-60 releases per day ~100 releases per day
  7. Study on Software Development for the Cloud “The Making of

    Cloud Applications” - FSE’15 Adapted from https://xkcd.com/1423/ Interview with 25 software developers that deploy in the cloud Developer Me
  8. Study on Software Development for the Cloud “The Making of

    Cloud Applications” - FSE’15 Adapted from https://xkcd.com/1423/ Topic: Solving problems that have been detected in production Nah, I rather go by intuition? Do you look at any metrics?
  9. Log Overload

  10. State-of-the-Art NewRelic ELK Stack

  11. State-of-the-Art NewRelic ELK Stack

  12. State-of-the-Art NewRelic ELK Stack

  13. State-of-the-Art NewRelic ELK Stack

  14. State-of-the-Art NewRelic ELK Stack

  15. State-of-the-Art NewRelic ELK Stack

  16. State-of-the-Art NewRelic ELK Stack

  17. Feedback Driven Development

  18. Feedback Driven Development Feedback Visualization Feedback Prediction

  19. Feedback Driven Development Feedback Visualization Feedback Prediction

  20. Feedback Driven Development Feedback Visualization Feedback Prediction integrating metrics into

    daily developer workflows
  21. Feedback Driven Development Feedback Visualization Feedback Prediction integrating metrics into

    daily developer workflows bring metrics into context
  22. Feedback Driven Development Feedback Visualization Feedback Prediction integrating metrics into

    daily developer workflows prevent bad things from happening bring metrics into context
  23. Conceptual Overview

  24. Code Artifacts Conceptual Overview

  25. Code Artifacts Deployment Conceptual Overview

  26. Code Artifacts Deployment Conceptual Overview Cloud Infrastructure

  27. Code Artifacts Deployment Conceptual Overview Cloud Infrastructure VM1 VM2 Supplier

    Service User Interface Purch S
  28. Code Artifacts Deployment observe Conceptual Overview Cloud Infrastructure VM1 VM2

    Supplier Service User Interface Purch S
  29. Code Artifacts Deployment observe Conceptual Overview Cloud Infrastructure VM1 VM2

    Supplier Service User Interface Purch S
  30. Code Artifacts Deployment ....... [26/06/2015:21205.0], responseTime, "CustomerService", 204 [26/06/2015:21215.0], responseTime,

    "CustomerService", 169 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ Operations Data observe Conceptual Overview Cloud Infrastructure VM1 VM2 Supplier Service User Interface Purch S
  31. Code Artifacts Deployment ....... [26/06/2015:21205.0], responseTime, "CustomerService", 204 [26/06/2015:21215.0], responseTime,

    "CustomerService", 169 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ Operations Data observe Conceptual Overview Cloud Infrastructure VM1 VM2 Supplier Service User Interface Purch S
  32. Code Artifacts Deployment ....... [26/06/2015:21205.0], responseTime, "CustomerService", 204 [26/06/2015:21215.0], responseTime,

    "CustomerService", 169 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ Operations Data observe Conceptual Overview Cloud Infrastructure VM1 VM2 Supplier Service User Interface Purch S
  33. Code Artifacts Deployment ....... [26/06/2015:21205.0], responseTime, "CustomerService", 204 [26/06/2015:21215.0], responseTime,

    "CustomerService", 169 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ Operations Data observe readConnecti on getConnectio ns ids connectionPo ol readConnecti on getConnectio ns ids connectionPo ol readConnection getConnections ids connectionPool Feedback Annotated Dependency Graph Conceptual Overview Cloud Infrastructure VM1 VM2 Supplier Service User Interface Purch S
  34. (a) Code in development environment (c) Annotated dependency graph combining

    code artefacts and operations data and creating feedback readConnections getConnections showConnections setConnectionImage setConnectionStatus ....... [26/06/2015:21205.0], responseTime, “showConnections, 204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35 [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216] ........ (b) Operations data gathered through monitoring getImage (d) Feedback Visualization in the IDE Feedback Mapping
  35. None
  36. readConnections

  37. readConnections connections getConnections

  38. readConnections connections getConnections getImage setConnectionImage setConnectionStatus

  39. readConnections connections getConnections getImage setConnectionImage setConnectionStatus ....... [26/06/2015:21205.0], responseTime, “showConnections,

    204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35 [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216] ........
  40. readConnections connections getConnections getImage setConnectionImage setConnectionStatus ....... [26/06/2015:21205.0], responseTime, “showConnections,

    204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35 [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216] ........
  41. readConnections connections getConnections getImage setConnectionImage setConnectionStatus ....... [26/06/2015:21205.0], responseTime, “showConnections,

    204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35 [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216] ........
  42. readConnections connections getConnections getImage setConnectionImage setConnectionStatus ....... [26/06/2015:21205.0], responseTime, “showConnections,

    204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35 [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216] ........ Feedback Specification
  43. readConnections connections getConnections getImage setConnectionImage setConnectionStatus ....... [26/06/2015:21205.0], responseTime, “showConnections,

    204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35 [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216] ........ Feedback Specification
  44. readConnections connections getConnections getImage setConnectionImage setConnectionStatus ....... [26/06/2015:21205.0], responseTime, “showConnections,

    204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35 [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216] ........ Feedback Specification
  45. readConnections connections getConnections getImage setConnectionImage setConnectionStatus ....... [26/06/2015:21205.0], responseTime, “showConnections,

    204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35 [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216] ........ Feedback Specification
  46. readConnections connections getConnections getImage setConnectionImage setConnectionStatus ....... [26/06/2015:21205.0], responseTime, “showConnections,

    204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35 [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216] ........ Feedback Specification
  47. readConnections connections getConnections getImage setConnectionImage setConnectionStatus ....... [26/06/2015:21205.0], responseTime, “showConnections,

    204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35 [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216] ........ Feedback Specification
  48. readConnections connections getConnections getImage setConnectionImage setConnectionStatus ....... [26/06/2015:21205.0], responseTime, “showConnections,

    204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35 [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216] ........ Feedback Specification
  49. Feedback Visualization (SAP HANA)

  50. Performance Spotter

  51. Performance Spotter

  52. Performance Spotter

  53. Code Change overallRating readConnection New Code (a) Code changed by

    developer (b) Dependency graph before and after the code change (existing feedback is already attached) overallRating readConnection size: suppliers getSuppliers Loop:suppliers getPurchaseRating overallRating readConnection size: suppliers getSuppliers Loop:suppliers getPurchaseRating Inferred Critical Entity [Feedback Propagation] [Statistical Inference] (c) Prediction through Statistical Inference of new nodes without feedback and Feedback Propagation of the changes (d) Feedback Prediction in the IDE ? Live Performance Prediction
  54. New Code

  55. New Code overallRating readConnection

  56. New Code overallRating readConnection Code Change

  57. New Code overallRating readConnection Code Change overallRating readConnection size: suppliers

    getSuppliers Loop:suppliers getPurchaseRating
  58. New Code overallRating readConnection Code Change overallRating readConnection size: suppliers

    getSuppliers Loop:suppliers getPurchaseRating ?
  59. New Code

  60. overallRating readConnection size: suppliers getSuppliers Loop:suppliers getPurchaseRating New Code

  61. overallRating readConnection size: suppliers getSuppliers Loop:suppliers getPurchaseRating New Code

  62. overallRating readConnection size: suppliers getSuppliers Loop:suppliers getPurchaseRating New Code

  63. overallRating readConnection size: suppliers getSuppliers Loop:suppliers getPurchaseRating [Statistical Inference] New

    Code
  64. overallRating readConnection size: suppliers getSuppliers Loop:suppliers getPurchaseRating [Statistical Inference] New

    Code
  65. overallRating readConnection size: suppliers getSuppliers Loop:suppliers getPurchaseRating [Statistical Inference] New

    Code Predicted Entity
  66. overallRating readConnection size: suppliers getSuppliers Loop:suppliers getPurchaseRating [Statistical Inference] New

    Code Predicted Entity
  67. overallRating readConnection size: suppliers getSuppliers Loop:suppliers getPurchaseRating [Statistical Inference] New

    Code Predicted Entity
  68. overallRating readConnection size: suppliers getSuppliers Loop:suppliers getPurchaseRating [Statistical Inference] New

    Code [Feedback Propagation] Predicted Entity
  69. Live Feedback Prediction (PerformanceHat)

  70. PerformanceHat

  71. PerformanceHat

  72. PerformanceHat

  73. Challenges

  74. Confounding Factors

  75. Confounding Factors

  76. Confounding Factors

  77. Real-Time Prediction

  78. Real-Time Prediction computationally expensive for non-trivial programs

  79. Real-Time Prediction computationally expensive for non-trivial programs technically challenging
 (stream

    processing)
  80. Conservative implementation of Live Programming Impact of change in cloud

    runtimes (Scaling, Cost, etc.) Prediction in combination with Refactoring and Diagnosis Overlap with data driven IDEs (Onward! Keynote) Outlook @citostyle Jürgen Cito <cito@ifi.uzh.ch> Slides: speakerdeck.com/citostyle “Modern software is all about data. Development environments should be, too.”