Runtime Metric Meets Developer Building Better Cloud Applications Using Feedback Jürgen Cito, Philipp Leitner, Harald C. Gall (University of Zurich) Aryan Dadashi, Anne Keller, Andreas Roth (SAP SE) SPLASH Pittsburgh 2015 - Onward 

No content

Continuous Delivery 

Continuous Delivery High velocity of change, push of smaller changes more frequently 

Continuous Delivery High velocity of change, push of smaller changes more frequently ~50-60 releases per day 

Continuous Delivery High velocity of change, push of smaller changes more frequently ~50-60 releases per day ~100 releases per day 

Study on Software Development for the Cloud “The Making of Cloud Applications” - FSE’15 Adapted from https://xkcd.com/1423/ Interview with 25 software developers that deploy in the cloud Developer Me 

Study on Software Development for the Cloud “The Making of Cloud Applications” - FSE’15 Adapted from https://xkcd.com/1423/ Topic: Solving problems that have been detected in production Nah, I rather go by intuition? Do you look at any metrics? 

Log Overload 

State-of-the-Art NewRelic ELK Stack 

State-of-the-Art NewRelic ELK Stack 

State-of-the-Art NewRelic ELK Stack 

State-of-the-Art NewRelic ELK Stack 

State-of-the-Art NewRelic ELK Stack 

State-of-the-Art NewRelic ELK Stack 

State-of-the-Art NewRelic ELK Stack 

Feedback Driven Development 

Feedback Driven Development Feedback Visualization Feedback Prediction 

Feedback Driven Development Feedback Visualization Feedback Prediction 

Feedback Driven Development Feedback Visualization Feedback Prediction integrating metrics into daily developer workflows 

Feedback Driven Development Feedback Visualization Feedback Prediction integrating metrics into daily developer workflows bring metrics into context 

Feedback Driven Development Feedback Visualization Feedback Prediction integrating metrics into daily developer workflows prevent bad things from happening bring metrics into context 

Conceptual Overview 

Code Artifacts Conceptual Overview 

Code Artifacts Deployment Conceptual Overview 

Code Artifacts Deployment Conceptual Overview Cloud Infrastructure 

Code Artifacts Deployment Conceptual Overview Cloud Infrastructure VM1 VM2 Supplier Service User Interface Purch S 

Code Artifacts Deployment observe Conceptual Overview Cloud Infrastructure VM1 VM2 Supplier Service User Interface Purch S 

Code Artifacts Deployment observe Conceptual Overview Cloud Infrastructure VM1 VM2 Supplier Service User Interface Purch S 

Code Artifacts Deployment ....... [26/06/2015:21205.0], responseTime, "CustomerService", 204 [26/06/2015:21215.0], responseTime, "CustomerService", 169 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ Operations Data observe Conceptual Overview Cloud Infrastructure VM1 VM2 Supplier Service User Interface Purch S 

Code Artifacts Deployment ....... [26/06/2015:21205.0], responseTime, "CustomerService", 204 [26/06/2015:21215.0], responseTime, "CustomerService", 169 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ Operations Data observe Conceptual Overview Cloud Infrastructure VM1 VM2 Supplier Service User Interface Purch S 

Code Artifacts Deployment ....... [26/06/2015:21205.0], responseTime, "CustomerService", 204 [26/06/2015:21215.0], responseTime, "CustomerService", 169 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ Operations Data observe Conceptual Overview Cloud Infrastructure VM1 VM2 Supplier Service User Interface Purch S 

Code Artifacts Deployment ....... [26/06/2015:21205.0], responseTime, "CustomerService", 204 [26/06/2015:21215.0], responseTime, "CustomerService", 169 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, "CustomerServiceVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ Operations Data observe readConnecti on getConnectio ns ids connectionPo ol readConnecti on getConnectio ns ids connectionPo ol readConnection getConnections ids connectionPool Feedback Annotated Dependency Graph Conceptual Overview Cloud Infrastructure VM1 VM2 Supplier Service User Interface Purch S 

(a) Code in development environment (c) Annotated dependency graph combining code artefacts and operations data and creating feedback readConnections getConnections showConnections setConnectionImage setConnectionStatus ....... [26/06/2015:21205.0], responseTime, “showConnections, 204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35 [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216] ........ (b) Operations data gathered through monitoring getImage (d) Feedback Visualization in the IDE Feedback Mapping 

No content

readConnections 

readConnections connections getConnections 

readConnections connections getConnections getImage setConnectionImage setConnectionStatus 

readConnections connections getConnections getImage setConnectionImage setConnectionStatus ....... [26/06/2015:21205.0], responseTime, “showConnections, 204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35 [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216] ........ 

readConnections connections getConnections getImage setConnectionImage setConnectionStatus ....... [26/06/2015:21205.0], responseTime, “showConnections, 204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35 [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216] ........ 

readConnections connections getConnections getImage setConnectionImage setConnectionStatus ....... [26/06/2015:21205.0], responseTime, “showConnections, 204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35 [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216] ........ 

readConnections connections getConnections getImage setConnectionImage setConnectionStatus ....... [26/06/2015:21205.0], responseTime, “showConnections, 204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35 [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216] ........ Feedback Specification 

readConnections connections getConnections getImage setConnectionImage setConnectionStatus ....... [26/06/2015:21205.0], responseTime, “showConnections, 204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35 [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216] ........ Feedback Specification 

readConnections connections getConnections getImage setConnectionImage setConnectionStatus ....... [26/06/2015:21205.0], responseTime, “showConnections, 204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35 [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216] ........ Feedback Specification 

readConnections connections getConnections getImage setConnectionImage setConnectionStatus ....... [26/06/2015:21205.0], responseTime, “showConnections, 204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35 [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216] ........ Feedback Specification 

readConnections connections getConnections getImage setConnectionImage setConnectionStatus ....... [26/06/2015:21205.0], responseTime, “showConnections, 204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35 [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216] ........ Feedback Specification 

readConnections connections getConnections getImage setConnectionImage setConnectionStatus ....... [26/06/2015:21205.0], responseTime, “showConnections, 204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35 [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216] ........ Feedback Specification 

readConnections connections getConnections getImage setConnectionImage setConnectionStatus ....... [26/06/2015:21205.0], responseTime, “showConnections, 204 [26/06/2015:21215.0], responseTime, “setConnectionImage, 169 [26/06/2015:21216.0], responseTime, “PaymentService”, 79 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, "CustomerServiceVM1", 0.35 [26/06/2015:21219.4], ids, "ids", [1,16,32,189,216] ........ ....... [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM2", 0.73 [26/06/2015:21216.0], cpuUtilization, “ConnectionsVM1", 0.69 [26/06/2015:21216.1], vmBilled, “PaymentServiceVM, 0.35 [26/06/2015:21219.4], ids, “connectionIDs, [1,16,32,189,216] ........ Feedback Specification 

Feedback Visualization (SAP HANA) 

Performance Spotter 

Performance Spotter 

Performance Spotter 

Code Change overallRating readConnection New Code (a) Code changed by developer (b) Dependency graph before and after the code change (existing feedback is already attached) overallRating readConnection size: suppliers getSuppliers Loop:suppliers getPurchaseRating overallRating readConnection size: suppliers getSuppliers Loop:suppliers getPurchaseRating Inferred Critical Entity [Feedback Propagation] [Statistical Inference] (c) Prediction through Statistical Inference of new nodes without feedback and Feedback Propagation of the changes (d) Feedback Prediction in the IDE ? Live Performance Prediction 

New Code 

New Code overallRating readConnection 

New Code overallRating readConnection Code Change 

New Code overallRating readConnection Code Change overallRating readConnection size: suppliers getSuppliers Loop:suppliers getPurchaseRating 

New Code overallRating readConnection Code Change overallRating readConnection size: suppliers getSuppliers Loop:suppliers getPurchaseRating ? 

New Code 

overallRating readConnection size: suppliers getSuppliers Loop:suppliers getPurchaseRating New Code 

overallRating readConnection size: suppliers getSuppliers Loop:suppliers getPurchaseRating New Code 

overallRating readConnection size: suppliers getSuppliers Loop:suppliers getPurchaseRating New Code 

overallRating readConnection size: suppliers getSuppliers Loop:suppliers getPurchaseRating [Statistical Inference] New Code 

overallRating readConnection size: suppliers getSuppliers Loop:suppliers getPurchaseRating [Statistical Inference] New Code 

overallRating readConnection size: suppliers getSuppliers Loop:suppliers getPurchaseRating [Statistical Inference] New Code Predicted Entity 

overallRating readConnection size: suppliers getSuppliers Loop:suppliers getPurchaseRating [Statistical Inference] New Code Predicted Entity 

overallRating readConnection size: suppliers getSuppliers Loop:suppliers getPurchaseRating [Statistical Inference] New Code Predicted Entity 

overallRating readConnection size: suppliers getSuppliers Loop:suppliers getPurchaseRating [Statistical Inference] New Code [Feedback Propagation] Predicted Entity 

Live Feedback Prediction (PerformanceHat) 

PerformanceHat 

PerformanceHat 

PerformanceHat 

Challenges 

Confounding Factors 

Confounding Factors 

Confounding Factors 

Real-Time Prediction 

Real-Time Prediction computationally expensive for non-trivial programs 

Real-Time Prediction computationally expensive for non-trivial programs technically challenging
 (stream processing) 

Conservative implementation of Live Programming Impact of change in cloud runtimes (Scaling, Cost, etc.) Prediction in combination with Refactoring and Diagnosis Overlap with data driven IDEs (Onward! Keynote) Outlook @citostyle Jürgen Cito Slides: speakerdeck.com/citostyle “Modern software is all about data. Development environments should be, too.”