Estimating Cloud Application Performance Based on Micro-Benchmark Profiling

Research Talk @IEEE CLOUD 2018 in San Francisco

Paper Reference:
Joel Scheuner, Philipp Leitner (2018). Estimating Cloud Application Performance Based on Micro-Benchmark Profiling in Proceedings of the 11th IEEE International Conference on Cloud Computing (CLOUD'18).

Cloud WorkBench: https://github.com/sealuzh/cloud-workbench

Abstract:
The continuing growth of the cloud computing market has led to an unprecedented diversity of cloud services. To support service selection, micro-benchmarks are commonly used to identify the best performing cloud service. However, it remains unclear how relevant these synthetic micro-benchmarks are for gaining insights into the performance of real-world applications.
Therefore, this paper develops a cloud benchmarking methodology that uses micro-benchmarks to profile applications and subsequently predicts how an application performs on a wide range of cloud services. A study with a real cloud provider (Amazon EC2) has been conducted to quantitatively evaluate the estimation model with 38 metrics from 23 micro-benchmarks and 2 applications from different domains. The results reveal remarkably low variability in cloud service performance and show that selected micro-benchmarks can estimate the duration of a scientific computing application with a relative error of less than 10% and the response time of a Web serving application with a relative error between 10% and 20%. In conclusion, this paper emphasizes the importance of cloud benchmarking by substantiating the suitability of micro-benchmarks for estimating application performance in comparison to common baselines but also highlights that only selected micro-benchmarks are relevant to estimate the performance of a particular application.

Transcript

1. Joel Scheuner ! scheuner@chalmers.se " joe4dev #@joe4dev Estimating Cloud Application Performance Based on Micro-Benchmark Profiling Joel Scheuner, Philipp Leitner Supported by
2. Context: Public Infrastructure-as-a-Service Clouds IaaS PaaS SaaS Applications Data Runtime Middleware OS Virtualization Servers Storage Networking Applications Data Runtime Middleware OS Virtualization Servers Storage Networking Applications Data Runtime Middleware OS Virtualization Servers Storage Networking User-Managed Provider-Managed Infrastructure-as-a-Service (IaaS) Platform-as-a-Service (PaaS) Software-as-a-Service (SaaS) 2018-07-02 IEEE CLOUD'18 2
3. 2018-07-02 IEEE CLOUD'18 3 Motivation: Capacity Planning in IaaS Clouds What cloud provider should I choose? https://www.cloudorado.com
4. 0 20 40 60 80 100 120 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 Number of Instance Type 2018-07-02 IEEE CLOUD'18 4 Motivation: Capacity Planning in IaaS Clouds What cloud service (i.e., instance type) should I choose? t2.nano 0.05-1 vCPU 0.5 GB RAM $0.006/h x1e.32xlarge 128 vCPUs 3904 GB RAM $26.688 hourly à Impractical to Test all Instance Types
5. 2018-07-02 IEEE CLOUD'18 5 Topic: Performance Benchmarking in the Cloud “The instance type itself is a very major tunable parameter” ! @brendangregg re:Invent’17 https://youtu.be/89fYOo1V2pA?t=5m4s
6. 2018-07-02 IEEE CLOUD'18 6 Background Generic Artificial Resource- specific Specific Real-World Resource- heterogeneous Micro Benchmarks CPU Memory I/O Network Overall performance (e.g., response time) Application Benchmarks Domain Workload Resource Usage
7. 2018-07-02 IEEE CLOUD'18 7 Problem: Isolation, Reproducibility of Execution Generic Artificial Resource-specific Specific Real-World Resource- heterogeneous Micro Benchmarks CPU Memory I/O Network Overall performance (e.g., response time) Application Benchmarks
8. 2018-07-02 IEEE CLOUD'18 8 Question Generic Artificial Resource-specific Specific Real-World Resource- heterogeneous Micro Benchmarks CPU Memory I/O Network Overall performance (e.g., response time) Application Benchmarks How relevant? ?
9. 2018-07-02 IEEE CLOUD'18 9 Research Questions PRE – Performance Variability Does the performance of equally configured cloud instances vary relevantly? RQ1 – Estimation Accuracy How accurate can a set of micro benchmarks estimate application performance? RQ2 – Micro Benchmark Selection Which subset of micro benchmarks estimates application performance most accurately?
10. 2018-07-02 IEEE CLOUD'18 10 Idea Micro Benchmarks CPU Memory I/O Network Overall performance (e.g., response time) Application Benchmarks Evaluate a Prediction Model performance cost performance cost VMN VMN
11. 2018-07-02 IEEE CLOUD'18 11 Methodology Benchmark Design
12. 2018-07-02 IEEE CLOUD'18 12 CPU • sysbench/cpu-single-thread • sysbench/cpu-multi-thread • stressng/cpu-callfunc • stressng/cpu-double • stressng/cpu-euler • stressng/cpu-ftt • stressng/cpu-fibonacci • stressng/cpu-int64 • stressng/cpu-loop • stressng/cpu-matrixprod Memory • sysbench/memory-4k-block-size • sysbench/memory-1m-block-size Broad resource coverage and specific resource testing Micro Benchmarks Micro Benchmarks CPU Memory I/O Network I/O • [file I/O] sysbench/fileio-1m-seq-write • [file I/O] sysbench/fileio-4k-rand-read • [disk I/O] fio/4k-seq-write • [disk I/O] fio/8k-rand-read Network • iperf/single-thread-bandwidth • iperf/multi-thread-bandwidth • stressng/network-epoll • stressng/network-icmp • stressng/network-sockfd • stressng/network-udp Software (OS) • sysbench/mutex • sysbench/thread-lock-1 • sysbench/thread-lock-128
13. 2018-07-02 IEEE CLOUD'18 13 Application Benchmarks Overall performance (e.g., response time) Application Benchmarks Molecular Dynamics Simulation (MDSim) WordPress Benchmark (WPBench) Multiple short blogging session scenarios (read, search, comment) 0 20 40 60 80 100 00:00 01:00 02:00 03:00 04:00 05:00 06:00 07:00 08:00 Elapsed Time [min] Number of Concurrent Threads
14. 2018-07-02 IEEE CLOUD'18 14 Methodology Benchmark Design Benchmark Execution A Cloud Benchmark Suite Combining Micro and Applications Benchmarks QUDOS@ICPE’18, Scheuner and Leitner
15. 2018-07-02 IEEE CLOUD'18 15 Execution Methodology guidelines, the authors point out that even with such guidelines, comparing multiple alternatives and conducting repeatable experi- ments on EC2 might be challenging or impossible due to the high variability. In the paper, we directly examine methodologies for conducting experiments in such environments. Despite, the high variation in the performance of cloud comput- ing systems, a number of studies [10, 4] choose to run the ex- periment only once. More rigorously conducted evaluations in- volve running experiments multiple times and reporting the aver- age value. For example, some studies [7, 12, 5, 6] use an eval- uation technique we call multiple consecutive trials (described in Section 3). In Section 4, we show how variations in the cloud en- vironment might lead to misleading results if these techniques are utilized for evaluation. We recently studied methodologies for conducting repeatable ex- periments and fair comparisons when performing performance eval- uations in WiFi networks [2]. This study [2] shows that many commonly used techniques for the experimental evaluation of WiFi networks are flawed and could result in misleading conclusions be- ing drawn. In that work, although we propose the use of random- ized multiple interleaved trials (RMIT) (described in Section 3) as a methodology for coping with changing wireless channel condi- tions, randomization was not necessary. In this paper, we examine commonly used approaches for measuring and comparing perfor- mance as well as the suitability of RMIT in a completely different scenario, namely cloud computing environments. We find that ran- domization is required, due to periodic changes in the environment, and that RMIT can be used to obtain repeatable results. Some work [3, 4] has proposed techniques to reduce the vari- ability of application performance when executing in cloud en- vironments by limiting variability of network performance. Un- fortunately, such work only reduces but does not eliminate vari- ability. Other shared resources such as disks can also cause per- formance measurements to be highly variable. One study [9] re- ports that during off-peak hours disk read bandwidths would range from 100–140 MB/sec, while during peak hours it ranged from 40– 70 MB/sec. Moreover, even with techniques to reduce variability, methodologies are still needed to ensure that differences in perfor- mance of different alternatives are due to the differences in those alternatives, rather than differences in the conditions under which they were executed. remaining alternatives. Figure 1-B shows the Multiple Consec utive Trials technique for 3 alternatives. • Multiple Interleaved Trials: One trial is conducted using th first alternative, followed by one trial with the second, and so on until each alternative has been run once. When one trial ha been conducted using each alternative we say that one round has been completed. Rounds are repeated until the appropriat number of trials has been conducted (Figure 1-C). • Randomized Multiple Interleaved Trials: If the cloud computing environment is affected at regular in tervals, and the intervening period coincides with the length o each trial, it is possible that some alternatives are affected mor than others. Therefore, the randomized multiple interleaved tri als methodology randomly reorders alternatives for each round (Figure 1-D). In essence, a randomized block design [13] i constructed where the blocks are intervals of time (rounds) and within each block all alternatives are tested, with a new random ordering of alternatives being generated for each block. A B C A B C B C B A C C B A A C B D) Randomized Multiple Interleaved Trials (RMIT) Figure 1: Different methodologies: with 3 alternatives 3.1 Methodologies used in Practice To illustrate that these methodologies are actually used in prac tice, we studied the performance evaluation methodologies used in the 38 research papers published in the ACM Symposium on Cloud Computing 2016 (SoCC’16). 9 papers conduct experimental evalu ations on public clouds (7 on Amazon EC2, 1 on Microsoft Azure and 1 on Google computing engine). We found that the single tria and multiple consecutive trials (MCT) methodologies are utilized by 7 and 4 papers respectively (some papers use both techniques) No other evaluation methodology is used in these papers. Addi tionally, 9 other papers also use these two methodologies when conducting evaluations on research clusters. 30 benchmark scenarios 3 trials ~2-3h runtime [1] A. Abedi and T. Brecht. Conducting repeatable experiments in highly variable cloud computing environments. ICPE’17 [1]
16. 2018-07-02 IEEE CLOUD'18 16 Benchmark Manager Cloud WorkBench (CWB) Tool for scheduling cloud experiments ! sealuzh/cloud-workbench Cloud Work Bench – Infrastructure-as- Code Based Cloud Benchmarking CloudCom’14, Scheuner, Leitner, Cito, and Gall Cloud WorkBench: Benchmarking IaaS Providers based on Infrastructure-as-Code Demo@WWW’15, Scheuner, Cito, Leitner, and Gall
17. 2018-07-02 IEEE CLOUD'18 17 Methodology Benchmark Design Benchmark Execution Data Pre- Processing Data Analysis 4.41 4.3 3.16 3.32 6.83 0 5 10 20 30 40 50 m1.small (eu) m1.small (us) m3.medium (eu) m3.medium (us) m3.large (eu) Configuration [Instance Type (Region)] Relative Standard Deviation (RSD) [%] A Cloud Benchmark Suite Combining Micro and Applications Benchmarks QUDOS@ICPE’18, Scheuner and Leitner Estimating Cloud Application Performance Based on Micro Benchmark Profiling CLOUD’18, Scheuner and Leitner
18. 2018-07-02 IEEE CLOUD'18 18 Performance Data Set eu + us eu + us eu * * ECU := Elastic Compute Unit (i.e., Amazon’s metric for CPU performance) >240 Virtual Machines (VMs) à 3 Iterations à ~750 VM hours >60’000 Measurements (258 per instance) PRE RQ1+2 m1.small 1 1 1.7 PV Low m1.medium 1 2 3.75 Instance Typ e vCPU ECU RAM [GiB] Virtualization Network Performance PV Moderate m3.medium 1 3 3.75 PV /HVM Moderate 2 m1.large 4 7.5 PV Moderate 2 m3.large 6.5 7.5 HVM Moderate 2 m4.large 6.5 8.0 HVM Moderate 2 c3.large 7 3.75 HVM Moderate c4.large 2 8 3.75 HVM Moderate 4 c3.xlarge 14 7.5 HVM Moderate 4 c4.xlarge 16 7.5 HVM High c1.xlarge 8 20 7 PV High
19. 2018-07-02 IEEE CLOUD'18 19 4.41 4.3 3.16 3.32 4.14 2 outliers (54% and 56%) 0 5 10 20 30 m1.small (eu) m1.small (us) m3.medium (eu) m3.medium (us) m3.large (eu) Configuration [Instance Type (Region)] Relative Standard Deviation (RSD) [%] Threads Latency Fileio Random Network Fileio Seq. mean PRE – Performance Variability Does the performance of equally configured cloud instances vary relevantly? Results
20. 2018-07-02 IEEE CLOUD'18 20 Instance Type1 (m1.small) Instance Type2 Instance Type12 (c1.xlarge) … micro1 , micro2 , …, microN app1 , app2 app1 micro1 Linear Regression Model RQ1 – Estimation Accuracy How accurate can a set of micro benchmarks estimate application performance? Approach Forward feature selection to optimize relative error
21. 2018-07-02 IEEE CLOUD'18 21 RQ1 – Estimation Accuracy How accurate can a set of micro benchmarks estimate application performance? 0 1000 2000 25 50 75 100 Sysbench − CPU Multi Thread Duration [s] WPBench Read − Response Time [ms] Instance Type m1.small m3.medium (pv) m3.medium (hvm) m1.medium m3.large m1.large c3.large m4.large c4.large c3.xlarge c4.xlarge c1.xlarge Group test train Relative Error (RE) = 12.5% !" = 99.2% Results
22. RQ2 – Micro Benchmark Selection Which subset of micro benchmarks estimates application performance most accurately? 8/25/18 Chalmers 22 Results Relative Error [%] Micro Benchmark Sysbench – CPU Multi Thread 12 Sysbench – CPU Single Thread 454 Baseline vCPUs 616 ECU 359 Cost 663 (i.e., Amazon’s metric for CPU performance)
23. 2018-07-02 IEEE CLOUD'18 23 RQ – Implications Suitability of selected micro benchmarks to estimate application performance Benchmarks cannot be used interchangeable à Configuration is important Baseline metrics vCPU and ECU are insufficient
24. 2018-07-02 IEEE CLOUD'18 24 Threats to Validity Construct Validity Almost 100% of benchmarking reports are wrong because benchmarking is "very very error-prone”1 [senior performance architect @Netflix] à Guidelines, rationalization, open source 1 https://www.youtube.com/watch?v=vm1GJMp0QN4&feature=youtu.be&t=18m29s Internal Validity the extent to which cloud environmental factors, such as multi-tenancy, evolving infrastructure, or dynamic resource limits, affect the performance level of a VM instance à Variability PRE, stop interfering process External Validity (Generalizability) Other cloud providers? Larger instance types? Other application domains? à Future work Reproducibility the extent to which the methodology and analysis is repeatable at any time for anyone and thereby leads to the same conclusions ! dynamic cloud environment à Fully automated execution, open source
25. 2018-07-02 IEEE CLOUD'18 25 Related Work [1] Athanasia Evangelinou, Michele Ciavotta, Danilo Ardagna, Aliki Kopaneli, George Kousiouris, and Theodora Varvarigou. Enterprise applications cloud rightsizing through a joint benchmarking and optimization approach. Future Generation Computer Systems, 2016 [2] Mauro Canuto, Raimon Bosch, Mario Macias, and Jordi Guitart. A methodology for full-system power modeling in heterogeneous data centers. In Proceedings of the 9th International Conference on Utility and Cloud Computing (UCC ’16), 2016 [3] Kenneth Hoste, Aashish Phansalkar, Lieven Eeckhout, Andy Georges, Lizy K. John, and Koen De Bosschere. Performance prediction based on inherent program similarity. In PACT ’06, 2006 Application Performance Prediction Application Performance Profiling • System-level resource monitoring [1,2] • Compiler-level program similarity [3] • Trace and reply with Cloud-Prophet [4,5] • Bayesian cloud configuration refinement for big data analytics [6] [4] Ang Li, Xuanran Zong, Ming Zhang, Srikanth Kandula, and Xiaowei Yang. Cloud-prophet: predicting web application performance in the cloud. ACM SIGCOMM Poster, 2011 [5] Ang Li, Xuanran Zong, Srikanth Kandula, Xiaowei Yang, and Ming Zhang. Cloud-prophet: Towards application performance prediction in cloud. In Proceedings of the ACM SIGCOMM 2011 Conference (SIGCOMM ’11), 2011 [6] Omid Alipourfard, Hongqiang Harry Liu, Jianshu Chen, Shivaram Venkataraman, Minlan Yu, and Ming Zhang. Cherrypick: Adaptively unearthing the best cloud configurations for big data analytics. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17), 2017
26. 2018-07-02 IEEE CLOUD'18 26 Conclusion Methodology Benchmark Design Benchmark Execution Data Pre- Processing Data Analysis 4.41 4.3 3.16 3.32 6.83 0 5 10 20 30 40 50 m1.small (eu) m1.small (us) m3.medium (eu) m3.medium (us) m3.large (eu) Configuration [Instance Type (Region)] Relative Standard Deviation (RSD) [%] A Cloud Benchmark Suite Combining Micro and Applications Benchmarks QUDOS@ICPE’18, Scheuner and Leitner Estimating Cloud Application Performance Based on Micro Benchmark Profiling CLOUD’18, Scheuner and Leitner RQ – Implications Suitability of selected micro benchmarks to estimate application performance Benchmarks cannot be used interchangeable à Configuration is important Baseline metrics vCPU and ECU are insufficient ! scheuner@chalmers.se " # joe4dev RQ1 – Estimation Accuracy How accurate can a set of micro benchmarks estimate application performance? 0 1000 2000 25 50 75 100 Sysbench − CPU Multi Thread Duration [s] WPBench Read − Response Time [ms] Instance Type m1.small m3.medium (pv) m3.medium (hvm) m1.medium m3.large m1.large c3.large m4.large c4.large c3.xlarge c4.xlarge c1.xlarge Group test train Relative Error (RE) = 12.5% !" = 99.2% Results RQ2 – Micro Benchmark Selection Which subset of micro benchmarks estimates application performance most accurately? Results Relative Error [%] Micro Benchmark Sysbench – CPU Multi Thread 12 Sysbench – CPU Single Thread 454 Baseline vCPUs 616 ECU 359 Cost 663 (i.e., Amazon’s metric for CPU performance) 0 20 40 60 80 100 120 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 Number of Instance Type Motivation: Capacity Planning in IaaS Clouds What cloud service (i.e., instance type) should I choose? t2.nano 0.05-1 vCPU 0.5 GB RAM $0.006/h x1e.32xlarge 128 vCPUs 3904 GB RAM $26.688 hourly à Impractical to Test all Instance Types