Performance Stability of Public Clouds

Slide 1

Slide 1 text

Performance Stability of (Public) Clouds Philipp Leitner [email protected] @xLeitix

Slide 2

Slide 2 text

Chalmers !2 Cloud Computing Image Credit: https://www.networkworld.com/article/3195527/did-cloud-kill-backup.html

Slide 3

Slide 3 text

Chalmers !3 Some disclaimers before we get started …. Image Credit: https://thenounproject.com/term/exclamation-mark/

Slide 4

Slide 4 text

Chalmers !4 Image Credit: https://nordicapis.com/living-in-the-cloud-stack-understanding-saas-paas-and-iaas-apis/

Slide 5

Slide 5 text

Chalmers !5 Cloud Usage in Automotive Source (Accenture): https://www.accenture.com/t20150914T170053__w__/us-en/_acnmedia/Accenture/Conversion-Assets/DotCom/Documents/Global/PDF/ Industries_18/Accenture-Cloud-Automative-PoV.pdf

Slide 6

Slide 6 text

Chalmers !6 Cloud Usage in Automotive Source (Accenture): https://www.accenture.com/t20150914T170053__w__/us-en/_acnmedia/Accenture/Conversion-Assets/DotCom/Documents/Global/PDF/ Industries_18/Accenture-Cloud-Automative-PoV.pdf

Slide 7

Slide 7 text

Chalmers !7 (One) Challenge for Cloud Adoption in Automotive: Predictability Image Credit: http://chittagongit.com

Slide 8

Slide 8 text

Chalmers !8 Predictability Do I know what I will get? Do I get the same thing every time? Image Credit: http://chittagongit.com

Slide 9

Slide 9 text

Chalmers !9 Aside: Cloud Instance Types (“flavors”) Image Credit (Rightscale): https://www.rightscale.com/about-cloud-management/cloud-cost-optimization/cloud-pricing-comparison

Slide 10

Slide 10 text

Chalmers !10 Predictability Inter-Instance Intra-Instance

Slide 11

Slide 11 text

Chalmers !11 Predictability Inter-Instance Intra-Instance

Slide 12

Slide 12 text

Chalmers !12 Predictability Inter-Instance Intra-Instance

Slide 13

Slide 13 text

Chalmers !13 Source (Leitner and Cito): https://arxiv.org/pdf/1411.2429.pdf

Slide 14

Slide 14 text

Chalmers !14 Relative Standard Deviations Benchmarks of identical instances Source (Leitner and Cito): https://arxiv.org/pdf/1411.2429.pdf (anno ~ 2015)

Slide 15

Slide 15 text

Chalmers !15 Recent Results (unpublished data) (Feb 2019) 2015

Slide 16

Slide 16 text

Chalmers !16 Instance Runtime (unpublished data) (Feb 2019) 2015 3.5 4.0 4.5 5.0 5.5 0 20 40 60 Benchmark Runtime [h] Benchmark Value Continuous io azure D2s

Slide 17

Slide 17 text

Chalmers !17 Changes Over the Years (mean of all measurements)

Slide 18

Slide 18 text

Chalmers !18 2015 2019 CPU 8.1 3.6 - 55% Changes Over the Years (mean of all measurements)

Slide 19

Slide 19 text

Chalmers !19 2015 2019 CPU 8.1 3.6 - 55% MEM 12.6 6.5 - 48% Changes Over the Years (mean of all measurements)

Slide 20

Slide 20 text

Chalmers !20 2015 2019 CPU 8.1 3.6 - 55% MEM 12.6 6.5 - 48% IO 38.6 15.9 - 59% Changes Over the Years (mean of all measurements)

Slide 21

Slide 21 text

Chalmers !21 What has changed?

Slide 22

Slide 22 text

Chalmers !22 For IO: multi-tenancy For CPU: hardware heterogeneity Traditional Reasons for Lack of Predictability

Slide 23

Slide 23 text

Chalmers !23 Reason 0: More commitment to predictable performance levels and transparency

Slide 24

Slide 24 text

Chalmers !24 Reason 1: Move towards guaranteed hardware

Slide 25

Slide 25 text

Chalmers !25 (anno ~ 2015) Heterogenous Hardware? Source (Leitner and Cito): https://arxiv.org/pdf/1411.2429.pdf

Slide 26

Slide 26 text

Chalmers !26 (anno ~ 2015) Heterogenous Hardware? Source (Leitner and Cito): https://arxiv.org/pdf/1411.2429.pdf (now) (Largely) guaranteed hardware

Slide 27

Slide 27 text

Chalmers !27 Reason 2: Move towards SLAs and credit systems over best-effort delivery

Slide 28

Slide 28 text

Chalmers !28 (anno ~ 2015) Best-Effort Delivery? Source (Leitner and Cito): https://arxiv.org/pdf/1411.2429.pdf

Slide 29

Slide 29 text

Chalmers !29 (anno ~ 2015) Best-Effort Delivery? Source (Leitner and Cito): https://arxiv.org/pdf/1411.2429.pdf (unpublished data) (now) 0 5 10 15 20 25 0 50 100 150 200 Benchmark Runtime [h] Benchmark Value c5−large / IO

Slide 30

Slide 30 text

Chalmers !30 Credit Models - General Idea Resources are distributed fairly between tenants based on usage tokens Available for: CPU (in case of shared CPU instance types) IO (some providers)

Slide 31

Slide 31 text

Chalmers !31 Credit Models at Runtime Source (Leitner and Scheuner): https://www.zora.uzh.ch/id/eprint/112940/

Slide 32

Slide 32 text

Chalmers !32 Summary Public clouds are not all that unpredictable (anymore)

Slide 33

Slide 33 text

Chalmers !33 Summary Public clouds are not all that unpredictable (anymore) … useful even for workloads sensitive to performance variation … but it’s still virtualized infrastructure

Slide 34

Slide 34 text

Chalmers !34 Summary Public clouds are not all that unpredictable (anymore) New developments have changed the game: Specialized hardware, credit models, provisioned IOPS

Slide 35

Slide 35 text

Chalmers !35 Cloud Workbench Tool for scheduling cloud experiments Code: https://github.com/sealuzh/cloud-workbench Demo: https://www.youtube.com/watch? v=0yGFGvHvobk

Slide 36

Slide 36 text

Chalmers !36 Questions? Source: https://dilbert.com/strip/2008-05-08