Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Performance Stability of Public Clouds
Search
xLeitix
April 03, 2019
Research
1
65
Performance Stability of Public Clouds
Talk given at VECS (automotive industry conference in Gothenburg)
xLeitix
April 03, 2019
Tweet
Share
More Decks by xLeitix
See All by xLeitix
CrossFit: Fine-Grained Benchmarking of Serverless Application Performance Across Cloud Providers
xleitix
0
170
Unit testing performance using code microbenchmarks - how far are we?
xleitix
0
350
Developer-Targeted Performance Engineering (ZHAW Colloquium)
xleitix
0
64
Developer-Targeted Performance Engineering
xleitix
0
200
Cachematic – Automatic Invalidation in Application-Level Caching Systems
xleitix
0
100
AWS Lambda and #serverless. What’s all the fuzz about?
xleitix
1
540
Performance Testing in a Public Cloud - How bad is it really?
xleitix
0
420
Performance Testing of and in the Cloud
xleitix
1
380
(Mis-)Adventures in Performance Engineering
xleitix
1
78
Other Decks in Research
See All in Research
大規模言語モデル (LLM) の技術と最新動向
ikuyamada
33
19k
Introducing Research Units of Matsuo-Iwasawa Laboratory
matsuolab
0
250
工学としてのSRE再訪 / Revisiting SRE as Engineering
yuukit
18
9.9k
Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve
eumesy
PRO
6
1.1k
多様かつ継続的に変化する環境に適応する情報システム/thesis-defense-presentation
monochromegane
1
370
RCEへの近道
kawakatz
1
770
医療分野におけるLLMの現状と応用可能性について
kento1109
11
3.2k
ブラックボックス機械学習モデルの判断根拠を説明する技術
yuyay
0
230
Physics of Language Models: Part 3.1, Knowledge Storage and Extraction
sosk
1
850
日本語医療LLM評価ベンチマークの構築と性能分析
fta98
3
460
出生抑制策と少子化
morimasao16
0
410
3次元点群の分類における評価指標について
kentaitakura
0
170
Featured
See All Featured
Large-scale JavaScript Application Architecture
addyosmani
509
110k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
246
1.3M
In The Pink: A Labor of Love
frogandcode
139
22k
[RailsConf 2023] Rails as a piece of cake
palkan
49
4.7k
Fantastic passwords and where to find them - at NoRuKo
philnash
50
2.8k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
227
52k
Optimizing for Happiness
mojombo
375
69k
Designing with Data
zakiwarfel
98
5.1k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
3
77
Adopting Sorbet at Scale
ufuk
73
8.9k
Fashionably flexible responsive web design (full day workshop)
malarkey
402
65k
BBQ
matthewcrist
83
9.2k
Transcript
Performance Stability of (Public) Clouds Philipp Leitner
[email protected]
@xLeitix
Chalmers !2 Cloud Computing Image Credit: https://www.networkworld.com/article/3195527/did-cloud-kill-backup.html
Chalmers !3 Some disclaimers before we get started …. Image
Credit: https://thenounproject.com/term/exclamation-mark/
Chalmers !4 Image Credit: https://nordicapis.com/living-in-the-cloud-stack-understanding-saas-paas-and-iaas-apis/
Chalmers !5 Cloud Usage in Automotive Source (Accenture): https://www.accenture.com/t20150914T170053__w__/us-en/_acnmedia/Accenture/Conversion-Assets/DotCom/Documents/Global/PDF/ Industries_18/Accenture-Cloud-Automative-PoV.pdf
Chalmers !6 Cloud Usage in Automotive Source (Accenture): https://www.accenture.com/t20150914T170053__w__/us-en/_acnmedia/Accenture/Conversion-Assets/DotCom/Documents/Global/PDF/ Industries_18/Accenture-Cloud-Automative-PoV.pdf
Chalmers !7 (One) Challenge for Cloud Adoption in Automotive: Predictability
Image Credit: http://chittagongit.com
Chalmers !8 Predictability Do I know what I will get?
Do I get the same thing every time? Image Credit: http://chittagongit.com
Chalmers !9 Aside: Cloud Instance Types (“flavors”) Image Credit (Rightscale):
https://www.rightscale.com/about-cloud-management/cloud-cost-optimization/cloud-pricing-comparison
Chalmers !10 Predictability Inter-Instance Intra-Instance
Chalmers !11 Predictability Inter-Instance Intra-Instance
Chalmers !12 Predictability Inter-Instance Intra-Instance
Chalmers !13 Source (Leitner and Cito): https://arxiv.org/pdf/1411.2429.pdf
Chalmers !14 Relative Standard Deviations Benchmarks of identical instances Source
(Leitner and Cito): https://arxiv.org/pdf/1411.2429.pdf (anno ~ 2015)
Chalmers !15 Recent Results (unpublished data) (Feb 2019) 2015
Chalmers !16 Instance Runtime (unpublished data) (Feb 2019) 2015 3.5
4.0 4.5 5.0 5.5 0 20 40 60 Benchmark Runtime [h] Benchmark Value Continuous io azure D2s
Chalmers !17 Changes Over the Years (mean of all measurements)
Chalmers !18 2015 2019 CPU 8.1 3.6 - 55% Changes
Over the Years (mean of all measurements)
Chalmers !19 2015 2019 CPU 8.1 3.6 - 55% MEM
12.6 6.5 - 48% Changes Over the Years (mean of all measurements)
Chalmers !20 2015 2019 CPU 8.1 3.6 - 55% MEM
12.6 6.5 - 48% IO 38.6 15.9 - 59% Changes Over the Years (mean of all measurements)
Chalmers !21 What has changed?
Chalmers !22 For IO: multi-tenancy For CPU: hardware heterogeneity Traditional
Reasons for Lack of Predictability
Chalmers !23 Reason 0: More commitment to predictable performance levels
and transparency
Chalmers !24 Reason 1: Move towards guaranteed hardware
Chalmers !25 (anno ~ 2015) Heterogenous Hardware? Source (Leitner and
Cito): https://arxiv.org/pdf/1411.2429.pdf
Chalmers !26 (anno ~ 2015) Heterogenous Hardware? Source (Leitner and
Cito): https://arxiv.org/pdf/1411.2429.pdf (now) (Largely) guaranteed hardware
Chalmers !27 Reason 2: Move towards SLAs and credit systems
over best-effort delivery
Chalmers !28 (anno ~ 2015) Best-Effort Delivery? Source (Leitner and
Cito): https://arxiv.org/pdf/1411.2429.pdf
Chalmers !29 (anno ~ 2015) Best-Effort Delivery? Source (Leitner and
Cito): https://arxiv.org/pdf/1411.2429.pdf (unpublished data) (now) 0 5 10 15 20 25 0 50 100 150 200 Benchmark Runtime [h] Benchmark Value c5−large / IO
Chalmers !30 Credit Models - General Idea Resources are distributed
fairly between tenants based on usage tokens Available for: CPU (in case of shared CPU instance types) IO (some providers)
Chalmers !31 Credit Models at Runtime Source (Leitner and Scheuner):
https://www.zora.uzh.ch/id/eprint/112940/
Chalmers !32 Summary Public clouds are not all that unpredictable
(anymore)
Chalmers !33 Summary Public clouds are not all that unpredictable
(anymore) … useful even for workloads sensitive to performance variation … but it’s still virtualized infrastructure
Chalmers !34 Summary Public clouds are not all that unpredictable
(anymore) New developments have changed the game: Specialized hardware, credit models, provisioned IOPS
Chalmers !35 Cloud Workbench Tool for scheduling cloud experiments Code:
https://github.com/sealuzh/cloud-workbench Demo: https://www.youtube.com/watch? v=0yGFGvHvobk
Chalmers !36 Questions? Source: https://dilbert.com/strip/2008-05-08