Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Reliability of Distributed Systems
Search
Piyush Verma
June 22, 2019
Technology
360
0
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Reliability of Distributed Systems
Piyush Verma
June 22, 2019
Transcript
70 slides
Reliability of Distributed Systems - Piyush Verma
Every product either dies a hero or lives long enough
to hit Reliability issues. 2
Customer Empathy No Chooran. Cost to Everything Architectures adapt to
$ Priority 01 02 03 04 3 Take it and Go
Service receives SMS User sends SMS Remind me to buy
milk at 6:30 PM to 53308 Cron gets Activated when time is right. Call the User 4 Sample Product
Service receives SMS User sends SMS Remind me to buy
milk at 6:30 PM to 53308 Cron gets Activated when time is right. Call the User 5 Sample Product: Inbound
Cron gets Activated when time is right. Call the User
6 Sample Product: Outbound Service receives SMS User sends SMS Remind me to buy milk at 6:30 PM to 53308
7 Inbound Connection
— Leslie Lamport https://www.microsoft.com/en-us/research/uploads/prod/2016 /12/Distribution.pdf “A distributed system is one
in which the failure of a computer you didn’t even know existed can render your own computer unusable” 8
Four Flavors of Failure 9 Disk Network CPU Memory
Network is Reliable Intra-LAN latency is ~ Zero Network is
Homogeneous Network cost is Zero 01 02 03 04 10
11 Scope of Failures: Again
At-least one server is online All servers are below 100%
All servers are responding within x ms. All of the above. 01 02 03 04 12
13 #1 Server is Unavailable
14 Replication Available
15 Replication Available
16 Available != Load Balanced
17 Load Balanced
18 Architecture of a Balancer
GCP AWS On-Prem Azure Who uses What? 19
Trilemma 20
Trilemma 21 Available Economical Endurable
22 Available + Load Balanced
Load Balancing 23
24 Monty Hall Problem: Was Marilyn vos Savant, right?
25 Server-side Load Balancing Example: Fabio
26 Look-aside Load Balancing Example: Consul/ DNS
27 Client-side Load Balancing Example: Ribbon
28 Client-side Load Balancing Example: Ribbon + Curator
29 Problems
30 Load Shedding
— Tyler McMullen https://www.infoq.com/presentations/load-balancing/ “Load Balancing is almost Impossible” 31
Alternate Reliability 32
33 Asynchronous Architectures
34 Asynchronous Architectures Example: RabbitMQ Kafka Kinesis SQS
Cron gets Activated when time is right. Call the User
35 Sample Product: Outbound Service receives SMS User sends SMS Remind me to buy milk at 6:30 PM to 53308 Part 1
36 Outbound
37 Scope of Failure: Outbound
38 Retries
39 Retries: Transient Failures
40 Exponential Backoff: Short term Transient Failures ✋ ✋ ✋
✋ ✋
41 Circuit Breaking: Long Term Transient Failures
42 Revisited
Dilemma 43 At-least Once Exactly-Once At-most Once
44 At-most once delivery
45 At-least once delivery
46 Exactly once delivery
47 Exactly once delivery = At-least-once Delivery + Exactly-once Processing
Keys to Only-Once delivery 48 Atomic Window Idempotent
Out-of-Order delivery 49
Revisit 50
Service receives SMS User sends SMS Remind me to buy
milk at 6:30 PM to 53308 Cron gets Activated when time is right. Call the User 51 Sample Product
52 Problems of State
53 Problems of State
54 Locked /Serialization
55 Master/ Master/ Slave
56 Clustering
Scalability 57 Data Replication Reduced Communication Logic/Data Decentralization
CAP Theorem [Sab topi pehna rahe] 58
59 Trilemma Available Partition Consistent
PACeLC Theorem 60
Dilemma 61 Consistency Latency
62 Revised Flow
63 What about Spanner? What about Calvin?
Reliable System 64 Scalable Correct Transparent
Access Transparency Location Transparency Concurrency Transparency Failure Transparency 01 02
03 04 65
Size Scalability Geographical Scalability 01 02 66
Summary 67 Consistent Available Economical Low Latency
68 All Put Together
Embrace your Bugs No Silver Bullet Cost to Everything Product
First 01 02 03 04 69
Does anyone have any questions?
[email protected]
Thanks 70
More Decks by Piyush Verma
See All by Piyush Verma
SLOs that Lie
meson10
0
160
Doing SRE the right way - 2
meson10
0
190
Doing SRE the right way
meson10
0
1k
Observability and Control Theory
meson10
1
1.2k
Reliability
meson10
0
220
My TLS was broken
meson10
0
190
Technology that builds Organizations
meson10
0
170
Namespace.go
meson10
0
230
Cgroups and Namespaces in Linux
meson10
0
460
Other Decks in Technology
See All in Technology
専門家に使ってもらうAIプロダクトをつくる - 病院の医事課向けDPCコーディング支援システムの設計と評価
hagino3000
4
880
人の努力に頼らないセキュリティをエンジニアリングする在り方 / An approach to engineering security that does not rely on human effort
carta_engineering
0
220
Kansai WordPress Meetup@大阪『ブロックテーマを作る上で知っておくべきこと
torounit
0
250
Linuxネットワークスタック進化の軌跡: 過去10年の歩みと最新トレンド
orimanabu
4
760
こんなに変わった! AWSネットワークの常識
shotashiratori
16
11k
Frontier Teams — Amazon は AI で「開発のやり方」をどう変えたのか
kmotohas
0
300
エンタープライズIAMについて - ID・権限をどう統制するか
oidfj
1
210
Del vibe coding a un Desarrollo (agéntico) fiable
jcabot
0
180
[NW-JAWS DAY]「うちのAWS、インターネット出られないんで」― AWS外の自社サービスをPrivateLinkでシュッと公開する方法
ykimi
3
880
Why start a new company, on Rails and AI, today. SF Ruby Conf 2026, Irina Nazarova
irinanazarova
0
160
変更差分を読むために環境を整備しよう
pipepasta
1
470
契約書SaaSにAIエージェントを組み込む
sansantech
PRO
0
190
Featured
See All Featured
Design of three-dimensional binary manipulators for pick-and-place task avoiding obstacles (IECON2024)
konakalab
0
670
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
33
5.8k
Site-Speed That Sticks
csswizardry
13
1.6k
技術選定の審美眼(2025年版) / Understanding the Spiral of Technologies 2025 edition
twada
PRO
121
130k
Measuring Dark Social's Impact On Conversion and Attribution
stephenakadiri
2
330
Building the Perfect Custom Keyboard
takai
2
960
What the history of the web can teach us about the future of AI
inesmontani
PRO
3
820
The browser strikes back
jonoalderson
0
2k
jQuery: Nuts, Bolts and Bling
dougneiner
66
8.7k
Making the Leap to Tech Lead
cromwellryan
135
10k
The Impact of AI in SEO - AI Overviews June 2024 Edition
aleyda
6
1.3k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
123
22k