Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Bulletproof communication in distributed systems
Search
Jakub Kubryński
March 30, 2019
Programming
0
520
Bulletproof communication in distributed systems
Jakub Kubryński
March 30, 2019
Tweet
Share
More Decks by Jakub Kubryński
See All by Jakub Kubryński
Autonomous deployments in modern distributed systems
jkubrynski
1
120
JVM Workshop
jkubrynski
1
190
Engineering architecture
jkubrynski
0
810
Case study of the good enough architecture
jkubrynski
0
430
JPA - Beyond Copy-Paste
jkubrynski
3
1.6k
Microservices - The Naked Truth Of The Maintainability
jkubrynski
1
720
Other Decks in Programming
See All in Programming
アプリの "かわいい" を支えるアニメーションツールRiveについて
uetyo
0
280
プロパティベーステストによるUIテスト: LLMによるプロパティ定義生成でエッジケースを捉える
tetta_pdnt
0
5k
AI Agents: How Do They Work and How to Build Them @ Shift 2025
slobodan
0
110
もうちょっといいRubyプロファイラを作りたい (2025)
osyoyu
1
460
Navigation 2 を 3 に移行する(予定)ためにやったこと
yokomii
0
360
OSS開発者という働き方
andpad
5
1.7k
2025 年のコーディングエージェントの現在地とエンジニアの仕事の変化について
azukiazusa1
24
12k
250830 IaCの選定~AWS SAMのLambdaをECSに乗り換えたときの備忘録~
east_takumi
0
400
機能追加とリーダー業務の類似性
rinchoku
2
1.4k
スケールする組織の実現に向けた インナーソース育成術 - ISGT2025
teamlab
PRO
2
180
Ruby×iOSアプリ開発 ~共に歩んだエコシステムの物語~
temoki
0
360
請來的 AI Agent 同事們在寫程式時,怎麼用 pytest 去除各種幻想與盲點
keitheis
0
130
Featured
See All Featured
The Illustrated Children's Guide to Kubernetes
chrisshort
48
50k
Context Engineering - Making Every Token Count
addyosmani
3
64
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
8
930
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
tammyeverts
8
530
Fireside Chat
paigeccino
39
3.6k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
36
2.5k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
48
9.7k
For a Future-Friendly Web
brad_frost
180
9.9k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
358
30k
How STYLIGHT went responsive
nonsquared
100
5.8k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
667
120k
How To Stay Up To Date on Web Technology
chriscoyier
790
250k
Transcript
@jkubrynski / kubrynski.com @jkubrynski / kubrynski.com BULLETPROOF BULLETPROOF COMMUNICATION COMMUNICATION
JAKUB KUBRYNSKI JAKUB KUBRYNSKI
[email protected]
/ @jkubrynski / kubrynski.blog
$ WHOAMI $ WHOAMI DEVSKILLER CO-FOUNDER DEVSKILLER CO-FOUNDER BOTTEGA TRAINER
BOTTEGA TRAINER DEVOXX.PL PROGRAM COMMITTEE MEMBER DEVOXX.PL PROGRAM COMMITTEE MEMBER SPRING CLOUD CONTRACT CO-AUTHOR SPRING CLOUD CONTRACT CO-AUTHOR
DEVSKILLER IS LOOKING FOR DEVSKILLER IS LOOKING FOR HELLO-WORLD DEVELOPERS
HELLO-WORLD DEVELOPERS
WHY DISTRIBUTED SYSTEMS? WHY DISTRIBUTED SYSTEMS? SCALABILITY SCALABILITY POLYGLOT POLYGLOT
RESILIENCE RESILIENCE
DISTRIBUTED SYSTEMS FALLACIES DISTRIBUTED SYSTEMS FALLACIES THE NETWORK IS RELIABLE
THE NETWORK IS RELIABLE LATENCY IS ZERO LATENCY IS ZERO THE NETWORK IS SECURE THE NETWORK IS SECURE TOPOLOGY DOESN'T CHANGE TOPOLOGY DOESN'T CHANGE
IF YOU WON'T RESPECT FALLACIES IF YOU WON'T RESPECT FALLACIES
YOU'LL BUILD DISTRIBUTED MONOLITH YOU'LL BUILD DISTRIBUTED MONOLITH
BRACE YOURSELF BRACE YOURSELF FAILURE IS COMING FAILURE IS COMING
IN MONOLITH ALL COLLABORATORS ARE IN MONOLITH ALL COLLABORATORS ARE
ALWAYS AVAILABLE ALWAYS AVAILABLE
IN DISTRIBUTED SYSTEMS THEY ARE NOT IN DISTRIBUTED SYSTEMS THEY
ARE NOT NETWORK ISSUES NETWORK ISSUES REDEPLOYS REDEPLOYS OUTAGES OUTAGES
AVAILABILITY VS DOWNTIME AVAILABILITY VS DOWNTIME 9 9 . 9
9 9 % 35d 4d 8h 50m 5m per year 2.5h 14m 1.5m 8s 0.8s per day
99.9 (1.5 MIN/DAY) 99.9 (1.5 MIN/DAY) 5 * 99.9 =>
99.5 (7 MIN/DAY) 5 * 99.9 => 99.5 (7 MIN/DAY)
TEMPORAL COUPLING TEMPORAL COUPLING
DESIGN FOR FAILURE DESIGN FOR FAILURE
PREFER ASYNCHRONOUS PREFER ASYNCHRONOUS OVER SYNCHRONOUS OVER SYNCHRONOUS
ASYNCHRONOUS ACTORS ASYNCHRONOUS ACTORS COMMANDS (QUEUES) COMMANDS (QUEUES) EVENTS (TOPICS)
EVENTS (TOPICS)
ASYNCHRONOUS PRINCIPLES ASYNCHRONOUS PRINCIPLES PERSISTENT PERSISTENT AT-LEAST-ONCE DELIVERY AT-LEAST-ONCE DELIVERY
EXPONENTIAL RETRIES EXPONENTIAL RETRIES
SYNCHRONOUS SYNCHRONOUS IS USED ONLY WHEN REQUIRED IS USED ONLY
WHEN REQUIRED
WHAT COULD POSSIBLY GO WRONG? WHAT COULD POSSIBLY GO WRONG?
¯\_(ツ)_/¯ ¯\_(ツ)_/¯
TIMEOUT?!?! TIMEOUT?!?! RETRY RETRY
CIRCUIT BREAKER CIRCUIT BREAKER KEEP CALM AND FALLBACK KEEP CALM
AND FALLBACK
IN MONOLITH TOPOLOGY IS FIXED IN MONOLITH TOPOLOGY IS FIXED
IN DISTRIBUTED SYSTEMS IT'S NOT IN DISTRIBUTED SYSTEMS IT'S NOT
CLOUD NATIVE / CONTAINERS CLOUD NATIVE / CONTAINERS DYNAMIC SCALABILITY DYNAMIC SCALABILITY
ALWAYS USE SERVICE DISCOVERY ALWAYS USE SERVICE DISCOVERY
LOAD BALANCERS SUCKS LOAD BALANCERS SUCKS IT'S SINGLE POINT OF
FAILURE IT'S SINGLE POINT OF FAILURE
ALWAYS USE CLIENT SIDE LOAD BALACING ALWAYS USE CLIENT SIDE
LOAD BALACING
IT'S ALL ABOUT COMMUNICATION IT'S ALL ABOUT COMMUNICATION SO NICE
TO KNOW IT WORKS SO NICE TO KNOW IT WORKS
DISTRIBUTED SYSTEMS DISTRIBUTED SYSTEMS VS VS MICROSERVICES MICROSERVICES
MICROSERVICES BRING MICROSERVICES BRING AUTONOMY AUTONOMY
AUTONOMY IS HARD AUTONOMY IS HARD BECAUSE WE NEED TO
COORDINATE DEPENDENCIES BECAUSE WE NEED TO COORDINATE DEPENDENCIES AND BE BACKWARD COMPATIBLE AND BE BACKWARD COMPATIBLE
LOOSE COUPLING LOOSE COUPLING OVER OVER CANONICAL MODEL CANONICAL MODEL
END-TO-END TESTS ARE HARD END-TO-END TESTS ARE HARD FALSE-NEGATIVE FALSE-NEGATIVE
FALSE-POSITIVE FALSE-POSITIVE
VERIFY CONTRACTS VERIFY CONTRACTS A == C && B ==
C A == C && B == C A == B A == B
CONTRACT CONVERTS TO CONTRACT CONVERTS TO SERVER TEST SERVER TEST
CLIENT STUB CLIENT STUB
PERFORMANCE? PERFORMANCE?
MONITORING IS THE NEW TESTING MONITORING IS THE NEW TESTING
MONITORING MONITORING EXCEPTIONS EXCEPTIONS BUSINESS METRICS BUSINESS METRICS APPLICATION PERFORMANCE
MANAGEMENT APPLICATION PERFORMANCE MANAGEMENT
ANSCOMBE'S QUARTET ANSCOMBE'S QUARTET Mean of x => 9 Sample
variance of x => 11 Mean of y => 7.50 Sample variance of y => 4.125 Correlation => 0.816 Linear regression => y = 3.00 + 0.500x
PERCENTILES PERCENTILES P50, P90, P95, P99 P50, P90, P95, P99
MONITOR MONITOR P99.9 OR P99.95 P99.9 OR P99.95
THANKS THANKS
DROGA.DEV DROGA.DEV
QUESTIONS? QUESTIONS?
"Have no fear of perfection - you'll never reach it."
― Salvador Dalí