Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Resilient Software Design
Search
Swanand Pagnis
January 21, 2015
Technology
1
380
Resilient Software Design
Lessons learned from Michal Nygard's "Release It!"
Swanand Pagnis
January 21, 2015
Tweet
Share
More Decks by Swanand Pagnis
See All by Swanand Pagnis
Principles of Project Management Flow Part 1 and 2
swanandp
0
190
The Road to Continuous Deployments
swanandp
0
600
Fearless Joins
swanandp
0
310
The Deep Work Divide: Restoring the balance between collaboration and deep work
swanandp
1
890
The Well Rounded Engineer
swanandp
23
14k
The Deep Work Mentality
swanandp
6
2k
Combinator Parsing
swanandp
0
760
It's 2017, and I still want to sell you a graph database
swanandp
2
860
Architecture Over Framework: Rethink Your App Structure
swanandp
1
230
Other Decks in Technology
See All in Technology
(機械学習システムでも) SLO から始める信頼性構築 - ゆる SRE#9 2025/02/21
daigo0927
0
260
Amazon Aurora のバージョンアップ手法について
smt7174
2
140
Amazon Q Developerの無料利用枠を使い倒してHello worldを表示させよう!
nrinetcom
PRO
2
110
MIMEと文字コードの闇
hirachan
2
1.4k
【内製開発Summit 2025】イオンスマートテクノロジーの内製化組織の作り方/In-house-development-summit-AST
aeonpeople
2
580
プロダクトエンジニア構想を立ち上げ、プロダクト志向な組織への成長を続けている話 / grow into a product-oriented organization
hiro_torii
1
350
PHPカンファレンス名古屋-テックリードの経験から学んだ設計の教訓
hayatokudou
2
540
AIエージェント元年
shukob
0
150
LINE NEWSにおけるバックエンド開発
lycorptech_jp
PRO
0
200
エンジニアが加速させるプロダクトディスカバリー 〜最速で価値ある機能を見つける方法〜 / product discovery accelerated by engineers
rince
4
560
株式会社Awarefy(アウェアファイ)会社説明資料 / Awarefy-Company-Deck
awarefy
3
11k
OPENLOGI Company Profile
hr01
0
60k
Featured
See All Featured
Code Review Best Practice
trishagee
67
18k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
507
140k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
32
2.1k
A Tale of Four Properties
chriscoyier
158
23k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
226
22k
Imperfection Machines: The Place of Print at Facebook
scottboms
267
13k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
160
15k
Scaling GitHub
holman
459
140k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
49
2.3k
The MySQL Ecosystem @ GitHub 2015
samlambert
250
12k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
33
2.8k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
46
2.3k
Transcript
Resilient Software Design Building software that doesn’t give up!
1. What’s all this about? 2. Anti-patterns: What not to
do! 3. Patterns: Make your life easier.
Find the odd one 1. :cloud_factory 2. "CloudFactory" 3. CloudFactory
4. CLOUD_FACTORY
What will this code print? if fork puts "I won
the lottery!" else puts "I am bankrupt!" end
What’s all this about?
Stability
It Just Works™
It Should Work™
Bad things happen
Good things happen as well
But those are rare!
Anything that can go wrong, will go wrong. - Murphy’s
Law
Developers think positive
Too much so!
Be negative.
Our goal is to build software
Our goal is also to minimise pain
Our goal is also to save money
Resilient software saves money by not breaking when needed
Resilient software saves money by using optimum infrastructure
Resilient software saves money by keeping developers happy
Anti-patterns
1. Integration points
Integration is not what you think™
Database is an integration
Third party services are integration.
Your cache layer is an integration
None
Networks fail more often
Socket based protocols have a special way of failing
Refused connection is bad.
Hanged connection is worse.
Micro-services that talk to each other, will stop talking abruptly
2. Unbalanced Capacities
Specially applicable to micro-services
3. Slow responses
4. Unbounded Result Sets
Major anti-pattern, overlooked by many
What is the size of an HTTP cookie?
Patterns
1. Use timeouts!
What is the default timeout on Ruby’s net/http?
Now and forever, networks will always be unreliable. - Michael
T Nygard
Every network call in your system must have a timeout
This includes database calls
This includes API calls
This includes cache lookups
What to do when the timeout occurs depends on where
it occurred
Highly context specific, so the dev team should make that
decision
ProTip: Do not use Ruby’s “timeout” module
Instead, depend on libraries for the timeout
If you’re a library author, just use net/http’s timeout
2. Circuit Breaker
None
3. Bulkheads
A ship is divided into several water- tight compartments
In case there is a leakage in one section, water
doesn’t flood into other sections
Same principle!
Use resource pools
Use rate limiting
Consideration: Capacity
Bulkheading often conflicts increasing or variable capacity
Consideration: Performance
Bulkheading often results in slightly reduced performance
It’s worth it, trust me™
4. Fail fast
Remember guard clauses in Ruby?
# class Event def closest_event return unless self.location # …
# … end
Same principle!
Any Ruby libraries?
Not a lot :(
shopify/semian
Thank you!
Questions?
Swanand Pagnis Principal Engineer @ First
Swanand Pagnis @_swanand on Twitter
Swanand Pagnis @swanandp on GitHub