Link
Embed
Share
Beginning
This slide
Copy link URL
Copy link URL
Copy iframe embed code
Copy iframe embed code
Copy javascript embed code
Copy javascript embed code
Share
Tweet
Share
Tweet
Slide 1
Slide 1 text
Resilient Software Design Building software that doesn’t give up!
Slide 2
Slide 2 text
1. What’s all this about? 2. Anti-patterns: What not to do! 3. Patterns: Make your life easier.
Slide 3
Slide 3 text
Find the odd one 1. :cloud_factory 2. "CloudFactory" 3. CloudFactory 4. CLOUD_FACTORY
Slide 4
Slide 4 text
What will this code print? if fork puts "I won the lottery!" else puts "I am bankrupt!" end
Slide 5
Slide 5 text
What’s all this about?
Slide 6
Slide 6 text
Stability
Slide 7
Slide 7 text
It Just Works™
Slide 8
Slide 8 text
It Should Work™
Slide 9
Slide 9 text
Bad things happen
Slide 10
Slide 10 text
Good things happen as well
Slide 11
Slide 11 text
But those are rare!
Slide 12
Slide 12 text
Anything that can go wrong, will go wrong. - Murphy’s Law
Slide 13
Slide 13 text
Developers think positive
Slide 14
Slide 14 text
Too much so!
Slide 15
Slide 15 text
Be negative.
Slide 16
Slide 16 text
Our goal is to build software
Slide 17
Slide 17 text
Our goal is also to minimise pain
Slide 18
Slide 18 text
Our goal is also to save money
Slide 19
Slide 19 text
Resilient software saves money by not breaking when needed
Slide 20
Slide 20 text
Resilient software saves money by using optimum infrastructure
Slide 21
Slide 21 text
Resilient software saves money by keeping developers happy
Slide 22
Slide 22 text
Anti-patterns
Slide 23
Slide 23 text
1. Integration points
Slide 24
Slide 24 text
Integration is not what you think™
Slide 25
Slide 25 text
Database is an integration
Slide 26
Slide 26 text
Third party services are integration.
Slide 27
Slide 27 text
Your cache layer is an integration
Slide 28
Slide 28 text
No content
Slide 29
Slide 29 text
Networks fail more often
Slide 30
Slide 30 text
Socket based protocols have a special way of failing
Slide 31
Slide 31 text
Refused connection is bad.
Slide 32
Slide 32 text
Hanged connection is worse.
Slide 33
Slide 33 text
Micro-services that talk to each other, will stop talking abruptly
Slide 34
Slide 34 text
2. Unbalanced Capacities
Slide 35
Slide 35 text
Specially applicable to micro-services
Slide 36
Slide 36 text
3. Slow responses
Slide 37
Slide 37 text
4. Unbounded Result Sets
Slide 38
Slide 38 text
Major anti-pattern, overlooked by many
Slide 39
Slide 39 text
What is the size of an HTTP cookie?
Slide 40
Slide 40 text
Patterns
Slide 41
Slide 41 text
1. Use timeouts!
Slide 42
Slide 42 text
What is the default timeout on Ruby’s net/http?
Slide 43
Slide 43 text
Now and forever, networks will always be unreliable. - Michael T Nygard
Slide 44
Slide 44 text
Every network call in your system must have a timeout
Slide 45
Slide 45 text
This includes database calls
Slide 46
Slide 46 text
This includes API calls
Slide 47
Slide 47 text
This includes cache lookups
Slide 48
Slide 48 text
What to do when the timeout occurs depends on where it occurred
Slide 49
Slide 49 text
Highly context specific, so the dev team should make that decision
Slide 50
Slide 50 text
ProTip: Do not use Ruby’s “timeout” module
Slide 51
Slide 51 text
Instead, depend on libraries for the timeout
Slide 52
Slide 52 text
If you’re a library author, just use net/http’s timeout
Slide 53
Slide 53 text
2. Circuit Breaker
Slide 54
Slide 54 text
No content
Slide 55
Slide 55 text
3. Bulkheads
Slide 56
Slide 56 text
A ship is divided into several water- tight compartments
Slide 57
Slide 57 text
In case there is a leakage in one section, water doesn’t flood into other sections
Slide 58
Slide 58 text
Same principle!
Slide 59
Slide 59 text
Use resource pools
Slide 60
Slide 60 text
Use rate limiting
Slide 61
Slide 61 text
Consideration: Capacity
Slide 62
Slide 62 text
Bulkheading often conflicts increasing or variable capacity
Slide 63
Slide 63 text
Consideration: Performance
Slide 64
Slide 64 text
Bulkheading often results in slightly reduced performance
Slide 65
Slide 65 text
It’s worth it, trust me™
Slide 66
Slide 66 text
4. Fail fast
Slide 67
Slide 67 text
Remember guard clauses in Ruby?
Slide 68
Slide 68 text
# class Event def closest_event return unless self.location # … # … end
Slide 69
Slide 69 text
Same principle!
Slide 70
Slide 70 text
Any Ruby libraries?
Slide 71
Slide 71 text
Not a lot :(
Slide 72
Slide 72 text
shopify/semian
Slide 73
Slide 73 text
Thank you!
Slide 74
Slide 74 text
Questions?
Slide 75
Slide 75 text
Swanand Pagnis Principal Engineer @ First
Slide 76
Slide 76 text
Swanand Pagnis @_swanand on Twitter
Slide 77
Slide 77 text
Swanand Pagnis @swanandp on GitHub