An Engineer's Guide to a Good Night's Sleep

@nickywrightson An Engineer’s Guide to a Good Night’s Sleep By
Nicky Wrightson @nickywrightson

@nickywrightson

@nickywrightson We are building REALLY complicated distributed systems

@nickywrightson Martin Fowler https://martinfowler.com/articles/microservice-trade-offs.html “You need a mature operations team
to manage lots of services, which are being redeployed regularly” https://martinfowler.com/articles/microservice-trade-offs.html

@nickywrightson

@nickywrightson Empowered teams means the team also control the support

@nickywrightson 2014 Consumers add a caching layer to protect against
our outages 2019 Out of hours calls to 3rd line have all but disappeared 2018 Migration to Kubernetes completed 2017 Our services were given an SLA of 15mins recovery time

@nickywrightson Approaches to reduce the risk of being called 5

@nickywrightson Engineer’s mindset 1

@nickywrightson 1

@nickywrightson Enable teams to own their own support models 1

@nickywrightson Operations Support Team A Support Team B 1

@nickywrightson The team triages issues during the day 1

@nickywrightson Engineers need to think about that out of hours
call with every error condition 1

@nickywrightson Design the severity levels within your service 1

@nickywrightson “The quality of a system will appear to be
declining unless it is rigorously maintained” Lehmans Laws of Software Evolution “Declining Quality” (1996) 1

@nickywrightson As system evolves, its complexity increases unless work is
done to maintain or reduce it Lehmans Laws of Software Evolution cont. "Increasing Complexity" (1974) 1

@nickywrightson Don’t get called for issues that could have been
caught in ofﬁce hours 2

@nickywrightson Releases during the day should never wake you up
at night 2

@nickywrightson Can our deployment times help this? 2

@nickywrightson Quick deployment 2

@nickywrightson 2 VERIFY VERIFY VERIFY

@nickywrightson By Cindy Sridharan (@copyconstruct) 2

@nickywrightson 3am batch jobs are a guarantee to get an
overnight call at some point 2

@nickywrightson 2

@nickywrightson Automate failure recovery where possible 3

@nickywrightson Let your platform recover for you 3

@nickywrightson Applications need to cope with change Graceful Termination Transactional
Clean restarts Stateless Queue Backed Idempotent 3

@nickywrightson Make your system idempotent so you can automatically replay
failed events 3

@nickywrightson Multi region automatic system failovers 3

@nickywrightson 3

@nickywrightson Our EU stack went 3

@nickywrightson 3

@nickywrightson Understand what your customers really care about 4

@nickywrightson You want to be the ﬁrst to know about
a critical failure 4

@nickywrightson “Only have alerts that you need to action” Sarah
Wells - Director of Operations and Reliability at FT 4

@nickywrightson Service that cleans old images from the repo Service
that takes payments Not all services are equal != 4

@nickywrightson Synthetic Requests 4

@nickywrightson Use tracing to monitor your critical ﬂows 4 Ben
Sigelman @ this morning’s keynote

@nickywrightson 4

@nickywrightson We are now ﬂagging important events close to the
code 4

@nickywrightson Understand what your customers really care about 4

@nickywrightson Break things and practice everything 5

@nickywrightson “a method of experimenting on infrastructure that lets you
expose weaknesses before they become a real problem.” 5

@nickywrightson Monolith to microservice timeline 5

@nickywrightson When can we release the chaos monkeys? 5

@nickywrightson Manual simulation of outages work too 5

@nickywrightson Spot the SPOF 5

@nickywrightson Multi region automatic system failovers 5

@nickywrightson Fixing things in hours helps team conﬁdence to support
out of hours 5

@nickywrightson Manual intervention should be simple FIX IT! 5

@nickywrightson 5

@nickywrightson Make sure your alerts have all the relevant information
to action the event 5

@nickywrightson Failed requests 5

@nickywrightson At 3am just get the system to limp into
hours 5

@nickywrightson Understand what your customers care about? 4

@nickywrightson The engineers are the ones called at 3am We
now own this!

@nickywrightson Thanks!

@nickywrightson Resources Testing Microservices, the sane way by Cindy Sridharan
https://medium.com/@copyconstruct/testing-microservices-the-sane- way-9bb31d158c16 Microservices trade offs by Martin Fowler https://martinfowler.com/articles/microservice-trade-offs.html https://medium.com/netﬂix-techblog/vizceral-open-source-acc0c32113fe

An Engineer's Guide to a Good Night's Sleep

An Engineer's Guide to a Good Night's Sleep

More Decks by Nicky Wrightson

Other Decks in Programming

Featured

Transcript