An Engineer's Guide to Good Nights Sleep

This is the latest version I about this topic as it changes every time I do it as this topic is ever evolving .... And I would love feedback :)

In a time of empowered teams, complex tech stacks, highly distributed systems the support model of years ago needed to adapt. The developers now own how they run and operate their systems including out of hours support. In this talk I will share several approaches and considerations that can help reduce the risk of that dreaded 3am call from Ops.

Transcript

1. @nickywrightson An Engineer’s Guide to a Good Night’s Sleep By Nicky Wrightson @nickywrightson
2. @nickywrightson
3. @nickywrightson
4. @nickywrightson
5. @nickywrightson We are building REALLY complicated distributed systems
6. @nickywrightson Martin Fowler https://martinfowler.com/articles/microservice-trade-offs.html “You need a mature operations team to manage lots of services, which are being redeployed regularly”
7. @nickywrightson Empowered teams means the team also control the support
8. @nickywrightson 2014 Consumers add a caching layer to protect against our outages 2019 Out of hours calls to 3rd line have all but disappeared 2018 Migration to Kubernetes completed 2017 Our services were given an SLA of 15mins recovery time
9. @nickywrightson Approaches to reduce the risk of being called 5
10. @nickywrightson Engineer’s mindset 1
11. @nickywrightson 1
12. @nickywrightson Enable teams to own their own support models 1
13. @nickywrightson Operations Support Team A Support Team B 1
14. @nickywrightson The team triages issues during the day 1
15. @nickywrightson Engineers need to think about that out of hours call with every error condition 1
16. @nickywrightson Design the severity levels within your service 1
17. @nickywrightson Engineer’s mindset 1
18. @nickywrightson Don’t get called for issues that could have been caught in office hours 2
19. @nickywrightson Releases during the day should never wake you up at night 2
20. @nickywrightson Can our deployment times help this? 2
21. @nickywrightson 2 Get your deployment system do automatic rollbacks
22. @nickywrightson 2 VERIFY VERIFY VERIFY
23. @nickywrightson By Cindy Sridharan (@copyconstruct) 2
24. @nickywrightson 3am batch jobs are a guarantee to get an overnight call at some point 2
25. @nickywrightson Don’t get called for issues that could have been caught in office hours 2
26. @nickywrightson Automate failure recovery where possible 3
27. @nickywrightson Let your platform recover for you 3
28. @nickywrightson Applications need to cope with change Graceful Termination Transactional Clean restarts Stateless Queue Backed Idempotent 3
29. @nickywrightson Multi region automatic system failovers 3
30. @nickywrightson 3
31. @nickywrightson Multi region automatic system failovers 3
32. @nickywrightson Healthchecks and liveness probes may not tell the whole story
33. @nickywrightson
34. @nickywrightson Automate failure recovery where possible 3
35. @nickywrightson Understand what your customers really care about 4
36. @nickywrightson You want to be the first to know about a critical failure 4
37. @nickywrightson “Only have alerts that you need to action” Sarah Wells - Director of Operations and Reliability at FT 4
38. @nickywrightson Synthetic Requests 4
39. @nickywrightson Use tracing to monitor your critical flows 4 Ben Sigelman Restoring Confidence in Microservices: Tracing That's More Than Traces
40. @nickywrightson 4
41. @nickywrightson 4
42. @nickywrightson 4
43. @nickywrightson 4
44. @nickywrightson Understand what your customers really care about 4
45. @nickywrightson Break things and practice everything 5
46. @nickywrightson “a method of experimenting on infrastructure that lets you expose weaknesses before they become a real problem.” 5
47. @nickywrightson Monolith to microservice timeline 5
48. @nickywrightson When can we release the chaos monkeys? 5
49. @nickywrightson Manual simulation of outages work too 5
50. @nickywrightson Spot the SPOF 5
51. @nickywrightson Multi region automatic system failovers 5
52. @nickywrightson Multi region automatic system failovers 5
53. @nickywrightson Manual intervention should be simple FIX IT! 5
54. @nickywrightson 5
55. @nickywrightson Provide all the information to action an alert 5
56. @nickywrightson At 3am just get the system to limp into office hours 5
57. @nickywrightson Break things and practice everything 5
58. @nickywrightson Engineer’s mindset 1
59. @nickywrightson Don’t get called for issues that could have been caught in office hours 2
60. @nickywrightson Automate failure recovery where possible 3
61. @nickywrightson Understand what your customers care about? 4
62. @nickywrightson Break things and practice everything 5
63. @nickywrightson The engineers are the ones called at 3am We now own this!
64. @nickywrightson Thanks! https://speakerdeck.com/nickywrightson We are hiring!
65. @nickywrightson Resources Testing Microservices, the sane way by Cindy Sridharan https://medium.com/@copyconstruct/testing-microservices-the-sane-way-9bb31d158c16 Microservices trade offs by Martin Fowler https://martinfowler.com/articles/microservice-trade-offs.html Ben Sigelman @ QCon 2019 https://www.infoq.com/presentations/microservices-distributed-tracing? itm_source=infoq&itm_medium=QCon_EarlyAccessVideos&itm_campaign=QConLondon2019 James Governor on progressive delivery: https://redmonk.com/jgovernor/2018/08/06/towards- progressive-delivery/ Charity Majors on Friday freezes: https://charity.wtf/2019/05/01/friday-deploy-freezes-are-exactly-like- murdering-puppies/ Skyscanner hiring link: https://grnh.se/579803f21