Microservices - What an Ops Team Needs to Know

An overview of the implications for an Ops team of the organisation moving towards microservices.
Given at Velocity Europe 2014

Transcript

1. Michael Brunton-Spall Technical Architect Government Digital Service @bruntonspall
2. I'm a civil servant GDS Michael Brunton-Spall
3. GDS Michael Brunton-Spall
4. This isn’t the crown’s opinion on microservices GDS Michael Brunton-Spall
5. Because everyone has an opinion GDS Michael Brunton-Spall
6. What am I not going to cover? GDS Michael Brunton-Spall
7. What are microservices GDS Michael Brunton-Spall
8. How to develop them GDS Michael Brunton-Spall
9. Microservice patterns GDS Michael Brunton-Spall
10. So what are microservices? GDS Michael Brunton-Spall
11. Vertically aligned stacks that communicate via simple and standard interfaces GDS Michael Brunton-Spall
12. Ownership of data GDS Michael Brunton-Spall
13. Team ownership of code GDS Michael Brunton-Spall
14. Team ownership of code and runtime GDS Michael Brunton-Spall
15. Small is beautiful GDS Michael Brunton-Spall
16. Why are your architect/ developers/CTO raving about them? GDS Michael Brunton-Spall
17. Because small owned systems can be updated more easily and frequently GDS Michael Brunton-Spall
18. Because teams can move fast and break stuff GDS Michael Brunton-Spall
19. Because teams can own the whole stack GDS Michael Brunton-Spall
20. So why do infrastructure teams hate them? GDS Michael Brunton-Spall
21. Because small owned systems can be updated more easily and frequently GDS Michael Brunton-Spall
22. Because teams can move fast and break stuff GDS Michael Brunton-Spall
23. Because teams can own the whole stack GDS Michael Brunton-Spall
24. GDS Michael Brunton-Spall
25. Theory of constraints GDS Michael Brunton-Spall
26. Concept -> Develop -> Deploy -> Learn GDS Michael Brunton-Spall
27. Development and deploy are just stages in the process GDS Michael Brunton-Spall
28. Focus on the constraints GDS Michael Brunton-Spall
29. Infrastructure is the constraint GDS Michael Brunton-Spall
30. Infrastructure is perceived as the constraint GDS Michael Brunton-Spall
31. So what can you do? GDS Michael Brunton-Spall
32. The role of operations is changing GDS Michael Brunton-Spall
33. The role of development is changing GDS Michael Brunton-Spall
34. Build tools to build and operate their stack GDS Michael Brunton-Spall
35. What tools do you need to supply? GDS Michael Brunton-Spall
36. Start small GDS Michael Brunton-Spall
37. Automate infrastructure or containers GDS Michael Brunton-Spall
38. Base Image GDS Michael Brunton-Spall
39. Base Image + Cloud Init GDS Michael Brunton-Spall
40. Base Image + Cloud Init + Deployed Application GDS Michael Brunton-Spall
41. Base Image = Ubuntu LTS GDS Michael Brunton-Spall
42. Cloud Init = "apt-get update; apt-get install xxx" GDS Michael Brunton-Spall
43. Deployed Application = "wget URL_TO_JAR; java - jar application.jar" GDS Michael Brunton-Spall
44. Time to new server = 1-2 minutes GDS Michael Brunton-Spall
45. Monitoring tools that are easy to hook into GDS Michael Brunton-Spall
46. "If it moves, graph it. If it doesn't move, graph it anyway just in case it does” @lozzd GDS Michael Brunton-Spall
47. Create a new metric and graph it GDS Michael Brunton-Spall
48. Automate log aggregation GDS Michael Brunton-Spall
49. These are solved problems GDS Michael Brunton-Spall
50. You are not *that* special, yet GDS Michael Brunton-Spall
51. Do 80% of the work with 20% of the effort GDS Michael Brunton-Spall
52. Cloud-friendly databases GDS Michael Brunton-Spall
53. Automate deployments ASAP GDS Michael Brunton-Spall
54. GDS Michael Brunton-Spall
55. Hand the keys for deployment to developers GDS Michael Brunton-Spall
56. Keep logs of deploys GDS Michael Brunton-Spall
57. (Graph them even!) GDS Michael Brunton-Spall
58. GDS Michael Brunton-Spall
59. Alerting GDS Michael Brunton-Spall
60. Give developers pagers too GDS Michael Brunton-Spall
61. "Developers should be exposed to the pain they cause" GDS Michael Brunton-Spall
62. Developers having root on their service? GDS Michael Brunton-Spall
63. "You break it, you fix it" GDS Michael Brunton-Spall
64. "You break it and cost £X million, you take the blame" GDS Michael Brunton-Spall
65. Automate Infrastructure Monitoring tools Log aggregation Databases Automated deployments Alerting GDS Michael Brunton-Spall
66. How about post-live? GDS Michael Brunton-Spall
67. Microservices are going to fail in more spectacular ways than monoliths GDS Michael Brunton-Spall
68. Simple Systems -> Complicated Systems -> Complex Systems GDS Michael Brunton-Spall
69. Diagnosis tools GDS Michael Brunton-Spall
70. Operations teams may have to do diagnosis across the services GDS Michael Brunton-Spall
71. Tracing call stacks GDS Michael Brunton-Spall
72. Spanner algorithm GDS Michael Brunton-Spall
73. Zipkin GDS Michael Brunton-Spall
74. GDS Michael Brunton-Spall
75. How are services monitored? GDS Michael Brunton-Spall
76. Deep checks vs shallow checks GDS Michael Brunton-Spall
77. Average response time isn't enough GDS Michael Brunton-Spall
78. 90%, 95%, 99% as well GDS Michael Brunton-Spall
79. Distributed systems need to embrace failure GDS Michael Brunton-Spall
80. Partitions are network breaks and latency spikes GDS Michael Brunton-Spall
81. Timeouts are tricky GDS Michael Brunton-Spall
82. 250ms timeout on network calls GDS Michael Brunton-Spall
83. We make 100 network calls to complete a user request GDS Michael Brunton-Spall
84. 25s timeout if everything fails? GDS Michael Brunton-Spall
85. Timeouts are tricky, use with care GDS Michael Brunton-Spall
86. Alerting tools GDS Michael Brunton-Spall
87. If the user can't purchase a item because inventory control failed, who gets alerted? GDS Michael Brunton-Spall
88. You still need 1st line support and triage GDS Michael Brunton-Spall
89. What about when the developers get bored? (They do that!) GDS Michael Brunton-Spall
90. Maturity model for support? GDS Michael Brunton-Spall
91. System must have < N faults in M weeks and complete run book to be handed off GDS Michael Brunton-Spall
92. Systems that have more than N faults in M weeks get handed back GDS Michael Brunton-Spall
93. Who pays for this support? GDS Michael Brunton-Spall
94. Are services technical or value adding? GDS Michael Brunton-Spall
95. If value adding, what value do they add? GDS Michael Brunton-Spall
96. For everything else, act as consultants GDS Michael Brunton-Spall
97. "You want an elastic search cluster, sure we'll help you …” GDS Michael Brunton-Spall
98. "... if we can document and automate it (+20% effort)" GDS Michael Brunton-Spall
99. Pioneers pay the cost GDS Michael Brunton-Spall
100. Settlers pave the roads behind GDS Michael Brunton-Spall
101. Everyone else follows GDS Michael Brunton-Spall
102. "Do it once, twice then automate" GDS Michael Brunton-Spall
103. Don't forget to keep the lights on at the monolith GDS Michael Brunton-Spall
104. Michael Brunton-Spall Technical Architect Government Digital Service @bruntonspall