Operating Microservices

Microservices is the new cool architecture, and developers and architects are all jumping on the bandwagon, often without thinking about how they are going to operate the microservice.

Microservices architecture changes your software from a monolithic complicated system to a complex distributed system, and there are ways the operations team will have to change how they operate the system, and the way they interact with the development teams.

We’ll cover how DevOps-like patterns can be applied on microservices to give the development teams more responsibility for their choices, and how monitoring, logging, auditing, security and other concerns can be managed in a distributed system.

Transcript

1. Michael Brunton-Spall Technical Architect Government Digital Service @bruntonspall
2. I'm a civil servant GDS Michael Brunton-Spall
3. GDS Michael Brunton-Spall
4. This isn’t the crown’s opinion on micro-services GDS Michael Brunton-Spall
5. Because everyone has an opinion GDS Michael Brunton-Spall
6. What am I not going to cover? GDS Michael Brunton-Spall
7. What are micro-services GDS Michael Brunton-Spall
8. How to develop them GDS Michael Brunton-Spall
9. Micro-service patterns GDS Michael Brunton-Spall
10. So what are micro-services? GDS Michael Brunton-Spall
11. Vertically aligned stacks that communicate via simple and standard interfaces GDS Michael Brunton-Spall
12. Ownership of data GDS Michael Brunton-Spall
13. Team ownership of code GDS Michael Brunton-Spall
14. Team ownership of code and runtime GDS Michael Brunton-Spall
15. Small is beautiful GDS Michael Brunton-Spall
16. Why are we all raving about them? GDS Michael Brunton-Spall
17. Because small owned systems can be updated more easily and frequently GDS Michael Brunton-Spall
18. Because teams can move fast and break stuff GDS Michael Brunton-Spall
19. Because teams can own the whole stack GDS Michael Brunton-Spall
20. So why do infrastructure teams hate them? GDS Michael Brunton-Spall
21. Because small owned systems can be updated more easily and frequently GDS Michael Brunton-Spall
22. Because teams can move fast and break stuff GDS Michael Brunton-Spall
23. Because teams can own the whole stack GDS Michael Brunton-Spall
24. “Microservices represent a new organisational model as much as a new architectural model” @JeffSussna GDS Michael Brunton-Spall
25. You (Engineering and Operations) exist purely for the benefit of the business GDS Michael Brunton-Spall
26. Micro-services can help you have a radical focus on delivering business value GDS Michael Brunton-Spall
27. The role of engineering and operations is changing GDS Michael Brunton-Spall
28. Operations is the constraint GDS Michael Brunton-Spall
29. Operations is perceived as the constraint GDS Michael Brunton-Spall
30. Micro-services in the small GDS Michael Brunton-Spall
31. How can we get started? GDS Michael Brunton-Spall
32. Start small GDS Michael Brunton-Spall
33. Automate infrastructure or containers GDS Michael Brunton-Spall
34. Base Image GDS Michael Brunton-Spall
35. Base Image = Ubuntu LTS GDS Michael Brunton-Spall
36. Base Image + Cloud Init GDS Michael Brunton-Spall
37. Cloud Init = "apt-get update; apt-get install xxx" GDS Michael Brunton-Spall
38. Base Image + Cloud Init + Deployed Application GDS Michael Brunton-Spall
39. Deployed Application = "wget URL_TO_JAR; java - jar application.jar" GDS Michael Brunton-Spall
40. Time to new server = 1-2 minutes GDS Michael Brunton-Spall
41. Why is this useful? GDS Michael Brunton-Spall
42. Developer stupidity should have a business impact GDS Michael Brunton-Spall
43. Monitoring tools that are easy to hook into GDS Michael Brunton-Spall
44. Developers can create a new metric and graph it GDS Michael Brunton-Spall
45. GDS Michael Brunton-Spall
46. "If it moves, graph it. If it doesn't move, graph it anyway just in case it does” @lozzd GDS Michael Brunton-Spall
47. Automate log aggregation GDS Michael Brunton-Spall
48. ElasticSearch, LogStash, Kibana GDS Michael Brunton-Spall
49. GDS Michael Brunton-Spall
50. Cloud-friendly databases GDS Michael Brunton-Spall
51. Automate deployments GDS Michael Brunton-Spall
52. GDS Michael Brunton-Spall
53. GDS Michael Brunton-Spall
54. GDS Michael Brunton-Spall
55. Hand the keys for deployment to developers GDS Michael Brunton-Spall
56. Keep logs of deploys GDS Michael Brunton-Spall
57. GDS Michael Brunton-Spall
58. (Graph them even!) GDS Michael Brunton-Spall
59. GDS Michael Brunton-Spall
60. Alerting GDS Michael Brunton-Spall
61. (Sensu Screenshot) GDS Michael Brunton-Spall
62. Give developers pagers too GDS Michael Brunton-Spall
63. Developers should be exposed to the pain they cause GDS Michael Brunton-Spall
64. Developers having root on their service? GDS Michael Brunton-Spall
65. "You break it, you fix it" GDS Michael Brunton-Spall
66. "You break it and cost £X million, you take the blame" GDS Michael Brunton-Spall
67. Automate Infrastructure Monitoring tools Log aggregation Databases Automated deployments Alerting GDS Michael Brunton-Spall
68. These are all solved problems GDS Michael Brunton-Spall
69. You are not *that* special GDS Michael Brunton-Spall
70. Most of what you do is wrapping libraries and other peoples code GDS Michael Brunton-Spall
71. Pareto principle GDS Michael Brunton-Spall
72. You now have a single, well supported micro service GDS Michael Brunton-Spall
73. Microservices in the large GDS Michael Brunton-Spall
74. Microservices are going to fail in more spectacular ways than equivalent monoliths GDS Michael Brunton-Spall
75. Simple GDS Michael Brunton-Spall
76. Simple Complicated GDS Michael Brunton-Spall
77. Simple Complicated Complex GDS Michael Brunton-Spall
78. Diagnosis tools GDS Michael Brunton-Spall
79. Someone will have to do diagnosis across all the services GDS Michael Brunton-Spall
80. How are services monitored? GDS Michael Brunton-Spall
81. Shallow Check - Is my service working GDS Michael Brunton-Spall
82. Deep check - Are my dependencies working? GDS Michael Brunton-Spall
83. Average response time isn't enough GDS Michael Brunton-Spall
84. 90%, 95%, 99% as well GDS Michael Brunton-Spall
85. Distributed systems need to embrace failure GDS Michael Brunton-Spall
86. Partitions are network breaks and latency spikes GDS Michael Brunton-Spall
87. Timeouts are tricky, use with care GDS Michael Brunton-Spall
88. Standardise admin interfaces GDS Michael Brunton-Spall
89. Alerting tools GDS Michael Brunton-Spall
90. If citizens can’t submit a claim at 1am because the mainframe is down, who’s problem is that? GDS Michael Brunton-Spall
91. You still need 1st line support and triage GDS Michael Brunton-Spall
92. What about when the developers get bored? (They do that!) GDS Michael Brunton-Spall
93. Maturity model for support? GDS Michael Brunton-Spall
94. System must have < N faults in M weeks and complete run book to be handed off GDS Michael Brunton-Spall
95. Systems that have more than N faults in M weeks get handed back GDS Michael Brunton-Spall
96. Who pays for this support? GDS Michael Brunton-Spall
97. Operations teams should act like consultants GDS Michael Brunton-Spall
98. "You want an elastic search cluster, sure we'll help you …” GDS Michael Brunton-Spall
99. "... if we can document and automate it (+20% effort)" GDS Michael Brunton-Spall
100. Pioneers pay the cost GDS Michael Brunton-Spall
101. Settlers pave the roads behind GDS Michael Brunton-Spall
102. Everyone else follows GDS Michael Brunton-Spall
103. Don't forget to keep the lights on at the monolith GDS Michael Brunton-Spall
104. Michael Brunton-Spall Technical Architect Government Digital Service @bruntonspall