Cloud Chaos and Microservices Mayhem - presented at OOP Digital

The cloud is just someone else's data center, but it has fundamentally changed how we design software and what we expect from our platforms. Our applications have gotten bigger, more distributed, and more complicated, and there are whole new categories of mistakes we can make. Some things that were a good idea ten years ago turn out to be a terrible idea in the cloud; and what used to be ‘good enough’ for testing really isn’t anymore. Managing microservices architecture demands a lot of us, to ensure observability, operational resiliency, and organisational agility. With a focus on Java, this talk will introduce some of the new tools, patterns, and best practices for modern distributed application development. It also gives a tour of some of the most painful anti-patterns Holly has seen as a cloud consultant.

Transcript

1. cloud chaos and microservices mayhem Holly Cummins Red Hat @holly_cummins OOP Virtual
4. #RedHat @holly_cummins things you need to do well in 2023:
5. #RedHat @holly_cummins handwashing (still) things you need to do well in 2023:
6. #RedHat @holly_cummins handwashing (still) apps things you need to do well in 2023:
7. #RedHat @holly_cummins handwashing (still) apps ops things you need to do well in 2023:
8. #RedHat @holly_cummins handwashing (still) apps ops devops things you need to do well in 2023:
9. #RedHat @holly_cummins handwashing (still) apps ops devops devsecops things you need to do well in 2023:
10. #RedHat @holly_cummins handwashing (still) apps ops devops devsecops finops things you need to do well in 2023:
13. #RedHat @holly_cummins a few
14. #RedHat @holly_cummins
15. #RedHat @holly_cummins technology changes fast (duh)
16. #RedHat @holly_cummins technology changes fast (duh) … but it’s taking us a while to catch up to cloud
17. #RedHat @holly_cummins technology changes fast (duh) … but it’s taking us a while to catch up to cloud
18. #IBMGarage + IBM Cloud © 2020 IBM Corporation @holly_cummins tracing
19. #IBMGarage + IBM Cloud © 2020 IBM Corporation @holly_cummins observability
20. #RedHat @holly_cummins hardware old way logs
21. #RedHat @holly_cummins hardware old way cloud way logs container logs
22. #RedHat @holly_cummins hardware old way cloud way logs logs
23. #RedHat @holly_cummins hardware old way cloud way logs oops
24. #RedHat @holly_cummins hardware old way cloud way logs container logs
25. #RedHat @holly_cummins hardware old way cloud way logs container logs
26. #RedHat @holly_cummins hardware old way cloud way logs logs
27. #RedHat @holly_cummins hardware old way cloud way logs logs container logs container logs container logs container logs
28. #RedHat @holly_cummins hardware old way cloud way logs logs container logs container logs container logs container logs
29. #RedHat @holly_cummins hardware old way cloud way logs logs container logs container logs container logs container logs aggregation + correlation
30. #RedHat @holly_cummins hardware old way cloud way logs logs container logs container logs container logs container logs aggregation + correlation open telemetry (open tracing)
31. #RedHat @holly_cummins
32. #RedHat @holly_cummins good news: observability is ‘fixed’
33. #RedHat @holly_cummins good news: observability is ‘fixed’ bad news: not all of us are using it
34. #IBMGarage + IBM Cloud © 2020 IBM Corporation @holly_cummins performance requirements
35. #RedHat @holly_cummins memory footprint is money startup time is money
36. #RedHat @holly_cummins all that dynamism we worked so hard is (now) a bad idea
37. #RedHat @holly_cummins ahead of time optimisation is (now) a good idea
38. #IBMGarage + IBM Cloud © 2020 IBM Corporation @holly_cummins performance optimisation
39. #RedHat @holly_cummins
40. #RedHat @holly_cummins garbage collection and microservices
41. #RedHat @holly_cummins garbage collection and microservices
42. #RedHat @holly_cummins garbage collection and microservices
43. #RedHat @holly_cummins garbage collection and microservices
44. #RedHat @holly_cummins garbage collection and microservices if you chain 100 microservices together, you are almost guaranteed a GC pause
45. #RedHat @holly_cummins
46. #RedHat @holly_cummins
47. #RedHat @holly_cummins the JVM tunes itself differently on servers
48. #RedHat @holly_cummins the JVM tunes itself differently on servers … but what’s a server?
49. #RedHat @holly_cummins answer(ish):
50. #RedHat @holly_cummins // This is the working definition of a server class machine: // >= 2 physical CPU's and >=2GB of memory answer(ish): actual hotspot source code
51. #RedHat @holly_cummins
52. #RedHat @holly_cummins takeaway: don’t shrink your containers too much or your JVM will make bad decisions advice accurate as of June 2022. all performance advice should be independently verified. do not try this at home without testing first. your mileage may vary.
53. #RedHat @holly_cummins wait, isn’t that the opposite of the advice in the previous section? should my containers be big or small?
54. #RedHat @holly_cummins this is the chaos part :) wait, isn’t that the opposite of the advice in the previous section? should my containers be big or small?
55. #IBMGarage + IBM Cloud © 2020 IBM Corporation @holly_cummins management
56. #IBMGarage + IBM Cloud © 2020 IBM Corporation @holly_cummins keeping track of hardware
57. #RedHat @holly_cummins The cloud makes it so easy to provision hardware.
58. #RedHat @holly_cummins That doesn’t mean the hardware is free.
59. #RedHat @holly_cummins That doesn’t mean the hardware is free.
60. #RedHat @holly_cummins That doesn’t mean the hardware is free. Or useful.
61. #RedHat @holly_cummins That doesn’t mean the hardware is free. Or useful. Or shuts itself off.
62. #RedHat @holly_cummins Hey boss, I created a Kubernetes cluster.
63. #RedHat @holly_cummins Hey boss, I created a Kubernetes cluster. I forgot it for 2 months.
64. #RedHat @holly_cummins Hey boss, I created a Kubernetes cluster. I forgot it for 2 months. … and it’s €1000 a month.
65. Photo by daveynin, flickr
66. #RedHat @holly_cummins 2017 survey 25% of 16,000 servers doing no useful work
67. #RedHat @holly_cummins 2017 survey 25% of 16,000 servers doing no useful work “perhaps someone forgot to turn them off”
68. #RedHat @holly_cummins finops figuring out who in your company forgot to turn off their cloud
69. #RedHat @holly_cummins companies that went out of business for cloud cost accidents
70. @holly_cummins #RedHat
71. #RedHat @holly_cummins (See Troy tomorrow at GOTO !)
72. #RedHat @holly_cummins (See Troy tomorrow at GOTO !)
73. #RedHat @holly_cummins $11,448.30 (See Troy tomorrow at GOTO !)
74. #RedHat @holly_cummins “I did find a setting on your plan that set the max cacheable file size to 15GB and it looks like your zipfile is 18GB big” $11,448.30 (See Troy tomorrow at GOTO !)
75. #RedHat @holly_cummins
76. #RedHat @holly_cummins maybe this is the chaos part
77. #IBMGarage + IBM Cloud © 2020 IBM Corporation @holly_cummins releasing
78. #IBMGarage + IBM Cloud © 2020 IBM Corporation @holly_cummins releasing
79. #IBMGarage + IBM Cloud © 2020 IBM Corporation @holly_cummins microservices
80. #IBMGarage + IBM Cloud © 2020 IBM Corporation @holly_cummins microservices
81. #IBMGarage + IBM Cloud © 2020 IBM Corporation @holly_cummins
82. #IBMGarage + IBM Cloud © 2020 IBM Corporation @holly_cummins “we’re moving too slowly.
83. #IBMGarage + IBM Cloud © 2020 IBM Corporation @holly_cummins “we’re moving too slowly. we should modernise our COBOL application into microservices.
84. #IBMGarage + IBM Cloud © 2020 IBM Corporation @holly_cummins “we’re moving too slowly. we should modernise our COBOL application into microservices. … but our release board only meets twice a year.”
85. #IBMGarage + IBM Cloud © 2020 IBM Corporation @holly_cummins modularity
86. #RedHat @holly_cummins microservices are not the goal
87. #RedHat @holly_cummins microservices are not the goal they are the means
88. #IBMGarage + IBM Cloud © 2020 IBM Corporation @holly_cummins wishful mimicry
89. #IBMGarage + IBM Cloud © 2020 IBM Corporation @holly_cummins cloud != microservices
90. #IBMGarage + IBM Cloud © 2020 IBM Corporation @holly_cummins cloud native != microservices
91. #IBMGarage + IBM Cloud © 2020 IBM Corporation @holly_cummins “every time we touch one microservice, all the others break.”
92. #RedHat @holly_cummins distributed monolith
93. #RedHat @holly_cummins distributed monolith but without compile-time checking … or guaranteed function execution
94. #RedHat @holly_cummins distributed monolith but without compile-time checking … or guaranteed function execution
95. #RedHat @holly_cummins distributed monolith but without compile-time checking … or guaranteed function execution
96. #RedHat @holly_cummins distributed monolith but without compile-time checking … or guaranteed function execution
97. #RedHat @holly_cummins “each of our microservices has duplicated the same object model … with twenty classes and seventy fields”
98. #RedHat @holly_cummins Microservice Domain
99. #RedHat @holly_cummins Microservice Domain
100. #RedHat @holly_cummins Microservice Domain (this is bad)
101. #IBMGarage + IBM Cloud © 2020 IBM Corporation @holly_cummins distributed != decoupled
102. #IBMGarage + IBM Cloud © 2020 IBM Corporation @holly_cummins “ohhhh, we weren’t expecting your service to do that … “
103. #IBMGarage + IBM Cloud © 2020 IBM Corporation @holly_cummins “uh, what do you mean you corrected the typo in your json?”
104. #IBMGarage + IBM Cloud © 2020 IBM Corporation @holly_cummins If you’re going to do microservices, you need to get good at automation. And testing.
105. IBM Cloud © 2020 IBM Corporation
106. IBM Cloud © 2020 IBM Corporation the test pyramid
107. IBM Cloud © 2020 IBM Corporation the test pyramid end-to-end tests
108. IBM Cloud © 2020 IBM Corporation the test pyramid end-to-end tests unit tests
109. IBM Cloud © 2020 IBM Corporation the test pyramid end-to-end tests unit tests contract tests
110. IBM Cloud © 2020 IBM Corporation the test pyramid (you can TDD at every level!) end-to-end tests unit tests contract tests
111. #RedHat @holly_cummins How to test a fire alarm?
112. IBM Cloud © 2020 IBM Corporation how not to test a fire alarm
113. IBM Cloud © 2020 IBM Corporation how not to test a fire alarm
114. #RedHat @holly_cummins
115. #RedHat @holly_cummins unit testing a fire alarm
116. #RedHat @holly_cummins uh … is that enough?
117. #RedHat @holly_cummins contract testing a fire alarm
118. #RedHat @holly_cummins contract testing a fire alarm
119. @holly_cummins to the code!
120. @holly_cummins #RedHat cold person knitter sheep farmer demo recap:
121. @holly_cummins #RedHat cold person knitter sheep farmer “white sweater, please” demo recap:
122. @holly_cummins #RedHat cold person knitter sheep farmer “white sweater, please” demo recap:
123. @holly_cummins #RedHat cold person white sweater knitter sheep farmer “white sweater, please” demo recap:
124. @holly_cummins #RedHat cold person white sweater knitter sheep farmer “white sweater, please” white wool demo recap:
125. @holly_cummins #RedHat cold person white sweater knitter sheep farmer “white sweater, please” “black sweater, please” white wool demo recap:
126. @holly_cummins #RedHat cold person white sweater knitter sheep farmer “white sweater, please” “black sweater, please” white wool demo recap:
127. @holly_cummins #RedHat cold person white sweater knitter sheep farmer “white sweater, please” “black sweater, please” white wool black sweater demo recap:
128. @holly_cummins #RedHat cold person white sweater knitter sheep farmer “white sweater, please” “black sweater, please” white wool black sweater black wool demo recap:
129. @holly_cummins #RedHat cold person white sweater knitter sheep farmer “white sweater, please” “black sweater, please” “pink sweater, please” white wool black sweater black wool demo recap:
130. @holly_cummins #RedHat cold person white sweater knitter sheep farmer “white sweater, please” “black sweater, please” “pink sweater, please” white wool black sweater black wool demo recap:
131. @holly_cummins #RedHat cold person white sweater knitter sheep farmer “white sweater, please” “black sweater, please” “pink sweater, please” white wool black sweater black wool pink sweater demo recap:
132. @holly_cummins #RedHat cold person white sweater knitter sheep farmer “white sweater, please” “black sweater, please” “pink sweater, please” white wool black sweater black wool pink sweater pink wool demo recap:
133. @holly_cummins #RedHat cold person white sweater knitter sheep farmer “white sweater, please” “black sweater, please” “pink sweater, please” white wool black sweater black wool pink sweater pink wool what?! pink sheep don’t exist demo recap:
134. @holly_cummins #RedHat cold person white sweater knitter sheep farmer “white sweater, please” “black sweater, please” “pink sweater, please” white wool black sweater black wool pink sweater pink wool should the knitter ask for white wool and bring in a dyeing service? what?! pink sheep don’t exist demo recap:
135. @holly_cummins #RedHat cold person white sweater knitter sheep farmer “white sweater, please” “black sweater, please” “pink sweater, please” white wool black sweater black wool pink sweater pink wool should the knitter ask for white wool and bring in a dyeing service? should the farmer dye the wool? what?! pink sheep don’t exist demo recap:
136. @holly_cummins #RedHat Photo of sheep at the Latitude Festival by Tim Parkinson, https://www.flickr.com/photos/timparkinson
137. @holly_cummins #RedHat Photo of sheep at the Latitude Festival by Tim Parkinson, https://www.flickr.com/photos/timparkinson “pink sheep don’t exist”
138. #RedHat @holly_cummins demo resources: https://github.com/holly-cummins/pact-quarkus-sweater-demo
139. #RedHat @holly_cummins demo recap: • consumer-driven contract testing can help avoid disaster • pact is a mock for the consumer • pact is a functional test for the producer • shared json contracts aligns expectations across services
140. #RedHat @holly_cummins so … don’t assume cloud is a solved problem don’t assume microservices will solve all your problems contract testing may solve some of your problems
141. @holly_cummins slides