Why Cloud Zombies Are Destroying the Planet and How You Can Stop Them

Wait, zombies? Really? Zombies are servers which aren’t doing useful work. They’re everywhere, costing money, eating electricity, and belching carbon. And they’re useless! So how do we get rid of them? In this talk, Holly will explain how utilization and elasticity relate to sustainability. She will also introduce a range of practical zombie-hunting techniques, including absurdly-simple-automation, LightSwitchOps, and FinOps.

Transcript

1. Holly Cummins Red Hat QCon London | March 29, 2023 Why Cloud Zombies Are Destroying the Planet and How You Can Stop Them
2. Holly Cummins Red Hat QCon London | March 29, 2023 Why Cloud Zombies Are Destroying the Planet and How You Can Stop Them
3. @holly_cummins #RedHat
4. @holly_cummins #RedHat
5. @therealmarkw1, twitter
6. what do these servers do? @therealmarkw1, twitter
7. what do these servers do? one is a backup for the other. @therealmarkw1, twitter
8. what do these servers do? one is a backup for the other. yes, but what do they do? @therealmarkw1, twitter
9. what do these servers do? one is a backup for the other. yes, but what do they do? @therealmarkw1, twitter no one has known for a couple of decades
10. #RedHat @holly_cummins@hachyderm.io Hey boss, I created a Kubernetes cluster. 2018
11. #RedHat @holly_cummins@hachyderm.io Hey boss, I created a Kubernetes cluster. I forgot it for 2 months. 2018
12. #RedHat @holly_cummins@hachyderm.io Hey boss, I created a Kubernetes cluster. I forgot it for 2 months. … and it’s €1000 a month. 2018
13. #RedHat @holly_cummins@hachyderm.io Hey boss, while I was working on a QCon talk about sustainability … 2023
14. #RedHat @holly_cummins@hachyderm.io Hey boss, while I was working on a QCon talk about sustainability … I left the Quarkus CI on Mac disabled 2023
15. #RedHat @holly_cummins@hachyderm.io Hey boss, while I was working on a QCon talk about sustainability … … and the instance is $159 a month. I left the Quarkus CI on Mac disabled 2023
16. @holly_cummins #RedHat “measure, don’t guess” (or decide based on stories on the internet)
17. @holly_cummins #RedHat actual picture of a zombie (it’s invisible)
18. @holly_cummins #RedHat actual picture of a zombie (it’s invisible)
19. #RedHat @holly_cummins@hachyderm.io 2015 survey 30% of 4,000 servers doing no useful work
20. #RedHat @holly_cummins@hachyderm.io 2017 survey 25% of 16,000 servers doing no useful work
21. #RedHat @holly_cummins@hachyderm.io zombie “they haven't delivered any information or computing services for six months or more”
22. #RedHat @holly_cummins@hachyderm.io “comatose servers”
23. #RedHat @holly_cummins@hachyderm.io under-utilised servers
24. #RedHat @holly_cummins@hachyderm.io “much of the energy consumed by U.S. data centers is used to power more than 12 million servers that do little or no work most of the time” NRDC
25. #RedHat @holly_cummins@hachyderm.io the average server: 12 - 18% of capacity 30 - 60 % of maximum power https://www.nrdc.org/sites/default/files/data-center-efficiency-assessment-IB.pdf
26. #RedHat @holly_cummins@hachyderm.io 2014 survey 29% of 4,000 active less than 5% of the time https://www.anthesisgroup.com/wp-content/uploads/2019/11/Comatose-Servers-Redux-2017.pdf
27. @holly_cummins #RedHat https://www.business2community.com/cloud-computing/overprovisioning-always-on-resources-lead-to-26-6-billion-in-public-cloud-waste-expected-in-2021-02381033 2021 study
28. @holly_cummins #RedHat $26.6 billion https://www.business2community.com/cloud-computing/overprovisioning-always-on-resources-lead-to-26-6-billion-in-public-cloud-waste-expected-in-2021-02381033 2021 study
29. @holly_cummins #RedHat $26.6 billion wasted by always-on cloud instances https://www.business2community.com/cloud-computing/overprovisioning-always-on-resources-lead-to-26-6-billion-in-public-cloud-waste-expected-in-2021-02381033 2021 study
30. #RedHat @holly_cummins@hachyderm.io it’s not just runtime costs
31. #RedHat @holly_cummins@hachyderm.io embodied carbon it’s not just runtime costs
32. #RedHat @holly_cummins@hachyderm.io why does this happen?
33. @holly_cummins #RedHat managing machines is hard
34. @holly_cummins #RedHat managing machines is hard
35. #RedHat @holly_cummins@hachyderm.io
36. #RedHat @holly_cummins@hachyderm.io “perhaps someone forgot to turn them off” Antithesis Institute
37. #RedHat @holly_cummins@hachyderm.io
38. #RedHat @holly_cummins@hachyderm.io projects ended
39. #RedHat @holly_cummins@hachyderm.io projects ended business processes changed
40. #RedHat @holly_cummins@hachyderm.io projects ended business processes changed over-provisioning
41. #RedHat @holly_cummins@hachyderm.io projects ended business processes changed over-provisioning isolation requirements
42. @holly_cummins #RedHat risk averse processes
43. @holly_cummins #RedHat “we run this as a batch job on weekends, but the servers stay up all week” “
44. @holly_cummins #RedHat “we run this as a batch job on weekends, but the servers stay up all week”
45. @holly_cummins #RedHat “we only use this system in UK working hours, but we leave it running 24/7 ” “
46. @holly_cummins #RedHat “we only use this system in UK working hours, but we leave it running 24/7 ”
47. @holly_cummins #RedHat auto-scaling algorithms are optimised for availability
48. @holly_cummins #RedHat green computing model: the four vowels
49. @holly_cummins #RedHat green computing model: the four vowels
50. @holly_cummins #RedHat green computing model: the four vowels elasticity
51. @holly_cummins #RedHat green computing model: the four vowels elasticity utilisation
52. @holly_cummins #RedHat green computing model: the four vowels elasticity utilisation efficiency
53. @holly_cummins #RedHat green computing model: the four vowels elasticity utilisation efficiency utility
54. @holly_cummins #RedHat green computing model: the four vowels elasticity utilisation efficiency utility
55. @holly_cummins #RedHat application utilisation
56. @holly_cummins #RedHat application utilisation high utilisation good case
57. @holly_cummins #RedHat application utilisation over-utilisation very bad case
58. @holly_cummins #RedHat application utilisation over-utilisation very bad case under-utilisation wasteful case
59. @holly_cummins #RedHat application elasticity high utilisation good case @holly_cummins
60. @holly_cummins #RedHat application elasticity scale-up good utilisation @holly_cummins
61. @holly_cummins #RedHat application elasticity scale-down good utilisation @holly_cummins
62. @holly_cummins #RedHat green computing model: the four vowels elasticity utilisation efficiency utility
63. @holly_cummins #RedHat green computing model: the four vowels elasticity utilisation efficiency utility
64. @holly_cummins #RedHat There is nothing so useless as doing efficiently that which should not be done at all. Peter Drucker why utility matters
65. @holly_cummins #RedHat “efficient zombies”
66. @holly_cummins #RedHat how do we solve the zombie problem?
67. @holly_cummins #RedHat how do we solve the zombie problem? detection and destruction
69. @holly_cummins #RedHat system archaeology … is not easy
70. @holly_cummins #RedHat scream test
71. @holly_cummins #RedHat “eco-monkey”
72. @holly_cummins #RedHat the scream is real
73. @holly_cummins #RedHat the scream is real this internal server doesn’t seem to have a purpose
74. @holly_cummins #RedHat the scream is real this internal server doesn’t seem to have a purpose let’s turn it off!
75. @holly_cummins #RedHat the scream is real this internal server doesn’t seem to have a purpose uh … why did the backbone of a client’s network just vanish? let’s turn it off!
76. @holly_cummins #RedHat the scream is real this internal server doesn’t seem to have a purpose uh … why did the backbone of a client’s network just vanish? let’s turn it off! oops.
77. @holly_cummins #RedHat IT Department, UK Bank let’s figure out what all these cloud workloads are, since I’m paying for them long meetings
78. @holly_cummins #RedHat IT Department, UK Bank let’s figure out what all these cloud workloads are, since I’m paying for them long meetings
79. @holly_cummins #RedHat long emails
80. @holly_cummins #RedHat tags
81. @holly_cummins #RedHat all the —opses
82. @holly_cummins #RedHat GreenOps
83. @holly_cummins #RedHat GreenOps greenops is a mid-sized trilobite (really)
84. @holly_cummins #RedHat FinOps figuring out who in your company forgot to turn off their cloud
85. @holly_cummins #RedHat
86. @holly_cummins #RedHat backstage.io
87. @holly_cummins #RedHat backstage.io •cost insights plugin
88. @holly_cummins #RedHat backstage.io •cost insights plugin •cloud carbon footprint plugin
89. • Densify • Granulate • Turbonomic Application Resource Management • TSO Logic • etc AIOps
90. 21% improvement from installing Turbonomic in IBM CIO office
91. @holly_cummins #RedHat traffic monitoring
92. @holly_cummins #RedHat but. knowing is only half the battle.
93. @holly_cummins #RedHat the ikea effect
94. @holly_cummins #RedHat the ikea effect labour
95. @holly_cummins #RedHat the ikea effect labour
96. @holly_cummins #RedHat the ikea effect labour love
97. @holly_cummins #RedHat shut it down? but … what if I need this cluster later?
98. @holly_cummins #RedHat elasticity native quarkus starts faster than a light bulb
99. @holly_cummins #RedHat ultimate elasticity
100. @holly_cummins #RedHat we don’t switch the light off because we’re not sure if it will come back on
101. @holly_cummins #RedHat we don’t switch the server off because we’re not sure if it will come back on happens all the time
102. @holly_cummins #RedHat we don’t switch the server off because it would be too much work to recreate it happens all the time
103. @holly_cummins #RedHat
104. @holly_cummins #RedHat
105. @holly_cummins #RedHat turning it off and on again must
106. @holly_cummins #RedHat turning it off and on again must • be fast
107. @holly_cummins #RedHat turning it off and on again must • be fast • actually work
108. @holly_cummins #RedHat turning it off and on again must • be fast • actually work • idempotency
109. @holly_cummins #RedHat turning it off and on again must • be fast • actually work • idempotency • resiliency
110. @holly_cummins #RedHat making turning servers off as safe and easy as turning lights off
111. @holly_cummins #RedHat LightSwitchOps making turning servers off as safe and easy as turning lights off
112. @holly_cummins #RedHat simple scripts we used to leave our applications running all the time @darkandnerdy, Chicago DevOpsDays
113. @holly_cummins #RedHat simple scripts we used to leave our applications running all the time when we scripted turning them off at night, we reduced our cloud bill by 30% @darkandnerdy, Chicago DevOpsDays
114. @holly_cummins #RedHat
115. @holly_cummins #RedHat GitOps
116. @holly_cummins #RedHat GitOps (infrastructure as code)
117. @holly_cummins #RedHat
118. @holly_cummins #RedHat spin it down
119. @holly_cummins #RedHat kubectl apply -f all-my-cluster/ spin it down spin it up
120. @holly_cummins #RedHat kubectl apply -f all-my-cluster/ spin it down spin it up
121. @holly_cummins #RedHat kubectl apply -f all-my-cluster/ ansible-playbook stuff.yml spin it down spin it up
122. reducing snowflakes reduces redundancy
123. we need to have another copy of our expensive cluster in another region so we have failover!
124. we need to have another copy of our expensive cluster in another region so we have failover! uh … sounds expensive. are you sure about that?
125. rapid recovery does not require redundant servers
126. zombie reduction does not need to be fancy
127. @holly_cummins #RedHat large bank, 2013 50% reduction in CPUs with a lease system
128. @holly_cummins #RedHat large bank, 2013 50% reduction in CPUs with a lease system
129. things that (maybe) don’t help
130. @holly_cummins #RedHat things that (maybe) don’t help “out of sight, out of mind” cloud
131. @holly_cummins #RedHat
132. @holly_cummins #RedHat things that (maybe) don’t help virtualisation 2019 survey 30% of virtual servers doing no useful work
133. @holly_cummins #RedHat things that (maybe) don’t help virtualisation 2019 survey 30% of virtual servers doing no useful work 50% of virtual servers active less than 5% of the time
134. #RedHat @holly_cummins@hachyderm.io you still need to remember to turn the virtual machine off
135. what about serverless?
136. modernising to serverless is a big lift
137. may not suit latency-sensitive workloads
138. “we solve the cold-start problem by … … keeping an instance running but not billing you”
139. @holly_cummins #RedHat application serverless systems may have high overheads
140. @holly_cummins #RedHat control plane application serverless systems may have high overheads
141. @holly_cummins #RedHat control plane application serverless systems may have high overheads
142. @holly_cummins #RedHat control plane application serverless systems may have high overheads
143. https://hotcarbon.org/pdf/hotcarbon22-sharma.pdf
144. https://hotcarbon.org/pdf/hotcarbon22-sharma.pdf virtualisation overheads mean each function request can use 30x more energy than a plain http server
145. are all parts of the system elastic?
146. things that definitely don’t help
147. @holly_cummins #RedHat things that don’t help prevention
148. @holly_cummins #RedHat things that don’t help prevention (?!)
149. surely shutting the barn door before the horse has left is a good idea?
150. prevention == heavy governance
151. remember the ikea effect?
152. remember the ikea effect? people will not surrender servers that were hard to get
153. zombies are not just servers
154. data
155. traffic
156. zombie packets
157. @holly_cummins #RedHat internet background noise
158. @holly_cummins #RedHat internet background noise 5.5 gigabits/s
159. @holly_cummins #RedHat unsolved problem == opportunity
160. @holly_cummins #RedHat the double-win turning things off saves a lot of money
161. @holly_cummins #RedHat
162. @holly_cummins #RedHat users …
163. @holly_cummins #RedHat up utilisation aim for elasticity limit kubesprawl de-zombify know what you’re using turn it off users …
164. @holly_cummins #RedHat 1-2%
165. @holly_cummins #RedHat tool creators, support 1-2%
166. @holly_cummins #RedHat better utilisation elasticity multi-tenancy de-zombification visibility disposability tool creators, support 1-2%
167. GreenOps FinOps AIOps GitOps LightSwitchOps
168. GreenOps FinOps AIOps GitOps LightSwitchOps
169. thank you @holly_cummins@hachyderm.io slides