Why Cloud Zombies Are Destroying the Planet and How You Can Stop Them

Holly Cummins Red Hat QCon London | March 29, 2023
Why Cloud Zombies Are Destroying the Planet and How You Can Stop Them

@holly_cummins #RedHat

@therealmarkw1, twitter

what do these servers do? @therealmarkw1, twitter

what do these servers do? one is a backup for
the other. @therealmarkw1, twitter

the other. yes, but what do they do? @therealmarkw1, twitter

the other. yes, but what do they do? @therealmarkw1, twitter no one has known for a couple of decades

#RedHat @[email protected] Hey boss, I created a Kubernetes cluster. 2018

#RedHat @[email protected] Hey boss, I created a Kubernetes cluster. I
forgot it for 2 months. 2018

#RedHat @[email protected] Hey boss, I created a Kubernetes cluster. I
forgot it for 2 months. … and it’s €1000 a month. 2018

#RedHat @[email protected] Hey boss, while I was working on a
QCon talk about sustainability … 2023

QCon talk about sustainability … I left the Quarkus CI on Mac disabled 2023

QCon talk about sustainability … … and the instance is $159 a month. I left the Quarkus CI on Mac disabled 2023

@holly_cummins #RedHat “measure, don’t guess” (or decide based on stories
on the internet)

@holly_cummins #RedHat actual picture of a zombie (it’s invisible)

#RedHat @[email protected] 2015 survey 30% of 4,000 servers doing no
useful work

#RedHat @[email protected] 2017 survey 25% of 16,000 servers doing no
useful work

#RedHat @[email protected] zombie “they haven't delivered any information or computing
services for six months or more”

#RedHat @[email protected] “comatose servers”

#RedHat @[email protected] under-utilised servers

#RedHat @[email protected] “much of the energy consumed by U.S. data
centers is used to power more than 12 million servers that do little or no work most of the time” NRDC

#RedHat @[email protected] the average server: 12 - 18% of capacity
30 - 60 % of maximum power https://www.nrdc.org/sites/default/files/data-center-efficiency-assessment-IB.pdf

#RedHat @[email protected] 2014 survey 29% of 4,000 active less than
5% of the time https://www.anthesisgroup.com/wp-content/uploads/2019/11/Comatose-Servers-Redux-2017.pdf

@holly_cummins #RedHat https://www.business2community.com/cloud-computing/overprovisioning-always-on-resources-lead-to-26-6-billion-in-public-cloud-waste-expected-in-2021-02381033 2021 study

@holly_cummins #RedHat $26.6 billion https://www.business2community.com/cloud-computing/overprovisioning-always-on-resources-lead-to-26-6-billion-in-public-cloud-waste-expected-in-2021-02381033 2021 study

@holly_cummins #RedHat $26.6 billion wasted by always-on cloud instances https://www.business2community.com/cloud-computing/overprovisioning-always-on-resources-lead-to-26-6-billion-in-public-cloud-waste-expected-in-2021-02381033
2021 study

#RedHat @[email protected] it’s not just runtime costs

#RedHat @[email protected] embodied carbon it’s not just runtime costs

#RedHat @[email protected] why does this happen?

@holly_cummins #RedHat managing machines is hard

#RedHat @[email protected]

#RedHat @[email protected] “perhaps someone forgot to turn them off” Antithesis
Institute

#RedHat @[email protected]

#RedHat @[email protected] projects ended

#RedHat @[email protected] projects ended business processes changed

#RedHat @[email protected] projects ended business processes changed over-provisioning

#RedHat @[email protected] projects ended business processes changed over-provisioning isolation requirements

@holly_cummins #RedHat risk averse processes

@holly_cummins #RedHat “we run this as a batch job on
weekends, but the servers stay up all week” “

@holly_cummins #RedHat “we run this as a batch job on
weekends, but the servers stay up all week”

@holly_cummins #RedHat “we only use this system in UK working
hours, but we leave it running 24/7 ” “

@holly_cummins #RedHat “we only use this system in UK working
hours, but we leave it running 24/7 ”

@holly_cummins #RedHat auto-scaling algorithms are optimised for availability

@holly_cummins #RedHat green computing model: the four vowels

@holly_cummins #RedHat green computing model: the four vowels elasticity

@holly_cummins #RedHat green computing model: the four vowels elasticity utilisation

efficiency

efficiency utility

@holly_cummins #RedHat application utilisation

@holly_cummins #RedHat application utilisation high utilisation good case

@holly_cummins #RedHat application utilisation over-utilisation very bad case

@holly_cummins #RedHat application utilisation over-utilisation very bad case under-utilisation wasteful
case

@holly_cummins #RedHat application elasticity high utilisation good case @holly_cummins

@holly_cummins #RedHat application elasticity scale-up good utilisation @holly_cummins

@holly_cummins #RedHat application elasticity scale-down good utilisation @holly_cummins

efficiency utility

@holly_cummins #RedHat There is nothing so useless as doing efficiently
that which should not be done at all. Peter Drucker why utility matters

@holly_cummins #RedHat “efficient zombies”

@holly_cummins #RedHat how do we solve the zombie problem?

@holly_cummins #RedHat how do we solve the zombie problem? detection
and destruction

@holly_cummins #RedHat system archaeology … is not easy

@holly_cummins #RedHat scream test

@holly_cummins #RedHat “eco-monkey”

@holly_cummins #RedHat the scream is real

@holly_cummins #RedHat the scream is real this internal server doesn’t
seem to have a purpose

seem to have a purpose let’s turn it off!

seem to have a purpose uh … why did the backbone of a client’s network just vanish? let’s turn it off!

seem to have a purpose uh … why did the backbone of a client’s network just vanish? let’s turn it off! oops.

@holly_cummins #RedHat IT Department, UK Bank let’s figure out what
all these cloud workloads are, since I’m paying for them long meetings

@holly_cummins #RedHat long emails

@holly_cummins #RedHat tags

@holly_cummins #RedHat all the —opses

@holly_cummins #RedHat GreenOps

@holly_cummins #RedHat GreenOps greenops is a mid-sized trilobite (really)

@holly_cummins #RedHat FinOps figuring out who in your company forgot
to turn off their cloud

@holly_cummins #RedHat backstage.io

@holly_cummins #RedHat backstage.io •cost insights plugin

@holly_cummins #RedHat backstage.io •cost insights plugin •cloud carbon footprint plugin

• Densify • Granulate • Turbonomic Application Resource Management •
TSO Logic • etc AIOps

21% improvement from installing Turbonomic in IBM CIO office

@holly_cummins #RedHat traffic monitoring

@holly_cummins #RedHat but. knowing is only half the battle.

@holly_cummins #RedHat the ikea effect

@holly_cummins #RedHat the ikea effect labour

@holly_cummins #RedHat the ikea effect labour love

@holly_cummins #RedHat shut it down? but … what if I
need this cluster later?

@holly_cummins #RedHat elasticity native quarkus starts faster than a light
bulb

@holly_cummins #RedHat ultimate elasticity

@holly_cummins #RedHat we don’t switch the light off because we’re
not sure if it will come back on

@holly_cummins #RedHat we don’t switch the server off because we’re
not sure if it will come back on happens all the time

@holly_cummins #RedHat we don’t switch the server off because it
would be too much work to recreate it happens all the time

@holly_cummins #RedHat turning it off and on again must

@holly_cummins #RedHat turning it off and on again must •
be fast

be fast • actually work

be fast • actually work • idempotency

be fast • actually work • idempotency • resiliency

@holly_cummins #RedHat making turning servers off as safe and easy
as turning lights off

@holly_cummins #RedHat LightSwitchOps making turning servers off as safe and
easy as turning lights off

@holly_cummins #RedHat simple scripts we used to leave our applications
running all the time @darkandnerdy, Chicago DevOpsDays

@holly_cummins #RedHat simple scripts we used to leave our applications
running all the time when we scripted turning them off at night, we reduced our cloud bill by 30% @darkandnerdy, Chicago DevOpsDays

@holly_cummins #RedHat GitOps

@holly_cummins #RedHat GitOps (infrastructure as code)

@holly_cummins #RedHat spin it down

@holly_cummins #RedHat kubectl apply -f all-my-cluster/ spin it down spin
it up

@holly_cummins #RedHat kubectl apply -f all-my-cluster/ ansible-playbook stuff.yml spin it
down spin it up

reducing snowflakes reduces redundancy

we need to have another copy of our expensive cluster
in another region so we have failover!

we need to have another copy of our expensive cluster
in another region so we have failover! uh … sounds expensive. are you sure about that?

rapid recovery does not require redundant servers

zombie reduction does not need to be fancy

@holly_cummins #RedHat large bank, 2013 50% reduction in CPUs with
a lease system

things that (maybe) don’t help

@holly_cummins #RedHat things that (maybe) don’t help “out of sight,
out of mind” cloud

@holly_cummins #RedHat things that (maybe) don’t help virtualisation 2019 survey
30% of virtual servers doing no useful work

@holly_cummins #RedHat things that (maybe) don’t help virtualisation 2019 survey
30% of virtual servers doing no useful work 50% of virtual servers active less than 5% of the time

#RedHat @[email protected] you still need to remember to turn the
virtual machine off

what about serverless?

modernising to serverless is a big lift

may not suit latency-sensitive workloads

“we solve the cold-start problem by … … keeping an
instance running but not billing you”

@holly_cummins #RedHat application serverless systems may have high overheads

@holly_cummins #RedHat control plane application serverless systems may have high
overheads

https://hotcarbon.org/pdf/hotcarbon22-sharma.pdf

https://hotcarbon.org/pdf/hotcarbon22-sharma.pdf virtualisation overheads mean each function request can use 30x
more energy than a plain http server

are all parts of the system elastic?

things that definitely don’t help

@holly_cummins #RedHat things that don’t help prevention

@holly_cummins #RedHat things that don’t help prevention (?!)

surely shutting the barn door before the horse has left
is a good idea?

prevention == heavy governance

remember the ikea effect?

remember the ikea effect? people will not surrender servers that
were hard to get

zombies are not just servers

traffic

zombie packets

@holly_cummins #RedHat internet background noise

@holly_cummins #RedHat internet background noise 5.5 gigabits/s

@holly_cummins #RedHat unsolved problem == opportunity

@holly_cummins #RedHat the double-win turning things off saves a lot
of money

@holly_cummins #RedHat users …

@holly_cummins #RedHat up utilisation aim for elasticity limit kubesprawl de-zombify
know what you’re using turn it off users …

@holly_cummins #RedHat 1-2%

@holly_cummins #RedHat tool creators, support 1-2%

@holly_cummins #RedHat better utilisation elasticity multi-tenancy de-zombification visibility disposability tool
creators, support 1-2%

GreenOps FinOps AIOps GitOps LightSwitchOps

thank you @[email protected] slides

Why Cloud Zombies Are Destroying the Planet and...

Why Cloud Zombies Are Destroying the Planet and How You Can Stop Them

More Decks by Holly Cummins

Other Decks in Programming

Featured

Transcript