Why Cloud Zombies Are Destroying the Planet and How You Can Stop Them

Holly Cummins

Red Hat

QCon London | March 29, 2023
Why Cloud Zombies Are
Destroying the Planet

and

How You Can Stop Them

View Slide

@holly_cummins #RedHat

View Slide

@therealmarkw1, twitter

View Slide

what do these servers do?

View Slide

one is a backup for the
other.

View Slide

other.
yes, but what do they do?

View Slide

other.
yes, but what do they do?
no one has known for a
couple of decades

View Slide

#RedHat
@[email protected]
Hey boss, I created a
Kubernetes cluster.
2018

View Slide

#RedHat
@[email protected]
Kubernetes cluster.
I forgot it for 2 months.
2018

View Slide

#RedHat
@[email protected]
Kubernetes cluster.
I forgot it for 2 months.
… and it’s €1000 a month.
2018

View Slide

#RedHat
@[email protected]
Hey boss, while I was
working on a QCon talk
about sustainability …
2023

View Slide

#RedHat
@[email protected]
I left the Quarkus CI
on Mac disabled
2023

View Slide

#RedHat
@[email protected]
… and the instance is $159 a
month.
I left the Quarkus CI
on Mac disabled
2023

View Slide

“measure, don’t guess”

(or decide based on stories on the internet)

View Slide

actual picture of a zombie

(it’s invisible)

View Slide

#RedHat
@[email protected]
2015 survey

30%

of 4,000 servers doing
no useful work

View Slide

#RedHat
@[email protected]
2017 survey

25%

of 16,000 servers doing
no useful work

View Slide

#RedHat
@[email protected]
zombie

“they haven't delivered any information or
computing services for six months or more”

View Slide

#RedHat
@[email protected]
“comatose servers”

View Slide

#RedHat
@[email protected]
under-utilised servers

View Slide

#RedHat
@[email protected]
“much of the energy consumed by U.S. data
centers is used to power more than 12
million servers that do little or no work most
of the time”

NRDC

View Slide

#RedHat
@[email protected]
the average server:

12 - 18% of capacity

30 - 60 % of maximum power
https://www.nrdc.org/sites/default/files/data-center-efficiency-assessment-IB.pdf

View Slide

#RedHat
@[email protected]
2014 survey

29%

of 4,000 active less than
5% of the time
https://www.anthesisgroup.com/wp-content/uploads/2019/11/Comatose-Servers-Redux-2017.pdf

View Slide

https://www.business2community.com/cloud-computing/overprovisioning-always-on-resources-lead-to-26-6-billion-in-public-cloud-waste-expected-in-2021-02381033
2021 study

View Slide

$26.6 billion
2021 study

View Slide

$26.6 billion
wasted by always-on
cloud instances
2021 study

View Slide

#RedHat
@[email protected]
it’s not just runtime costs

View Slide

#RedHat
@[email protected]
embodied carbon
it’s not just runtime costs

View Slide

#RedHat
@[email protected]
why does this happen?

View Slide

managing machines is hard

View Slide

#RedHat
@[email protected]

View Slide

#RedHat
@[email protected]
“perhaps someone
forgot to turn them off”

Antithesis Institute

View Slide

#RedHat
@[email protected]

View Slide

#RedHat
@[email protected]
projects ended

View Slide

#RedHat
@[email protected]
projects ended
business processes changed

View Slide

#RedHat
@[email protected]
projects ended
over-provisioning

View Slide

#RedHat
@[email protected]
projects ended
over-provisioning
isolation requirements

View Slide

risk averse processes

View Slide

“we run this as a batch job on weekends,
but the servers stay up all week”
“

View Slide

“we run this as a batch job on weekends,
but the servers stay up all week”

View Slide

“we only use this system in UK working hours,
but we leave it running 24/7 ”
“

View Slide

“we only use this system in UK working hours,
but we leave it running 24/7 ”

View Slide

auto-scaling algorithms are optimised for availability

View Slide

green computing model: the four vowels

View Slide

elasticity

View Slide

elasticity
utilisation

View Slide

elasticity
utilisation
efficiency

View Slide

elasticity
utilisation
efficiency
utility

View Slide

application
utilisation

View Slide

application
utilisation
high utilisation

good case

View Slide

application
utilisation
over-utilisation

very bad case

View Slide

application
utilisation
over-utilisation

very bad case
under-utilisation

wasteful case

View Slide

application
elasticity
high utilisation

good case
@holly_cummins

View Slide

application
elasticity
scale-up

good utilisation
@holly_cummins

View Slide

application
elasticity
scale-down

good utilisation
@holly_cummins

View Slide

elasticity
utilisation
efficiency
utility

View Slide

There is nothing so useless as
doing efficiently that which
should not be done at all.

Peter Drucker
why utility matters

View Slide

“efficient zombies”

View Slide

how do we solve the zombie problem?

View Slide

how do we solve the zombie problem?
detection and destruction

View Slide

View Slide

system archaeology
… is not easy

View Slide

scream test

View Slide

“eco-monkey”

View Slide

@holly_cummins
#RedHat
the scream is real

View Slide

@holly_cummins
#RedHat
the scream is real
this internal server
doesn’t seem to have
a purpose

View Slide

@holly_cummins
#RedHat
the scream is real
a purpose
let’s turn it off!

View Slide

@holly_cummins
#RedHat
the scream is real
a purpose
uh … why did the
backbone of a
client’s network
just vanish?

View Slide

@holly_cummins
#RedHat
the scream is real
a purpose
uh … why did the
backbone of a
client’s network
just vanish?
oops.

View Slide

IT Department, UK Bank
let’s figure out what all
these cloud workloads are,
since I’m paying for them
long meetings

View Slide

long emails

View Slide

tags

View Slide

all the —opses

View Slide

GreenOps

View Slide

GreenOps
greenops is a mid-sized trilobite (really)

View Slide

FinOps
figuring out who in your company forgot to turn off their cloud

View Slide


View Slide

backstage.io

View Slide

backstage.io
•cost insights plugin

View Slide

backstage.io
•cost insights plugin
•cloud carbon footprint plugin

View Slide

• Densify

• Granulate

• Turbonomic Application Resource Management

• TSO Logic

• etc
AIOps

View Slide

21%
improvement from installing Turbonomic

in IBM CIO office

View Slide

traffic monitoring

View Slide

but.

knowing is only half the battle.

View Slide

the ikea effect

View Slide

the ikea effect
labour

View Slide

the ikea effect
labour love

View Slide

shut it down?

but … what if I
need this
cluster later?

View Slide

elasticity
native quarkus starts
faster than a light bulb

View Slide

@holly_cummins
#RedHat
ultimate elasticity

View Slide

we don’t switch the light off
because we’re not sure if it will
come back on

View Slide

we don’t switch the server off
because we’re not sure if it will come
back on
happens all the time

View Slide

we don’t switch the server off
because it would be too much work
to recreate it
happens all the time

View Slide

@holly_cummins
#RedHat

View Slide

@holly_cummins
#RedHat
turning it off and on again must

View Slide

@holly_cummins
#RedHat
• be fast

View Slide

@holly_cummins
#RedHat
• be fast
• actually work

View Slide

@holly_cummins
#RedHat
• be fast
• actually work
• idempotency

View Slide

@holly_cummins
#RedHat
• be fast
• actually work
• idempotency
• resiliency

View Slide

@holly_cummins
#RedHat
making turning servers off as safe and easy as turning lights off

View Slide

@holly_cummins
#RedHat
LightSwitchOps
making turning servers off as safe and easy as turning lights off

View Slide

@holly_cummins
#RedHat
simple scripts
we used to leave
our applications
running all the time
@darkandnerdy, Chicago DevOpsDays

View Slide

@holly_cummins
#RedHat
simple scripts
we used to leave
our applications
running all the time
when we
scripted turning
them off at night,
we reduced our
cloud bill by
30%
@darkandnerdy, Chicago DevOpsDays

View Slide


View Slide

GitOps

View Slide

GitOps
(infrastructure as code)

View Slide


View Slide

spin it down

View Slide

kubectl apply -f all-my-cluster/
spin it down
spin it up

View Slide

kubectl apply -f all-my-cluster/
ansible-playbook stuff.yml
spin it down
spin it up

View Slide

reducing snowflakes
reduces redundancy

View Slide

we need to have another
copy of our expensive cluster in
another region so we have
failover!

View Slide

we need to have another
copy of our expensive cluster in
another region so we have
failover!
uh … sounds
expensive. are you
sure about that?

View Slide

rapid recovery does not
require redundant servers

View Slide

zombie reduction does
not need to be fancy

View Slide

large bank, 2013

50%

reduction in CPUs with a
lease system

View Slide

things that (maybe) don’t help

View Slide


“out of sight, out of mind”
cloud

View Slide


View Slide


virtualisation
2019 survey

30%

of virtual servers doing
no useful work

View Slide


virtualisation
2019 survey

30%

of virtual servers doing
no useful work

50%

of virtual servers active
less than 5% of the time

View Slide

#RedHat
@[email protected]
you still need to remember to
turn the virtual machine off

View Slide

what about serverless?

View Slide

modernising to serverless is a big lift

View Slide

may not suit latency-sensitive workloads

View Slide

“we solve the cold-start problem by …

… keeping an instance running but not billing you”

View Slide

application
serverless systems may have high overheads

View Slide

control plane
application
serverless systems may have high overheads

View Slide

https://hotcarbon.org/pdf/hotcarbon22-sharma.pdf

View Slide

https://hotcarbon.org/pdf/hotcarbon22-sharma.pdf
virtualisation overheads
mean each function request
can use 30x more energy
than a plain http server

View Slide

are all parts of the system elastic?

View Slide

things that definitely don’t help

View Slide

things that don’t help

prevention

View Slide

things that don’t help

prevention (?!)

View Slide

surely shutting the barn door before
the horse has left is a good idea?

View Slide

prevention == heavy governance

View Slide

remember the ikea effect?

View Slide

remember the ikea effect?
people will not surrender
servers that were hard to get

View Slide

zombies are not just servers

View Slide

data

View Slide

traffic

View Slide

zombie packets

View Slide

internet background noise

View Slide

internet background noise
5.5 gigabits/s

View Slide

unsolved problem == opportunity

View Slide

the double-win
turning things off saves a lot of money

View Slide


View Slide

users …

View Slide

up utilisation

aim for elasticity

limit kubesprawl

de-zombify

know what you’re using

turn it off
users …

View Slide

1-2%

View Slide

tool creators, support
1-2%

View Slide

better utilisation

elasticity

multi-tenancy

de-zombification

visibility

disposability
tool creators, support
1-2%

View Slide

GreenOps

FinOps

AIOps

GitOps

LightSwitchOps

View Slide

thank you

@[email protected]
slides

View Slide

Why Cloud Zombies Are Destroying the Planet and How You Can Stop Them

Why Cloud Zombies Are Destroying the Planet and How You Can Stop Them

Holly Cummins

More Decks by Holly Cummins

Other Decks in Programming

Featured

Transcript