Measure, Don't Guess: Observability as the key to performance tuning Software Delivery

Slide 1

Slide 1 text

Trisha Gee  Head of Developer Advocacy, Gradle Measure, Don’t Guess Observability as the key to performance tuning Software Delivery

Slide 2

Slide 2 text

Have you ever done performance tuning on an application without:

Slide 3

Slide 3 text

Have you ever done performance tuning on an application without: • Benchmarking • Pro fi ling

Slide 4

Slide 4 text

Have you ever done performance tuning on an application without: • Benchmarking • Pro fi ling

Slide 5

Slide 5 text

More computing sins are committed in the name of efficiency (without necessarily achieving it) than for any other single reason - including blind stupidity.” A Case Against the GOTO William Wulf (1972)

Slide 6

Slide 6 text

Slide 7

Slide 7 text

What if we applied the same performance engineering mindset from optimising application performance to optimising developer experience?

Slide 8

Slide 8 text

The Pain

Slide 9

Slide 9 text

Waiting for Builds

Slide 10

Slide 10 text

Troubleshooting failures

Slide 11

Slide 11 text

Flaky Tests

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

Oh and security

Slide 14

Slide 14 text

CI cost Developer frustration Security risk Flaky tests Slow builds 🔒 🤯 💸 🔁 🐌

Slide 15

Slide 15 text

Quick Fix

Slide 16

Slide 16 text

Throw hardware at it!

Slide 17

Slide 17 text

…when does it make sense to throw hardware at a programming problem? As a general rule, I’d say almost always.” http://blog.codinghorror.com/hardware-is-cheap-programmers-are-expensive Jeff Atwood (2008)

Slide 18

Slide 18 text

Did it fix the problem?

Slide 19

Slide 19 text

Was it cost effective?

Slide 20

Slide 20 text

How can you tell?

Slide 21

Slide 21 text

90% But still have queuing issues of CPUs in CI are unused

Slide 22

Slide 22 text

We are way behind where we ought to be as an industry. We are shipping code we don’t understand, to systems we have never understood.” https://charity.wtf/2020/03/03/observability-is-a-many-splendored-thing Charity Majors (2020)

Slide 23

Slide 23 text

No content

Slide 24

Slide 24 text

Slide 25

Slide 25 text

No content

Slide 26

Slide 26 text

Slide 27

Slide 27 text

We’re talking about measuring the “path to production”

Slide 28

Slide 28 text

And we’re talking about observability, not monitoring

Slide 29

Slide 29 text

Monitoring is about known-unknowns and actionable alerts, observability is about unknown-unknowns and empowering you to ask arbitrary new questions and explore where the cookie crumbs take you.” Observability: A Manifesto Charity Majors

Slide 30

Slide 30 text

The Solution

Slide 31

Slide 31 text

If you can not measure it, you can not improve it.” Lord Kelvin (1847)

Slide 32

Slide 32 text

I often say that when you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely, in your thoughts, advanced to the stage of science, whatever the matter may be.” Lord Kelvin (1847)

Slide 33

Slide 33 text

If you can not measure it, you can not improve it.” Lord Kelvin (1847)

Slide 34

Slide 34 text

How do you get your gold star if you can’t prove it worked?

Slide 35

Slide 35 text

Let’s use science!

Slide 36

Slide 36 text

The ability to experiment is a critical part of innovation and creating business value.” Accelerate: The Science of Lean Software and DevOps Nicole Forsgren, Jez Humble, Gene Kim

Slide 37

Slide 37 text

No content

Slide 38

Slide 38 text

How do you measure?

Slide 39

Slide 39 text

No content

Slide 40

Slide 40 text

No content

Slide 41

Slide 41 text

No content

Slide 42

Slide 42 text

No content

Slide 43

Slide 43 text

MCP

Slide 44

Slide 44 text

Your best and wisest refuge from all troubles is in your science.” Ada Lovelace

Slide 45

Slide 45 text

What do you see?

Slide 46

Slide 46 text

“It just takes a long time” • Assumption: “It’s just big.” • Measure: Task-level breakdown shows dependency resolution dominating.

Slide 47

Slide 47 text

30%-40% of all CI build time at large organizations is spent downloading dependencies

Slide 48

Slide 48 text

Add caching

Slide 49

Slide 49 text

“Flaky tests are randomly occurring” • Assumption: “It’s random.” • Measure: • A small number of tests are responsible for most fl akiness • Tests are “random” but failures align with increased load • More fl akiness on certain infrastructure

Slide 50

Slide 50 text

Flaky tests are hard to fix because you have to find the problem

Slide 51

Slide 51 text

“…our analysis revealed that re-running the failing build and attempting to repair the flaky test were the most common actions.” Surveying the Developer Experience of Flaky Tests https://mcminn.info/publications/c72.pdf

Slide 52

Slide 52 text

“…our analysis revealed that re-running the failing build and attempting to repair the flaky test were the most common actions. Our findings also suggested that developers who experience flaky tests more often are more likely to take no action in response to them.” Surveying the Developer Experience of Flaky Tests https://mcminn.info/publications/c72.pdf

Slide 53

Slide 53 text

• Assumption: “Our pipeline is secure.” • Measure: An unveri fi ed dependency download identi fi ed in a build

Slide 54

Slide 54 text

Without obsesrvability you can’t know your pipeline is secure

Slide 55

Slide 55 text

DORA • Shorter builds/tests → faster lead time. • Reliable pipelines → higher deployment frequency. • Catching fl akes/failures → lower change failure rate. • Faster troubleshooting → lower MTTR. • And: complete visibility into builds/tests/dependencies → better security and audit readiness (Pervasive Security)