Tweet
Tweet

Slide 1

Slide 1 text

Finding Metrics that Matter and Using them Safely @everydaykanban | @TeamLagom #devopscon18

Slide 2

Slide 2 text

@everydaykanban | @TeamLagom #devopscon18 Too many metrics?

Slide 3

Slide 3 text

@everydaykanban | @TeamLagom #devopscon18 Too many metrics? When we don’t know what to measure, we take what’s easy / available…

Slide 4

Slide 4 text

There is no one-size-fits-all perfect set of metrics (sorry!) @everydaykanban | @TeamLagom #devopscon18

Slide 5

Slide 5 text

@everydaykanban | @TeamLagom #devopscon18 …though there may be some common, good baseline metrics

Slide 6

Slide 6 text

What you measure shows people what you value, so take care when choosing your metrics @everydaykanban | @TeamLagom #devopscon18

Slide 7

Slide 7 text

Doing Metrics the “right way” @everydaykanban | @TeamLagom #devopscon18 #winning safe ^

Slide 8

Slide 8 text

@everydaykanban | @TeamLagom #devopscon18 Start with a clear outcome in mind

Slide 9

Slide 9 text

Walk the path to discover your metrics @everydaykanban | @TeamLagom #devopscon18

Slide 10

Slide 10 text

Walk the path to discover your metrics @everydaykanban | @TeamLagom Deliver value more frequently #devopscon18

Slide 11

Slide 11 text

Walk the path to discover your metrics @everydaykanban | @TeamLagom Deliver value more frequently Start more or help finish? #devopscon18

Slide 12

Slide 12 text

Walk the path to discover your metrics @everydaykanban | @TeamLagom Deliver value more frequently Start more or help finish? do WiP levels correlates to delivery speed? #devopscon18

Slide 13

Slide 13 text

Walk the path to discover your metrics @everydaykanban | @TeamLagom Deliver value more frequently Start more or help finish? do WiP levels correlates to delivery speed? Team’s current WiP levels AND duration (cycle time) #devopscon18

Slide 14

Slide 14 text

Know what each metric really tells you… @everydaykanban | @TeamLagom #devopscon18 Fooling yourself with vanity metrics everydaykanban I sure do get a lot done! I add so much value!

Slide 15

Slide 15 text

Know what each metric really tells you… @everydaykanban | @TeamLagom #devopscon18 Fooling yourself with vanity metrics everydaykanban I sure do get a lot done! I add so much value! • lines of code written • # of issues closed • # of deploys per day • # of points finished

Slide 16

Slide 16 text

Know what each metric really tells you… @everydaykanban | @TeamLagom #devopscon18 Fooling yourself with vanity metrics everydaykanban I sure do get a lot done! I add so much value! • lines of code written • # of issues closed • # of deploys per day • # of points finished • revenue generated • revenue protected • cost avoided • cost saved

Slide 17

Slide 17 text

Put each metric to the “So What?” test #devopscon18 @everydaykanban | @TeamLagom

Slide 18

Slide 18 text

Put each metric to the “So What?” test #devopscon18 @everydaykanban | @TeamLagom Does it: 1. matter to my customer? 2. help me make a decision or take action?

Slide 19

Slide 19 text

Choose metrics that compete with each other @everydaykanban | @TeamLagom #devopscon18

Slide 20

Slide 20 text

@everydaykanban | @TeamLagom #devopscon18 Quality How good? Responsiveness How fast? Sustainability How repeatable? Productivity How much?

Slide 21

Slide 21 text

Quality: Issues by Application / Service over time @everydaykanban | @TeamLagom #devopscon18 0 25 50 75 100 Sys A Sys B Sys C Sys D Q4 2017 Q1 2018 Q2 2018 Find your problem apps and focus improvement efforts there…

Slide 22

Slide 22 text

Quality: Customer Satisfaction @everydaykanban | @TeamLagom #devopscon18 0 25 50 75 100 Jan Feb Mar Apr May Phone Website Measure it however you can, even if its qualitative…

Slide 23

Slide 23 text

Responsiveness: Duration (Cycle Time) @everydaykanban | @TeamLagom #devopscon18 Can you see different patterns for different types of work?

Slide 24

Slide 24 text

Responsiveness: Mean Time to Recovery (MTTR) @everydaykanban | @TeamLagom #devopscon18 Not all outages are created equal

Slide 25

Slide 25 text

Productivity: Throughput @everydaykanban | @TeamLagom #devopscon18 Gold star if you show how much value the items generate!

Slide 26

Slide 26 text

Productivity: Deployment Frequency @everydaykanban | @TeamLagom #devopscon18 Gold star if you show how much value the deploys generate!

Slide 27

Slide 27 text

Sustainability: Open vs Closed @everydaykanban | @TeamLagom #devopscon18 Are we keeping pace with demand?

Slide 28

Slide 28 text

Sustainability: Team Satisfaction @everydaykanban | @TeamLagom #devopscon18 Would you recommend people to work here?

Slide 29

Slide 29 text

@everydaykanban | @TeamLagom #devopscon18 0 25 50 75 100 Sys A Sys B Sys C Sys D Q4 2017 Q1 2018 Q2 2018 Incidents by System MTTR Throughput Open vs Closed

Slide 30

Slide 30 text

Improvement: Flow Efficiency @everydaykanban | @TeamLagom #devopscon18 The best metric you’re not using! Lead Time Wait Wait Wait Work Work Work http://www.everydaykanban.com/2016/09/25/flow-efficiency/

Slide 31

Slide 31 text

@everydaykanban | @TeamLagom #devopscon18 It’s all 3 ways of DevOps rolled up into one metric! Improvement: Flow Efficiency

Slide 32

Slide 32 text

@everydaykanban | @TeamLagom #devopscon18

Slide 33

Slide 33 text

@everydaykanban | @TeamLagom #devopscon18 Better and with company trend Oops. Still good, but trending adversely Creeping up…

Slide 34

Slide 34 text

You don’t need perfect data, you just need more information than you have right now. If you have none, even just a little information is a big improvement. @everydaykanban | @TeamLagom #devopscon18 Remind yourself…

Slide 35

Slide 35 text

@everydaykanban | @TeamLagom #devopscon18 Avoid these metric missteps

Slide 36

Slide 36 text

#devopscon18 Over-optimizing for a single metric @everydaykanban | @TeamLagom What happens when you take it to extremes?

Slide 37

Slide 37 text

@everydaykanban | @TeamLagom #devopscon18

Slide 38

Slide 38 text

@everydaykanban | @TeamLagom #devopscon18

Slide 39

Slide 39 text

@everydaykanban | @TeamLagom #devopscon18

Slide 40

Slide 40 text

@everydaykanban | @TeamLagom #devopscon18 Created by Julia Wester 2018 More Info on https://lagom.solutions Lagom Discovery Canvas Key Decision Hypothesis Plan / Do Study Adjust Lagom (Just Enough) Too Much (Extreme) Too Little (Extreme) What decision are you trying to make? What steps do you think will get you closer to lagom? What consequences occur when you operate at this extreme… positive and negative! How will you test your hypothesis? How will you know if you are closer to lagom? When will you review? What do your success metrics tell you? Are you closer to lagom, farther away or the same distance? Do you need to tweak the plan & continue or make a new hypothesis? What consequences occur when you operate at this extreme… positive and negative! What are the ideal outcomes when you do this just right? Natural Tendency Check the box in the section below that resembles your normal operating condition

Slide 41

Slide 41 text

@everydaykanban | @TeamLagom #devopscon18 Created by Julia Wester 2018 More Info on https://lagom.solutions Lagom Discovery Canvas Key Decision Hypothesis Plan / Do Study Adjust Lagom (Just Enough) Too Much (Extreme) Too Little (Extreme) What decision are you trying to make? What steps do you think will get you closer to lagom? What consequences occur when you operate at this extreme… positive and negative! How will you test your hypothesis? How will you know if you are closer to lagom? When will you review? What do your success metrics tell you? Are you closer to lagom, farther away or the same distance? Do you need to tweak the plan & continue or make a new hypothesis? What consequences occur when you operate at this extreme… positive and negative! What are the ideal outcomes when you do this just right? Natural Tendency Check the box in the section below that resembles your normal operating condition

Slide 42

Slide 42 text

@everydaykanban | @TeamLagom #devopscon18 Created by Julia Wester 2018 More Info on https://lagom.solutions Lagom Discovery Canvas Key Decision Hypothesis Plan / Do Study Adjust Lagom (Just Enough) Too Much (Extreme) Too Little (Extreme) What decision are you trying to make? What steps do you think will get you closer to lagom? What consequences occur when you operate at this extreme… positive and negative! How will you test your hypothesis? How will you know if you are closer to lagom? When will you review? What do your success metrics tell you? Are you closer to lagom, farther away or the same distance? Do you need to tweak the plan & continue or make a new hypothesis? What consequences occur when you operate at this extreme… positive and negative! What are the ideal outcomes when you do this just right? Natural Tendency Check the box in the section below that resembles your normal operating condition

Slide 43

Slide 43 text

@everydaykanban | @TeamLagom #devopscon18 • Fast delivery • Keeping pace • Team challenged, not overwhelmed • Low # of bugs / High Customer Satisfaction scores • Fast delivery ✔ • Keeping pace ✔ • Bored People • Mediocre quality • Seen as overstaffed • Slow delivery • Not Keeping Pace • Team Burnout • Poor quality • No time for improvement How much work should we have in process at once? ❌ Just measuring these gives a false + measure a balanced set of metrics to see negative consequences

Slide 44

Slide 44 text

Measuring Individuals when you value Team outcomes @everydaykanban | @TeamLagom #devopscon18

Slide 45

Slide 45 text

#devopscon18 Superstars aren’t always the best team members @everydaykanban | @TeamLagom

Slide 46

Slide 46 text

Using shame as an incentive @everydaykanban | @TeamLagom #devopscon18 Courtesy of Troy Magennis

Slide 47

Slide 47 text

“Fear invites wrong figures. Bearers of bad news fare badly. To keep his job, anyone may present to his boss only good news.” –W. Edwards Deming #devopscon18 @everydaykanban | @TeamLagom

Slide 48

Slide 48 text

Blindly pursuing targets @everydaykanban | @TeamLagom #devopscon18 95%

Slide 49

Slide 49 text

–W. Edwards Deming “If you give a manager a numerical target, he’ll make it, even if he has to destroy the organization in the process.” @everydaykanban | @TeamLagom #devopscon18

Slide 50

Slide 50 text

To get ahead of these issues, intentionally explore the dark side of metrics @everydaykanban | @TeamLagom #devopscon18 I’m just misunderstood!

Slide 51

Slide 51 text

@everydaykanban | @TeamLagom #devopscon18 Remember, what you measure shows what you value…

Slide 52

Slide 52 text

–Eli Godratt “Tell me how you’ll measure me and I’ll tell you how I’ll behave. If you measure me in an illogical way, do not complain about illogical behavior.” @everydaykanban | @TeamLagom #devopscon18

Slide 53

Slide 53 text

Know when to discard a metric @everydaykanban | @TeamLagom #devopscon18

Slide 54

Slide 54 text

#devopscon18 How can we make this safer? What is the impact? How can you game it? “How Can I Game It?” Ask your team to play and get a wider perspective @everydaykanban | @TeamLagom

Slide 55

Slide 55 text

@everydaykanban | @TeamLagom #devopscon18

Slide 56

Slide 56 text

@everydaykanban | @TeamLagom #devopscon18 What will you do differently on Monday?

Slide 57

Slide 57 text

Do you have any metric tips to share? Any questions? @everydaykanban | @TeamLagom #devopscon18

Slide 58

Slide 58 text

@everydaykanban | @TeamLagom #devopscon18 [email protected] @everydaykanban & @TeamLagom https://lagom.solutions http://everydaykanban.com https://www.linkedin.com/in/juliawester/ Contact me to talk more