Dev and Test Metrics 101 - Agile 2018

Slide 1

Slide 1 text

Development and Test Metrics 101 @catswetel | @everydaykanban

Slide 2

Slide 2 text

@catswetel | @everydaykanban

Slide 3

Slide 3 text

QUALITY RESPONSIVENESS PRODUCTIVITY PREDICTABILITY

Slide 4

Slide 4 text

QUALITY RESPONSIVENESS PRODUCTIVITY PREDICTABILITY Software Development Performance Index (SDPI) Larry Maccherone

Slide 5

Slide 5 text

QUALITY RESPONSIVENESS PRODUCTIVITY PREDICTABILITY

Slide 6

Slide 6 text

QUALITY RESPONSIVENESS PRODUCTIVITY PREDICTABILITY What’s missing?

Slide 7

Slide 7 text

QUALITY RESPONSIVENESS PRODUCTIVITY PREDICTABILITY

Slide 8

Slide 8 text

“The RIGHTER we do the WRONG thing, the WRONGER we become.” -- Dr Russell Ackoff @catswetel | @everydaykanban

Slide 9

Slide 9 text

QUALITY RESPONSIVENESS PRODUCTIVITY PREDICTABILITY QUALITY RESPONSIVENESS @catswetel | @everydaykanban

Slide 10

Slide 10 text

@catswetel | @everydaykanban

Slide 11

Slide 11 text

Time in Process Units of time per unit of work @catswetel | @everydaykanban

Slide 12

Slide 12 text

TIME IN PROCESS DATE DELIVERED TIME IN PROCESS SCATTER PLOT

Slide 13

Slide 13 text

TIME IN PROCESS DATE DELIVERED When can we expect X? 90% 50%

Slide 14

Slide 14 text

TIME IN PROCESS DATE DELIVERED How likely by X? 30d: 77% 21d: 50%

Slide 15

Slide 15 text

TIME IN PROCESS DATE DELIVERED What’s the story?

Slide 16

Slide 16 text

QUALITY RESPONSIVENESS PRODUCTIVITY PREDICTABILITY

Slide 17

Slide 17 text

FREQUENCY TIME IN PROCESS TIME IN PROCESS DISTRIBUTION Groups are easier to see This one happens more often

Slide 18

Slide 18 text

QUALITY RESPONSIVENESS PRODUCTIVITY PREDICTABILITY

Slide 19

Slide 19 text

@catswetel | @everydaykanban

Slide 20

Slide 20 text

WEIBULL DISTRIBUTION @catswetel | @everydaykanban

Slide 21

Slide 21 text

WEIBULL DISTRIBUTION ...maybe @catswetel | @everydaykanban

Slide 22

Slide 22 text

FREQUENCY TIME IN PROCESS NOT Normal

Slide 23

Slide 23 text

FREQUENCY TIME IN PROCESS NOT Normal @catswetel | @everydaykanban

Slide 24

Slide 24 text

TIME IN PROCESS DATE DELIVERED Notice anything else?

Slide 25

Slide 25 text

TIME IN PROCESS DATE DELIVERED Notice anything else?

Slide 26

Slide 26 text

TIME IN PROCESS DATE DELIVERED 90% AVERAGE

Slide 27

Slide 27 text

MULTI MODAL @catswetel | @everydaykanban

Slide 28

Slide 28 text

TIME IN PROCESS DATE DELIVERED

Slide 29

Slide 29 text

QUALITY RESPONSIVENESS PRODUCTIVITY PREDICTABILITY

Slide 30

Slide 30 text

TIME IN PROCESS DATE DELIVERED What else can we learn?

Slide 31

Slide 31 text

TIME IN PROCESS DATE DELIVERED THROUGHPUT Units of work per unit of time

Slide 32

Slide 32 text

TIME IN PROCESS DATE DELIVERED Boxes show variability of Time in Process, NOT Throughput

Slide 33

Slide 33 text

TIME IN PROCESS DATE DELIVERED Does it always make sense to count all the dots?

Slide 34

Slide 34 text

TIME IN PROCESS DATE DELIVERED 9 9 11 6 4 6 4 10 13

Slide 35

Slide 35 text

TIME IN PROCESS DATE DELIVERED 9 9 11 6 4 6 4 10 13 1 2 4 1 0 3 1 1 1

Slide 36

Slide 36 text

QUALITY RESPONSIVENESS PRODUCTIVITY PREDICTABILITY

Slide 37

Slide 37 text

THROUGHPUT

Slide 38

Slide 38 text

85% 50%

Slide 39

Slide 39 text

“We are forced to consider knowledge as something that changes as new evidence is provided by more data, or as new predictions are made from the same data by new theories.” Statistical Method from the Viewpoint of Quality Control by Walter A Shewhart @catswetel | @everydaykanban

Slide 40

Slide 40 text

QUALITY RESPONSIVENESS PRODUCTIVITY PREDICTABILITY

Slide 41

Slide 41 text

Lagging quality @catswetel | @everydaykanban

Slide 42

Slide 42 text

ESCAPED DEFECTS Jan defects found in production Feb Mar Apr May Jun Jul Aug Sep 4 5 8 8 10 6 12 8 11 @catswetel | @everydaykanban type

Slide 43

Slide 43 text

ESCAPED DEFECTS Jan defects found in production Feb Mar Apr May Jun Jul Aug Sep 4 5 8 8 10 6 12 8 11 @catswetel | @everydaykanban type How can we tell if we’re getting better at finding defects earlier?

Slide 44

Slide 44 text

% OF DEFECTS DEFECTS BY ENV percentage Key: Jan Feb Mar Apr May Jun Jul Aug Sep @catswetel | @everydaykanban STAGE PROD DEV INT env

Slide 45

Slide 45 text

QUALITY RESPONSIVENESS PRODUCTIVITY PREDICTABILITY

Slide 46

Slide 46 text

@catswetel | @everydaykanban

Slide 47

Slide 47 text

No content

Slide 48

Slide 48 text

Predicting the movements of the one bee is nearly impossible without the context of the swarm of bees. Complex Adaptive Systems: An Introduction to Computational Models of Social Life by John H. Miller, Scott E. Page

Slide 49

Slide 49 text

@catswetel | @everydaykanban

Slide 50

Slide 50 text

MORE FINISHED THAN STARTED MORE STARTED THAN FINISHED @catswetel | @everydaykanban

Slide 51

Slide 51 text

@catswetel | @everydaykanban

Slide 52

Slide 52 text

@catswetel | @everydaykanban

Slide 53

Slide 53 text

Total WIP @catswetel | @everydaykanban

Slide 54

Slide 54 text

OLDEST NEWEST @catswetel | @everydaykanban

Slide 55

Slide 55 text

Average WIP @catswetel | @everydaykanban

Slide 56

Slide 56 text

@catswetel | @everydaykanban

Slide 57

Slide 57 text

@catswetel | @everydaykanban

Slide 58

Slide 58 text

@catswetel | @everydaykanban

Slide 59

Slide 59 text

LITTLE’S LAW @catswetel | @everydaykanban

Slide 60

Slide 60 text

It’s ok to make necessary trade-offs

Slide 61

Slide 61 text

Where should we focus improvement efforts? @catswetel | @everydaykanban

Slide 62

Slide 62 text

@catswetel | @everydaykanban FLOW EFFICIENCY Work Wait Total Duration

Slide 63

Slide 63 text

...per state (incl. waiting states) @catswetel | @everydaykanban

Slide 64

Slide 64 text

@catswetel | @everydaykanban You can do any chart for any portion of the process!

Slide 65

Slide 65 text

Are we there yet? @catswetel | @everydaykanban

Slide 66

Slide 66 text

QUALITY RESPONSIVENESS PRODUCTIVITY PREDICTABILITY

Slide 67

Slide 67 text

#NOESTIMATES @catswetel | @everydaykanban

Slide 68

Slide 68 text

“Close examination reveals that every meaningful interpretation involves a prediction.” Statistical Method from the Viewpoint of Quality Control Walter A Shewhart @catswetel | @everydaykanban

Slide 69

Slide 69 text

3 Components of Knowledge: 1. Evidence 2. Prediction 3. Degree of Belief @catswetel | @everydaykanban

Slide 70

Slide 70 text

EVIDENCE Agile Quantified (Measuring the impact of Agility) Larry Maccherone Wed 3:45 PREDICTION Forecasting and Estimation Hands-On Ozzie Yuce Thu 10:45 DEGREE OF BELIEF Data Driven Coaching Julia Wester & Cheryl Hammond Thu 9:00 @catswetel | @everydaykanban

Slide 71

Slide 71 text

Julia Wester Co-Founder & Principal Consultant Lagom Solutions [email protected] @everydaykanban Cat Swetel Engineering Manager Ticketmaster [email protected] @catswetel

Slide 72

Slide 72 text

Pablo Picasso “Computers are useless. They can only give you answers.”

Slide 73

Slide 73 text

No content

Slide 74

Slide 74 text

“STATISTICS are useless. They can only give you answers.”

Slide 75

Slide 75 text

No content

Slide 76

Slide 76 text

No content

Slide 77

Slide 77 text

LITTLE’S LAW @catswetel at #qconnyc Are your decisions making this more or less true? FEEL FREE TO CUT

Slide 78

Slide 78 text

Some joke about defining quality

Slide 79

Slide 79 text

VARIABILITY FEEL FREE TO CUT

Slide 80

Slide 80 text

TIME IN PROCESS DATE DELIVERED VARIABILITY FEEL FREE TO CUT

Slide 81

Slide 81 text

QUALITY RESPONSIVENESS PRODUCTIVITY PREDICTABILITY VARIABILITY VARIABILITY VARIABILITY SHEWHART: the cause of v in an expected attribute. Re JUDGEMENT to know which “important quality characte (causes of undesirable varia within the bounds of reason expected economic results. FEEL FREE TO CUT