Slide 1

Slide 1 text

Metrics You are what you measure @robdcrowley robdcrowley

Slide 2

Slide 2 text

Empower you with new ideas to bring your team together! Metrics. What are they? What to measure? How to visualise? Tooling. Some secret sauce to make our lives easier Metrics as a catalyst for cultural change Goals for Session

Slide 3

Slide 3 text

Metrics 101 What metrics are and why we measure

Slide 4

Slide 4 text

To inform To compare To motivate To understand To improve To protect Not: Just because successful companies do! Why do we measure

Slide 5

Slide 5 text

A quantifiable measure of any component or process used to gauge the performance of your business – Application – Business – People – Process – System What is a metric

Slide 6

Slide 6 text

Basis of measurement Assumptions Level and usage Desired trend When to use When to stop How to game Warnings Facets of a metric

Slide 7

Slide 7 text

Red is good, Green is worthless

Slide 8

Slide 8 text

Metrics 201 What should you measure

Slide 9

Slide 9 text

“Lack of direction, not lack of time, is the problem. We all have twenty-four hour days.” - Zig Zigler

Slide 10

Slide 10 text

Pivot: Base conversations on metrics instead of non actionable opinions Use stakeholder input as the basis for your initial set of metrics – “What do you want out of this?” – “How quickly do you want this?” Metrics can help guide us

Slide 11

Slide 11 text

Awesome book, poor title Core Principles – Minimum Viable Product (MVP) – Continuous Deployment – Split Testing (aka A/B Testing) – Actionable Metrics – Pivot – Innovation Accounting – Build-Measure-Learn Loop

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

Acquisition – finding new users Activation – getting users to give your product a try Retention – making sure those users stick around Referral – have your loyal users invite others Revenue – making some money from all this Note: These steps are not strictly sequential Pirate Funnel (AARRR!)

Slide 14

Slide 14 text

Do not focus solely on metrics that focus on reducing errors Focus on achieving excellence and positive outcomes will ensue

Slide 15

Slide 15 text

DevOps State of the Union focuses on these four core metrics – Deployment Frequency – Lead Time for Changes – Mean Time to Recover (MTTR) – Failed Change Rate Example Metrics

Slide 16

Slide 16 text

Fact: We only realise business value from the work we do when it’s released into production The bottleneck or constraint is the limiting factor on our ability to deliver more quickly Strengthening any link other than the weakest is a waste of time and effort Use metrics to identify current constraint Theory of Constraints

Slide 17

Slide 17 text

Metrics 301 How to visualise and share metrics

Slide 18

Slide 18 text

Display all metrics on a dashboard – Business (i.e. Application dropouts) – Dev (i.e. Performance metrics) – Ops (i.e. Web Server CPU Usage) Single URL for all data Make it easy for colleagues to access dashboard Visualisation

Slide 19

Slide 19 text

Use a tool that can handle different kinds of metrics A few standouts are: – StatsD/Graphite/Grafana – Heka – Logstash/Elastic Seach/Kibana Tools

Slide 20

Slide 20 text

StatsD Whisper Carbon Graphite Dashboard Apps UDP TCP HTTP Graphite and StatsD

Slide 21

Slide 21 text

Metrics 401 Potential pitfalls when using metrics

Slide 22

Slide 22 text

Case Study: US Health Service In the 1990s the US Health Service decided to make hospital mortality rates public. What better metric for hospital performance? Observed Behaviour: Best way to improve rating was not to admit critically ill patients in the first place You are what you measure

Slide 23

Slide 23 text

“The actual company values, as opposed to the nice-sounding values, are shown by who gets rewarded, promoted or let go” - Patty McCord

Slide 24

Slide 24 text

Case Study: Enron Enron which went bankrupt from fraud and whose leaders went to jail, had the following company values: Integrity, Communication, Respect, Excellence Where these really the core values at Enron? You are what you measure

Slide 25

Slide 25 text

“In the absence of clarity around an objective, any measure will do” - Bob Paladino Too Many Metrics

Slide 26

Slide 26 text

Depict rosiest picture possible but do not accurately reflect the key drivers of your business – Inaccessible – Not auditable – Not actionable Vanity Metrics

Slide 27

Slide 27 text

Metrics based on output rather than outcomes Aim to maximise outcomes and minimise output – Total lines of code – Number of bugs fixed – Total hours worked – Lines of code per developer Useless Metrics

Slide 28

Slide 28 text

“We’re not doing this data analysis thing just because. We’re gathering data points because we’ve seen it work to great effect in other organisations and industries.” - Anonymous Cargo Cult Metrics

Slide 29

Slide 29 text

Metrics being used for evil rather than for good Confronting this organisational anti-pattern can be a good first step Information sharing is the best defense Break down knowledge silos in the organisation Weaponized Metrics

Slide 30

Slide 30 text

Questions