Measuring DevOps: the Key Metrics that Matter

Presented at Agile 2016, Atlanta

How is your DevOps transformation coming along?
How do you measure Agility? Reliability? Efficiency? Quality? Culture? Success?!
How do you optimize your software delivery processes?

You can’t improve what you cannot measure. But are you measuring the right things? Are you measuring too little (or too late), or are you drowning in disparate data points that make it hard for you to get to the bottom line: where should you be focusing on next as you optimize your process?

Having the right goals, asking the right questions and learning by doing are paramount to achieving success with DevOps. Having specific milestones and shared KPIs play a critical role in guiding your DevOps adoption and lead to continuous improvement - towards realizing true agility, improved quality, and faster time to market throughout your organization.

This session will walk you through a practical framework for implementing measurement and tracking of your DevOps efforts and software delivery performance that will provide you with data you can act on!

These KPIs include metrics related to your software delivery pipeline and technical progress, as well as cultural indicators and business impact. In addition, we will cover common use cases and real world examples for implementing these metrics to drive DevOps success, as well as best practices for how to address certain challenges and problematic areas along your process that these metrics may bring to light.

Learning Outcomes:

In this session, you will learn:

1. What are some best practices for identifying the metrics you should be tracking to assess your current state of delivery success, your key bottlenecks and your maturity level on your DevOps journey.
2. How do you track your performance along your software delivery pipeline, as well as gauge your cultural evolution and team dynamics
3. How to identify the key metrics that matter for the different stakeholders and for the different stages throughout your software delivery pipeline
4. What are some examples and best practices for converging specific data points to organizational-level KPIs that are agreed upon by all stakeholders, and provide a barometer to guide your on-going optimization?
5. If your KPIs speak to a possible problem area: what are some common use cases and real world examples for what you should be looking into as you try to alleviate a possible bottleneck or inefficiency.
6. How to automate metrics collection and analysis

Transcript

2. Anders Wallgren Measuring DevOps - Key Metrics that Matter
3. We are here to talk metrics
4. What we hear: Today’s software delivery challenges Every business is a software business • High cost (risk, time) per product release • Manually operated, non-integrated tool chains • Lack of shared visibility across Dev-QA-Ops • No repeatability, predictability • No traceability, auditability • Inefficient infrastructure, low utilization • Non-standard practices
5. DevOps by the numbers 2015 (Super High vs. Low) 2014 (High vs. Low) Deployment Frequency 30x 30x Deployment Lead Time 200x 200x Mean Time to Recover 168x 48x Change Success Rate 60x 3x From: IT Revolution and Puppet Labs’ 2015 State of DevOps
6. DevOps by the numbers From: IT Revolution and Puppet Labs’ 2016 State of DevOps
7. The scientific method Observation is fundamental to the scientific method “You can't improve what you can't measure”
8. Why Metrics?
9. Key principles of measuring software Software’s most important quality is its adaptiveness and ease of change. Efficiency Rate and cost per release Effectiveness Ability to add more value http://electric-cloud.com/wp-content/uploads/DOES15_forum_metrics_102015.pdf
10. Best Practices FOCUS ON OUTCOMES IDENTIFY PREDICTORS OBJECTIVITY SUBJECTIVITY WATCH SIGNAL TO NOISE RATIO
11. Best Practices AVOID METRICS THAT CAN BE GAMED VALUE TEAM PERFORMANCE VALUE TEAM PERFORMANCE SHAMING ENCOURAGES GAMING
12. How HOW Automatic Measure-ability Not only production One pane of glass Unobtrusive Technology
13. How HOW VANITY METRICS Actionable “Customer Sat” Auditable “Tickets closed” Accessible “Uptime” “MTTR”
14. Types of metrics • Internal: inside-out measurements of efficiency -(cost/time/materials) -Tech progress, product pipeline trends and resource utilization -Measures of efficiency and resource consumption • External: outside-in measurements of effectiveness -(value delivered) -Quality, usefulness, performance, and business outcomes -Effectiveness and value delivered • Culture: -(objective and subjective trends in team dynamics) -Process overhead, trustworthiness, shared objectives, morale, motivation, team/product/enterprise identity
15. Pipeline KPIs
16. What to measure in the pipeline Idle time Defects discovered/ escaped, impact of defects MTTD Release frequency Time/cost per release Predictability Deployment lead time Deployment frequency, duration Change success rate MTTR MTTR Cost/frequency of outages On-call after business hours Performance / utilization Development lead time Rework required by defects, build breakage,downtime Idle time Work-in-progress and technical debt Cycle time -- Cycle Time -- -- Visibility -- -- Scale -- DEV/CI QA Deploy Release Operate
17. What to measure in the pipeline Idle time Defects discovered/ escaped, impact of defects MTTD Release frequency Time/cost per release Predictability Deployment lead time Deployment frequency, duration Change success rate MTTR MTTR Cost/frequency of outages On-call after business hours Performance / utilization Development lead time Rework required by defects, build breakage,downtime Idle time Work-in-progress and technical debt Cycle time -- Cycle Time -- -- Visibility -- -- Scale -- QA Deploy Release Operate DEV/CI
18. Dev/CI: Where to Start? • Trunk-based development (or very- short-lived branches) • Self-service automation for environment provisioning • *-as-code • Build quality in (less unplanned work downstream) • Build security in (less unplanned work downstream)
19. What to measure in the pipeline Release frequency Time/cost per release Predictability Deployment lead time Deployment frequency, duration Change success rate MTTR MTTR Cost/frequency of outages On-call after business hours Performance / utilization Development lead time Rework required by defects, build breakage,downtime Idle time Work-in-progress and technical debt Cycle time -- Cycle Time -- -- Visibility -- -- Scale -- Deploy Release Operate DEV/CI Idle time Defects discovered/ escaped, impact of defects MTTD QA
20. QA: Where to Start? • Automated testing • Fidelity of environments vs. prod • Self-service automation for environment provisioning • Continuous Delivery
21. What to measure in the pipeline Development lead time Rework required by defects, build breakage,downtime Idle time Work-in-progress and technical debt Cycle time -- Cycle Time -- -- Visibility -- -- Scale -- Release Operate DEV/CI Idle time Defects discovered/ escaped, impact of defects MTTD QA Deployment lead time Deployment frequency, duration Change success rate MTTR Release frequency Time/cost per release Predictability MTTR Cost/frequency of outages On-call after business hours Performance / utilization Deploy
22. Deploy: Where to Start? • Automate all the things • The first deployment shouldn’t be to PROD…or even PRE-PROD… • Continuous delivery • Improves deployment frequency, reliability • Artifact version control • *-as-code
23. What to measure in the pipeline Development lead time Rework required by defects, build breakage,downtime Idle time Work-in-progress and technical debt Cycle time -- Cycle Time -- -- Visibility -- -- Scale -- Operate DEV/CI Idle time Defects discovered/ escaped, impact of defects MTTD QA Deploy Deployment lead time Deployment frequency, duration Change success rate MTTR MTTR Cost/frequency of outages On-call after business hours Performance / utilization Release frequency Time/cost per release Predictability Release
24. Release: Where to Start? • Model the software delivery pipeline • Ensures reuse, predictability, visibility • Fidelity of everything -- tools, processes, environments • Visibility
25. What to measure in the pipeline Development lead time Rework required by defects, build breakage,downtime Idle time Work-in-progress and technical debt Cycle time -- Cycle Time -- -- Visibility -- -- Scale -- DEV/CI Idle time Defects discovered/ escaped, impact of defects MTTD QA Deploy Deployment lead time Deployment frequency, duration Change success rate MTTR Release frequency Time/cost per release Predictability Release Operate MTTR Cost/frequency of outages On-call after business hours Performance / utilization
26. Operate: Where to Start? • Version control all artifacts (rollback, governance, visibility) • Monitor health of systems and applications • Self-service provisioning of environments • Shared infrastructure to drive down opex/capex
27. What to measure in the pipeline Dev/CI QA Deploy Release Operate Cycle Time -- Visibility -- -- Scale --
29. 1M+ System Integrations/year 10K+ Releases/year 30M Lines of Code 100K Builds/day 480K Code reviews/year 100M Test cases run/day
30. 2 min For Test System Provisioning 100+ Applications $2.5B Online sales, 2014 31->8 People per release (before->after)
31. Gap and Huawei: Two tales of DevOps at scale Developer Build Production Build Regression Test Full Test Feature Delivery Time 10 minutes 300 minute 240 minutes 24 hours 30 days Huawei – Before 1 minute 10 minutes 60 minutes 6 hours 7 days Huawei – After 20 minutes 150 minutes 300 minutes 24 hours 15 days Gap – Before 20 minutes 120 minutes 150 minutes 6 hours 1 day Gap – After
32. What to measure in the pipeline Dev/CI QA Deploy Release Operate Visibility -- Cycle Time -- -- Scale --
33. What to measure in the pipeline Dev/CI QA Deploy Release Operate Scale -- Cycle Time -- -- Visibility -- Doing it like a unicorn!
34. Business KPIs
35. What to measure in the business LEAD TIME STORIES DELIVERED CUSTOMER SATISFACTION ACQUISITION, RETENTION COST
36. Culture KPIs
37. What to measure in the culture • Satisfaction • Retention
38. How to create a generative culture Pathological Bureaucratic Generative Low cooperation Modest cooperation High cooperation Messengers shot Messengers neglected Messengers trained Responsibilities shirked Narrow responsibilities Risk are shared Bridging discouraged Bridging tolerated Bridging encouraged Failure -> scapegoating Failure -> justice Failure -> inquiry Novelty crushed Novelty -> problems Novelty implements A typology of organizational cultures – R. Westrum
39. Top Seven Measures of Culture 1. Organizational investment in DevOps 2. The experience and effectiveness of team leaders 3. Continuous delivery practices 4. Achieving “win-win” outcomes for dev, ops, and infosec teams 5. Organizational performance 6. Deployment pain 7. Lean management practices From: IT Revolution and Puppet Labs’ 2015 State of DevOps
40. Predictors of Strong Performance 1. Peer-reviewed change approval process 2. Version control for all production artifacts 3. Proactive monitoring 4. High-trust organizational culture 5. Win-win relationship between dev and ops From: IT Revolution and Puppet Labs’ 2014 State of DevOps
41. Transformative Benefits 10 min FASTER DEVELOP TO DEPLOY 90 days 99% improvement TIME 10 min FASTER DEVELOP TO DEPLOY 120+ min 12X improvement TIME 6 hours FASTER DEVELOP TO DEPLOY 24 hours 75% improvement TIME minutes FASTER AUDITABILITY who, what, when, how 20 days 90% improvement TIME
42. Resources • http://electric-cloud.com/wp-content/uploads/DOES15_forum_metrics_102015.pdf • https://puppet.com/resources/white-paper/2015-state-of-devops-report • http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1765804/pdf/v013p0ii22.pdf • http://devops.com/2014/11/10/devops-scorecard/ • http://www.datical.com/blog/9-metrics-devops-teams-tracking/ • http://devops.com/2015/01/26/metrics-devops/ • https://blog.appdynamics.com/devops/quantified-devops/ • http://www.slideshare.net/jedi4ever/devops-metrics • http://www.slideshare.net/ITRevolution/does15-troy-magennis-and-julia-wester-metrics-and- modeling-helping-teams-see-how-to-improve
43. Questions? ?
44. Thank You! Anders Wallgren | @anders_wallgren Measuring DevOps - Key Metrics that Matter