IT Metrics in Real Life

It's all about measuring, and that's just the easy part. The hard part is focusing on what really matters.
I'll show you a bunch of metrics at different levels and from different perspectives, trying to make some sense out of them. I'll try to expand the current vision of what could be a metric, how it affects you and how you can choose your next ones.
The main goal is being able to understand what you need to monitor and improve in order to get better at what you're doing.

Transcript

1. IT Metrics in Real Life IMPROVING YOURSELF BY MEASURING RIGHT
2. Hello! I AM FABIO CICERCHIA Chief Technology Officer @ Skuola.net You can find me at: @fabiocicerchia
3. Let’s Clarify ‘em First
4. What’s a Metric? Metrics are parameters or measures of quantitative assessment used for measurement, comparison, or to track performance or production. https://www.investopedia.com/terms/m/metrics.asp
5. Are they useful? • Provide Transparency • Highlight Improvements, Regressions or Mistakes • Lead to Self Improvement • Drive Decisions • Reduce Costs • Allow you to be Objective
6. What’s a KPI? A KPI is a measurable value that demonstrates how effectively a company is achieving key business objectives. Organisations use KPIs at multiple levels to track performance measures. https://www.klipfolio.com/resources/articles/what-is-a-key-performance-indicator
7. Are they useful? • I’ll leave it to you
8. Recommendation #1 Measure what matters while it matters https://www.oreilly.com/ideas/ask-the-cto-measuring-team-performance
9. Recommendation #2 The focus should be on the goal not the metric
10. Recommendation #3 Don't use raw metrics as performance evaluation
11. An Example By The Book
12. McCalls’ Software Quality Factors https://www.researchgate.net/profile/Caroline_Jay/publication/264239621/figure/fig1/AS:614318999625766@1523476471560/McCalls-Software-Quality-Factors.png
13. - Auditability – The ease with which conformance to standards can be checked. - Accuracy – The precision of computation and control. - Communication Commonality – The degree to which standard interfaces, protocols and bandwidth are used. - Completeness – The degree to which full implementation of required function has been achieved. - Complexity – The degree to which the program is complex. - Conciseness – The compactness of the program in terms of lines of codes. - Consistency – The use of uniform design and documentation technique throughout the software development project. - Data Commonality – The use of standard data structures and types throughout the program. McCalls’ Software Quality Factors http://education.dewsoftoverseas.com/QE/QUickReference/Software%20Enginering/2.6.asp
14. - Error Tolerance – The damage that occurs when the program encounters an error - Execution Efficiency – The run time performance of a program. - Expandability – The degree to which architectural, data or procedural data can be extended. - Generality – The breadth of potential application of program components. - Hardware Independence – The degree to which the software is decoupled from the hardware on which it operates. - Instrumentation – The degree to which the program monitors it’s own operation and identifies errors that do occur. - Modularity – The functional independence of program components. McCalls’ Software Quality Factors (cont’d) http://education.dewsoftoverseas.com/QE/QUickReference/Software%20Enginering/2.6.asp
15. McCalls’ Software Quality Factors (cont’d) - Operability – The ease of operation of a program. - Security – The availability of mechanisms that control or protect programs and data. - Self-Documentation – The degree to which the source code provides meaningful documentation. - Simplicity – The degree to which a program can be understood without difficulty. - Software System Independence – The degree to which the program is independent of nonstandard programming language features, operating system characteristics, and other environmental constraints. - Traceability – The ability to trace a design representation or actual program component back to requirements. - Training – The degree to which the software assists in enabling new users to apply the system. http://education.dewsoftoverseas.com/QE/QUickReference/Software%20Enginering/2.6.asp
16. Metrics by Perspectives
17. Dev’s Perspective
18. LOC - Lines Of Code - Commented Lines - Logic Lines - New Lines vs Deleted Lines - Impact (new block vs lots of small changes) - Bugs Rate
19. Commit Rate
20. Cyclomatic Complexity https://www.researchgate.net/profile/Vladimir_Riabov/publication/260038796/figure/fig2/AS:297180960182275@1447864870689/Three-methods-of-evaluating-the-cyclomatic-complexity-of-the-graph.png
21. C.R.A.P Index (Change Risk Analysis and Prediction) https://i.stack.imgur.com/x3XAY.png
22. Code Duplication % https://i.pinimg.com/originals/8a/39/28/8a3928c74f0e472624a48b741f8a21bc.png
23. Code Coverage % https://i2.wp.com/blog.mallow-tech.com/wp-content/uploads/2018/05/code_coverage_example.png
24. Broken Tests % - Marked as Incomplete - Marked as Skipped - Risky - Warnings
25. % New Tests - Per Sprint - Per Project
26. Coding Style Violations https://www.integer-net.com/wp-content/uploads/sites/3/2018/07/grumphp_no.png
27. SonarQube Scores
28. Technical Debt https://blog.insight.symfony.com/img/screenshots/technical_debt_php.png
29. # 3rd Party Vendors http://knowledge.wharton.upenn.edu/wp-content/uploads/2016/12/legos.jpg
30. TTFC (Time To First Commit) • How well-organised is the code base? • How supportive is the team? • How much confidence do we have in our tests? • How mature is the CI pipeline? • How is the dev environment?
31. # Pending PR
32. Cache Hit Ratio https://support.stackpath.com/hc/article_attachments/360002767066/Cache_Hit_Ratio.PNG
33. DevOp’s Perspective
34. Broken Builds https://d3r49iyjzglexf.cloudfront.net/blog/content/insights-1.0-large@2x-f9d4748edfb82ee810b499a39e0671bda4c9e99b849bde044e4ccdee2c5b8e08.gif
35. Build Time https://www.automatetheplanet.com/wp-content/uploads/2018/04/16-hours-bellatrix-build.png
36. Deploys of Unreviewed PR https://media-cdn.tripadvisor.com/media/photo-s/0e/21/22/31/the-three-monkeys-cafe.jpg
37. Deploys of Partially Fixed PR https://media-cdn.tripadvisor.com/media/photo-s/0e/21/22/31/the-three-monkeys-cafe.jpg
38. Deployment Frequency https://blog.newrelic.com/wp-content/uploads/EOYBlog_Graph_Opt1.jpg
39. Deployment Time
40. Change Fail % - Rollback - Manual Patches
41. # Application Errors https://blog.sentry.io/img/post-images/epsagon/monitored-backend.png
42. Sysadmin’s Perspective
43. Uptime https://www.pingdom.com/wp-content/themes/pingdom-2018/public/assets/images/uptime-monitoring/uptime-specific-still.png
44. MTTR (Mean Time To Recover) https://archive.sap.com/image/456241
45. MTBF (Mean Time Between Failures) https://archive.sap.com/image/456241
46. Size Error Logs https://preview.ibb.co/eaygbH/Capture_d_e_cran_2018_02_08_11_16_54.png
47. Free Disk Space % https://i.stack.imgur.com/cRr4H.jpg
48. Disk I/O https://haydenjames.io/linux-server-performance-disk-io-slowing-application/
49. Bandwidth Saturation %
50. # Infrastructure Alerts - Disaster - High - Average - All the others
51. Customer Support’s Perspective
52. Unsolved Issues/Tickets
53. % Returning Problems https://marketplace-cdn.atlassian.com/files/images/f16e86e7-c5ec-41ae-8c0a-8b0cbec978d4.png
54. FRT (First Response Time) https://www.geckoboard.com/assets/1-first-response-time-visualizations-customer-support-metric.png
55. Defect Escape Rate https://image.slidesharecdn.com/webcastpresentation-6-140822160616-phpapp01/95/webcast-presentation-accelerate-continuous-delivery-with-development-testing-and-rational-team-concert-6-638.j pg
56. Project Manager’s Perspective
57. Lead Time https://image.slidesharecdn.com/leanleadershipwebinar-part3of3slides-161029141925/95/lean-leadership-part-3-of-3-30-638.jpg
58. Process Time https://image.slidesharecdn.com/leanleadershipwebinar-part3of3slides-161029141925/95/lean-leadership-part-3-of-3-30-638.jpg
59. Total vs Open https://www.glostone.com/wp-content/uploads/2016/10/piles-of-paperwork.png
60. Closed Issues/Tickets - Per Person - Per Project - Per Team - Per Urgency
61. WIP Issues/Tickets - Per Person - Per Project - Per Team - Per Urgency
62. Estimates Accuracy https://manifesto.co.uk/wp-content/uploads/2014/08/estimating-in-agile-and-scrum.png
63. Velocity http://www.softwaretestingstudio.com/wp-content/uploads/2017/05/Initial-Agile-Velocity.png
64. Burndown Variance https://upload.wikimedia.org/wikipedia/commons/0/05/SampleBurndownChart.png
65. APF (Augmented Pony Factor) The lowest number of committers whose total contribution constitutes the majority of the codebase https://ke4qqq.wordpress.com/2015/02/08/pony-factor-math/
66. HR’s Perspective
67. Tech Test Passing Rate https://i.kinja-img.com/gawker-media/image/upload/s--EWLoqHzx--/c_scale,fl_progressive,q_80,w_800/18kz3r7mcex5bgif.jpg
68. Team Morale https://cdn-images-1.medium.com/max/1600/1*3LM47GMbaG9CvSCIHjjPYQ.png
69. Manager’s Perspective
70. PSV (Project Schedule Variance) http://agilenucleus.com/wp-content/uploads/2015/04/evmchart.png
71. PCV (Project Cost Variance) http://agilenucleus.com/wp-content/uploads/2015/04/evmchart.png
72. EV (Earned Value) https://www.pmi.org/kasimage/f58c4893-c369-409a-acf3-227ce7f0074e/p_2.jpg
73. Overall Budget - Infrastructure (Cloud / Bare Metal) - Equipment (PC, Devices, Dongles, …) - Software (Licences, …) - SaaS - Personnel - Consulting - Outsourcing - Training (Workshops, Conferences, Books, …) - Recreational (Pastries, Drinks, Snacks, …)
74. EOL Running Stuff https://zdnet4.cbsistatic.com/hub/i/2018/10/14/7b07994d-008a-4f9b-82f4-01af045f1646/7f62145966cfb80731a8e9676cd59c13/php-eols.png https://symfony.com/doc/4.0/_images/release-process.jpg https://www.dev-metal.com/wp-content/uploads/2014/12/eol-centos.png
75. New Tech Deployed - ThoughtWorks Radar - Zalando Radar - Codemotion Community Radar (WIP)
76. Employee Turnover Rate https://www.officevibe.com/blog/how-to-calculate-employee-turnover
77. Bus Factor https://cdn-images-1.medium.com/max/1200/0*A7I4FMXuGkcmJ95A.
78. How Can You Get The Data?
79. - Custom Metrics - APM (Application Performance Monitoring) - RUM (Real User Monitoring) Application Telemetry
80. Internal Tools - GitHub/GitLab/BitBucket - Sentry - SonarQube - SiteSpeed.io - StatsD - Ticket System - Zabbix/Nagios/Cacti/New Relic/… - ...
81. What Else?!
82. Graphs
83. Trends
84. Forecasts
85. Aggregates
86. KPIs
87. Pick the Metrics that have the Most Sense For You
88. Now go, and do your homeworks
89. In-House Tools
90. Scripting
91. Excel
92. InfluxDB & Grafana
93. ELK
94. References • https://blog.usenotion.com/13-essential-software-development-metrics-to-ensure-quality-219cfc264ed1 • https://www.frontrowagile.com/blog/posts/69-30-metrics-for-agile-software-development-teams • https://ardalis.com/static-code-analysis-and-quality-metrics • https://github.com/mre/awesome-static-analysis • https://www.slideshare.net/JerryTan2/metrics-driven-dev-ops-2017 • https://www.slideshare.net/mattiabattiston/kanban-metrics-in-practice-for-leading-continuous-improvement • https://github.com/chaoss/metrics • http://shop.oreilly.com/product/0636920020134.do
95. Thanks! ANY QUESTIONS? You can find me here: https://fabiocicerchia.it