Upgrade to Pro — share decks privately, control downloads, hide ads and more …

IT Metrics in Real Life

IT Metrics in Real Life

It's all about measuring, and that's just the easy part. The hard part is focusing on what really matters.
I'll show you a bunch of metrics at different levels and from different perspectives, trying to make some sense out of them. I'll try to expand the current vision of what could be a metric, how it affects you and how you can choose your next ones.
The main goal is being able to understand what you need to monitor and improve in order to get better at what you're doing.


Fabio Cicerchia

March 26, 2019



  2. Hello! I AM FABIO CICERCHIA Chief Technology Officer @ Skuola.net

    You can find me at: @fabiocicerchia
  3. Let’s Clarify ‘em First

  4. What’s a Metric? Metrics are parameters or measures of quantitative

    assessment used for measurement, comparison, or to track performance or production. https://www.investopedia.com/terms/m/metrics.asp
  5. Are they useful? • Provide Transparency • Highlight Improvements, Regressions

    or Mistakes • Lead to Self Improvement • Drive Decisions • Reduce Costs • Allow you to be Objective
  6. What’s a KPI? A KPI is a measurable value that

    demonstrates how effectively a company is achieving key business objectives. Organisations use KPIs at multiple levels to track performance measures. https://www.klipfolio.com/resources/articles/what-is-a-key-performance-indicator
  7. Are they useful? • I’ll leave it to you

  8. Recommendation #1 Measure what matters while it matters https://www.oreilly.com/ideas/ask-the-cto-measuring-team-performance

  9. Recommendation #2 The focus should be on the goal not

    the metric
  10. Recommendation #3 Don't use raw metrics as performance evaluation

  11. An Example By The Book

  12. McCalls’ Software Quality Factors https://www.researchgate.net/profile/Caroline_Jay/publication/264239621/figure/fig1/AS:614318999625766@1523476471560/McCalls-Software-Quality-Factors.png

  13. - Auditability – The ease with which conformance to standards

    can be checked. - Accuracy – The precision of computation and control. - Communication Commonality – The degree to which standard interfaces, protocols and bandwidth are used. - Completeness – The degree to which full implementation of required function has been achieved. - Complexity – The degree to which the program is complex. - Conciseness – The compactness of the program in terms of lines of codes. - Consistency – The use of uniform design and documentation technique throughout the software development project. - Data Commonality – The use of standard data structures and types throughout the program. McCalls’ Software Quality Factors http://education.dewsoftoverseas.com/QE/QUickReference/Software%20Enginering/2.6.asp
  14. - Error Tolerance – The damage that occurs when the

    program encounters an error - Execution Efficiency – The run time performance of a program. - Expandability – The degree to which architectural, data or procedural data can be extended. - Generality – The breadth of potential application of program components. - Hardware Independence – The degree to which the software is decoupled from the hardware on which it operates. - Instrumentation – The degree to which the program monitors it’s own operation and identifies errors that do occur. - Modularity – The functional independence of program components. McCalls’ Software Quality Factors (cont’d) http://education.dewsoftoverseas.com/QE/QUickReference/Software%20Enginering/2.6.asp
  15. McCalls’ Software Quality Factors (cont’d) - Operability – The ease

    of operation of a program. - Security – The availability of mechanisms that control or protect programs and data. - Self-Documentation – The degree to which the source code provides meaningful documentation. - Simplicity – The degree to which a program can be understood without difficulty. - Software System Independence – The degree to which the program is independent of nonstandard programming language features, operating system characteristics, and other environmental constraints. - Traceability – The ability to trace a design representation or actual program component back to requirements. - Training – The degree to which the software assists in enabling new users to apply the system. http://education.dewsoftoverseas.com/QE/QUickReference/Software%20Enginering/2.6.asp
  16. Metrics by Perspectives

  17. Dev’s Perspective

  18. LOC - Lines Of Code - Commented Lines - Logic

    Lines - New Lines vs Deleted Lines - Impact (new block vs lots of small changes) - Bugs Rate
  19. Commit Rate

  20. Cyclomatic Complexity https://www.researchgate.net/profile/Vladimir_Riabov/publication/260038796/figure/fig2/AS:297180960182275@1447864870689/Three-methods-of-evaluating-the-cyclomatic-complexity-of-the-graph.png

  21. C.R.A.P Index (Change Risk Analysis and Prediction) https://i.stack.imgur.com/x3XAY.png

  22. Code Duplication % https://i.pinimg.com/originals/8a/39/28/8a3928c74f0e472624a48b741f8a21bc.png

  23. Code Coverage % https://i2.wp.com/blog.mallow-tech.com/wp-content/uploads/2018/05/code_coverage_example.png

  24. Broken Tests % - Marked as Incomplete - Marked as

    Skipped - Risky - Warnings
  25. % New Tests - Per Sprint - Per Project

  26. Coding Style Violations https://www.integer-net.com/wp-content/uploads/sites/3/2018/07/grumphp_no.png

  27. SonarQube Scores

  28. Technical Debt https://blog.insight.symfony.com/img/screenshots/technical_debt_php.png

  29. # 3rd Party Vendors http://knowledge.wharton.upenn.edu/wp-content/uploads/2016/12/legos.jpg

  30. TTFC (Time To First Commit) • How well-organised is the

    code base? • How supportive is the team? • How much confidence do we have in our tests? • How mature is the CI pipeline? • How is the dev environment?
  31. # Pending PR

  32. Cache Hit Ratio https://support.stackpath.com/hc/article_attachments/360002767066/Cache_Hit_Ratio.PNG

  33. DevOp’s Perspective

  34. Broken Builds https://d3r49iyjzglexf.cloudfront.net/blog/content/insights-1.0-large@2x-f9d4748edfb82ee810b499a39e0671bda4c9e99b849bde044e4ccdee2c5b8e08.gif

  35. Build Time https://www.automatetheplanet.com/wp-content/uploads/2018/04/16-hours-bellatrix-build.png

  36. Deploys of Unreviewed PR https://media-cdn.tripadvisor.com/media/photo-s/0e/21/22/31/the-three-monkeys-cafe.jpg

  37. Deploys of Partially Fixed PR https://media-cdn.tripadvisor.com/media/photo-s/0e/21/22/31/the-three-monkeys-cafe.jpg

  38. Deployment Frequency https://blog.newrelic.com/wp-content/uploads/EOYBlog_Graph_Opt1.jpg

  39. Deployment Time

  40. Change Fail % - Rollback - Manual Patches

  41. # Application Errors https://blog.sentry.io/img/post-images/epsagon/monitored-backend.png

  42. Sysadmin’s Perspective

  43. Uptime https://www.pingdom.com/wp-content/themes/pingdom-2018/public/assets/images/uptime-monitoring/uptime-specific-still.png

  44. MTTR (Mean Time To Recover) https://archive.sap.com/image/456241

  45. MTBF (Mean Time Between Failures) https://archive.sap.com/image/456241

  46. Size Error Logs https://preview.ibb.co/eaygbH/Capture_d_e_cran_2018_02_08_11_16_54.png

  47. Free Disk Space % https://i.stack.imgur.com/cRr4H.jpg

  48. Disk I/O https://haydenjames.io/linux-server-performance-disk-io-slowing-application/

  49. Bandwidth Saturation %

  50. # Infrastructure Alerts - Disaster - High - Average -

    All the others
  51. Customer Support’s Perspective

  52. Unsolved Issues/Tickets

  53. % Returning Problems https://marketplace-cdn.atlassian.com/files/images/f16e86e7-c5ec-41ae-8c0a-8b0cbec978d4.png

  54. FRT (First Response Time) https://www.geckoboard.com/assets/1-first-response-time-visualizations-customer-support-metric.png

  55. Defect Escape Rate https://image.slidesharecdn.com/webcastpresentation-6-140822160616-phpapp01/95/webcast-presentation-accelerate-continuous-delivery-with-development-testing-and-rational-team-concert-6-638.j pg

  56. Project Manager’s Perspective

  57. Lead Time https://image.slidesharecdn.com/leanleadershipwebinar-part3of3slides-161029141925/95/lean-leadership-part-3-of-3-30-638.jpg

  58. Process Time https://image.slidesharecdn.com/leanleadershipwebinar-part3of3slides-161029141925/95/lean-leadership-part-3-of-3-30-638.jpg

  59. Total vs Open https://www.glostone.com/wp-content/uploads/2016/10/piles-of-paperwork.png

  60. Closed Issues/Tickets - Per Person - Per Project - Per

    Team - Per Urgency
  61. WIP Issues/Tickets - Per Person - Per Project - Per

    Team - Per Urgency
  62. Estimates Accuracy https://manifesto.co.uk/wp-content/uploads/2014/08/estimating-in-agile-and-scrum.png

  63. Velocity http://www.softwaretestingstudio.com/wp-content/uploads/2017/05/Initial-Agile-Velocity.png

  64. Burndown Variance https://upload.wikimedia.org/wikipedia/commons/0/05/SampleBurndownChart.png

  65. APF (Augmented Pony Factor) The lowest number of committers whose

    total contribution constitutes the majority of the codebase https://ke4qqq.wordpress.com/2015/02/08/pony-factor-math/
  66. HR’s Perspective

  67. Tech Test Passing Rate https://i.kinja-img.com/gawker-media/image/upload/s--EWLoqHzx--/c_scale,fl_progressive,q_80,w_800/18kz3r7mcex5bgif.jpg

  68. Team Morale https://cdn-images-1.medium.com/max/1600/1*3LM47GMbaG9CvSCIHjjPYQ.png

  69. Manager’s Perspective

  70. PSV (Project Schedule Variance) http://agilenucleus.com/wp-content/uploads/2015/04/evmchart.png

  71. PCV (Project Cost Variance) http://agilenucleus.com/wp-content/uploads/2015/04/evmchart.png

  72. EV (Earned Value) https://www.pmi.org/kasimage/f58c4893-c369-409a-acf3-227ce7f0074e/p_2.jpg

  73. Overall Budget - Infrastructure (Cloud / Bare Metal) - Equipment

    (PC, Devices, Dongles, …) - Software (Licences, …) - SaaS - Personnel - Consulting - Outsourcing - Training (Workshops, Conferences, Books, …) - Recreational (Pastries, Drinks, Snacks, …)
  74. EOL Running Stuff https://zdnet4.cbsistatic.com/hub/i/2018/10/14/7b07994d-008a-4f9b-82f4-01af045f1646/7f62145966cfb80731a8e9676cd59c13/php-eols.png https://symfony.com/doc/4.0/_images/release-process.jpg https://www.dev-metal.com/wp-content/uploads/2014/12/eol-centos.png

  75. New Tech Deployed - ThoughtWorks Radar - Zalando Radar -

    Codemotion Community Radar (WIP)
  76. Employee Turnover Rate https://www.officevibe.com/blog/how-to-calculate-employee-turnover

  77. Bus Factor https://cdn-images-1.medium.com/max/1200/0*A7I4FMXuGkcmJ95A.

  78. How Can You Get The Data?

  79. - Custom Metrics - APM (Application Performance Monitoring) - RUM

    (Real User Monitoring) Application Telemetry
  80. Internal Tools - GitHub/GitLab/BitBucket - Sentry - SonarQube - SiteSpeed.io

    - StatsD - Ticket System - Zabbix/Nagios/Cacti/New Relic/… - ...
  81. What Else?!

  82. Graphs

  83. Trends

  84. Forecasts

  85. Aggregates

  86. KPIs

  87. Pick the Metrics that have the Most Sense For You

  88. Now go, and do your homeworks

  89. In-House Tools

  90. Scripting

  91. Excel

  92. InfluxDB & Grafana

  93. ELK

  94. References • https://blog.usenotion.com/13-essential-software-development-metrics-to-ensure-quality-219cfc264ed1 • https://www.frontrowagile.com/blog/posts/69-30-metrics-for-agile-software-development-teams • https://ardalis.com/static-code-analysis-and-quality-metrics • https://github.com/mre/awesome-static-analysis •

    https://www.slideshare.net/JerryTan2/metrics-driven-dev-ops-2017 • https://www.slideshare.net/mattiabattiston/kanban-metrics-in-practice-for-leading-continuous-improvement • https://github.com/chaoss/metrics • http://shop.oreilly.com/product/0636920020134.do
  95. Thanks! ANY QUESTIONS? You can find me here: https://fabiocicerchia.it