Things We've Learned about Better Software Delivery Principles through Pandemic

Transcript

1. 1 Masahiko Funaki (舟木将彦) Principal Developer Advocate @ CircleCI Things We’ve Learned about Better Software Delivery Principles through Pandemic Cloud Expo Asia Singapore 2022
2. 2 CircleCI Organization at a Glance Users 1M+ Employees 600+ Founded 2011 Raised $315M CircleCI Overview San Francisco, USA Tokyo, Japan Amsterdam, Netherlands Denver, USA London, UK Paris, France
3. 3 CircleCI helps software teams validate and ship change quickly and confidently. Best-in-class CI/CD. Build Test Deploy Release Code Debug
5. 5 257 mil+ workflows 44,000+ orgs 290,000+ projects 1,000x Larger than surveys Dataset from Automated Build, Test, Deploy and Release Not from manual (impression-based) measurement
6. 6 Lead time to change Recovery from failure time Deployment frequency Change failure rate Four Classic Metrics
7. 7 Suggested Benchmarks APAC (Diff from Global) Throughput The average number of workflow runs per day Merge on any pull request 1.39 (- 2.8%) Duration The average length of time for a workflow to run 10 minutes (Not under 10 minutes) 3.82 minutes (+ 3.2%) Mean time to recovery The average time between failures & their next success Under 1 hour 644 minutes (+ 802%) Success rate The number of successful runs / the total number of runs over a period of time 90% or better on default (master, main) branch 65% (- 15.6%) CI/CD Performance for High Performance Teams
8. 8 Four Key Metrics - 1 Throughput
9. 9 Four Key Metrics - 1 Throughput
10. 10 April
11. 11 Four Key Metrics - 2 Duration
12. 12 Four Key Metrics - 2 Duration
13. 13 Parallelism and Test Split based on Historical Data 1 5 50 5 45 40 100 60 1 0 30 20 100 1 5 50 60 20 1 0 45 40 30 5 100 50 1 0 45 1 5 60 20 5 40 30 Parallel: 1 Parallel: 3 Parallel: 5 1’44” 1’14” 1’09” 1’14” 1’15” 6:20 132 credits (¢7.92) 2’09” 2’09” 2’09” 147 credits (¢8.82) 161credits (¢ 9.66)
14. 14 April
15. 15 Four Key Metrics - 3 Mean time to recovery
16. 16 Four Key Metrics - 3 Mean time to recovery
17. 17 April
18. 18 High-performing Teams & MTTR • Duration is the most important factor to optimizing TTR - optimize it first • Implement tooling for rapid identification and notification of failure • Write tests to include expert error reporting to quickly identify the problem • Debug on the remote machine where failure occurs, or at least rich, robust, verbose log output
19. 19 Four Key Metrics - 4 Success rate
20. 20 Four Key Metrics - 4 Success rate
21. 21 April
22. 22 What should a high-performance team look like?
23. 23 2019 Global (median) 2020 Global (median) 2021 Global 2021 APAC (median) Benchmark Duration The average length of time for a workflow to run 3.38 min 3.96 min 3.7 min 3.82 min 10 minutes TTR The average time between failures & their next success 52.5 55.11 73.6 min 644 min < 60 minutes Success rate The number of successful runs / the total number of runs over a period of time 60% 61% 77% 65% Average should be +90% on default branch Throughput The average number of workflow runs per day 0.80/day 0.70/day 1.43/day 1.39/day As often as your business requires - not a function of your tooling
24. 24 Insights allow teams to use data to improve engineering productivity. Throughput Duration Mean Time to Recovery Success Rate
25. 25 https://circle.ci/ssd2020 2020 Report https://circle.ci/ssd2022 Full 2022 Report
26. Thank you. 26 mfunaki /in/mfunaki For feedback: