Key Metrics to Release Better Software Faster

Asad Ali (@AsadThoughts) from Dynatrace shares a number of metrics that can be used to gauge the effectiveness of your software delivery pipeline.

Presented at the June 2016 Boston DevOps meetup.

http://www.meetup.com/Boston-Devops/events/231351549/

Transcript

1. Asad Ali @AsadThoughts Key Metrics to Release Better Software Faster - More on http://blog.dynatrace.com - Dynatrace Free Trial: http://bit.ly/dttrial
2. Waterfall -> Agile: 3 Years 1 Deployment per Month EVERYONE ca do Co ti uous Delive y EVERY MANUAL TE“TER DOE“ AUTOMATION!
5. SO – Why did they do it?
8. Rapidly Changing Markets, Requirements & User Expectations
9. GOAL!
10. Utmost Goal: Minimize Cycle Time feature cycle time time Customer / Market Users
11. Utmost Goal: Minimize Cycle Time feature cycle time time minimize Users
12. Utmost Goal: Minimize Cycle Time feature cycle time time This is when you create value! minimize You
13. ROI?
14. High Performers Are More Agile 30x 8,000x more frequent deployments faster lead times than their peers Source: Puppet Labs 2013 State Of DevOps: http://puppetlabs.com/2013-state-of-devops-infographic
15. High Performers Are More Reliable 2x 12x the change success rate faster mean time to recover (MTTR) Source: Puppet Labs 2013 State Of DevOps: http://puppetlabs.com/2013-state-of-devops-infographic
16. Challenges?
17. Deploy Faster!!
18. Fail Faster!!?
19. Right Focus?!
20. 3 Use Cases on WHY this happened, HOW to avoid it, METRICS to look at
22. #Push without a Plan
23. Mobile Landing Page of Super Bowl Ad 434 Resources in total on that page: 0 JPEGs, 75 PNGs, 50 GIFs, … Total size of ~ 20MB
24. Fifa.com during Worldcup http://apmblog.compuware.com/2014/05/21/is-the-fifa-world-cup-website-ready-for-the-tournament/
25. Page Size Item Size # Objects
26. #No Agile Deployment
28. Load Spike resulted in Unavailability Ad on air
29. Alte ative: GoDaddy goes DevOps 1h before SuperBowl KickOff 1h after Game ended
30. # of Domains Total Size
31. Deploy e t Gone Bad!
32. Test Environment Production Environment 8x slower 3x more SQL
33. Test Environment Production Environment Hibernate, Classloading, XML – The Key Hotspots Hibernate, Classloading, XML – The Key Hotspots I/O for Web Re uests does ’t even show up! That’s No al: Having I/O for Web Request as main contributor
34. Top Contributor Class.getInterfaces Called from Hibernates FieldInterceptionHelper These calls all originate form thousands of calls to find item by code
35. Top Methods related to XML Processing Classloading is triggered through CustomMonnkey and the Xalan Parser Classloading is triggered through CustomMonkey and the Xalan Parser
36. Time Spent in API # Calls to API
37. Too Many Queries crash Undersized Pools
38. Online Banking: Slow Balance Check 1.69m (=101s!) To Check Balance! 87% spent in IIS 600! SQL Executions
39. SQL Executions Finding: EVERY SQL statement is executed on ITS OWN Connection! Tip: Look at GetConnection
40. SQL Executions #1: Same SQL is executed 67! times #2: NO PREPARATION because everything executed on new Connection
41. Lesson Learned ASP.NET Worker Thread Pool Sizing! DB Connection Pools More Efficient SQL
42. Webserver Modules # SQLs / Request # GetConnection
44. •# Resources •Size of Resources •Page Size •# Functional Errors •3rd Party calls •# SQL Executions •# of SAME SQLs Remember: Metrics-based decisions •And there is more … •Time Spent in API •# Calls into API •# of Domains •Total Size •# Items per Page •# AJAX per Page •…
45. How do we get from here …
47. Look behind the scenes
48. Build 17 testNewsAlert OK testSearch OK Build # Use Case Stat # API Calls # SQL Payload CPU 1 5 2kb 70ms 1 3 5kb 120ms Use Case Tests and Monitors Service & App Metrics Build 19 testNewsAlert OK testSearch OK Build 18 testNewsAlert OK testSearch OK 1 4 1kb 60ms 34 171 104kb 550ms Ops #ServInst Usage RT 1 0.5% 7.2s 1 63% 5.2s 1 4 1kb 60ms 2 3 10kb 150ms 1 0.2% 5.2s 5 75% 2.5s Build 25 testNewsAlert - testSearch OK - - - - 2 3 10kb 150ms - - - 8 80% 2.0s Metrics from and for Dev(to)Ops
49. #1: Analyzing every Unit, Integration & REST API test #2: Key Architectural Metrics for each test #3: Detecting regression based on measure per Checkin Stop Bad Builds in CI
50. Quality-Metrics based Build Status
51. Better Software Faster
53. 53 @Dynatrace Asad Ali Dynatrace Product Specialist @AsadThoughts http://blog.dynatrace.com