Avoiding software fails. Few metrics to improve application reliability by Slawomir Michalik

Transcript

1. Avoiding software fails Few metrics to improve application reliability slawomir.michalik@dynatrace.com
2. 2 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Dynatrace What to do with the fastest car …
3. 3 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Dynatrace … if it fails to (deliver) reach the finish line
4. Failed delivery #1
6. 10 73 / 60 I hope I’ll not fail to deliver
7. Nobody wants this …
8. Unless you work for Microsoft or Google…
9. Because it often leads to this …
10. The “War Room” Facebook – December 2012
11. And to this …
13. How do you feel about this? :-)
15. I don’t always Test my code But when I do it I do it in Production
17. Performance issues increase costs 63% of IT organizations spend 20%+ of the time working on performance issues Inability to Innovate 40% of Developers’ time is wasted in triage, stealing a focus from activities that innovates
18. The good thing is 80:20
20. Lets start on the frontend 80/20 rule from Steve
21. But - then we have to focus on the backend
22. 5 Use Cases… & metrics that really pay off
26. Pushing without a Plan
28. Web Site: this shoudn’t happen Some Ad Company during American Super-Bowl Total size ~ 20MB 434 Resources in on that page
29. Web Site: this could be easily eliminated Obama Care 16 individual jQuery -related files that should be merged Most JavaScript files contains Dev documentation, which makes up to 80% of the file size
30. Web Site: this shoudn’t happen Fifa.com doring Woldcup Favicon the Largest element Some heavy CSS & JS +150kb
31. • Developers not using the browser built-in diagnostics tools • Testers not doing a sanity checks with the same tools • Some tools for you • Built-in Inspectors via Ctrl-Shift-I in Chrome and Firefox • YSlow, PageSpeed • Dynatrace Ajax Edition • Level-Up: Automate Testing & Diagnostics Check Lessons Learnt – NO Excuses for …
32. # Resources # of Domains Usage of CDNs Page Load & Size
36. Not every Architect makes good decisions
37. • Symptoms • HTML takes 60-120s to render • High GC Time • Developer Assumptions • Bad GC Tuning • Probably bad DB performance as rendering was simple • Resulted in: months of finger-pointing between Dev & DBA Project: Online Room Reservation System
38. Developers-built monitoring void roomreservationReport(int officeId) { long startTime = System.currentTimeMillis(); Object data = loadDataForOffice(officeId); long dataLoadTime = System.currentTimeMillis() - startTime; generateReport(data, officeId); } Result: Avg. Data Load Time: 41s! DB Tool says: Avg. SQL Query: <1ms!
40. #1: Loading too much data 24889! Calls to the DB API High CPU & High Memory Usage to keep all data in Memory
41. #2: On individual connections 12444! individual connections Individual SQL really fast <1ms Classical N+1 Query Problem
42. #3: Putting all data in temp Hashtable Lots of time spent in Hashtable.get Called from their Entity Objects
43. • …You know what code is doing • Challenge the developers • Don’t use Hashtabels as workaround, use O/R mappers • Explore Tools that “might seem” out of your league! • Built-In Database Analysis Tools • “Logging” options of Frameworks such as Hibernate, … • JMX, Perf Counters, … of your Application Servers • APM (Performance Tracing) Tools: Dynatrace, Ruxit,… Lessons Learned – Don’t Assume …
44. # SQL Executions # of Same SQLs Conn. Acquisition Time
45. Root Cause: Deployment Considerations Log Service provides a Synchronized File across all JVMs 1M Log exceptions over 30 min
46. Production Deployment leads to Log SYNC Issues Log message Time In Sync Two calls comming from Customr coded methods
47. Time Spent in Sync & Logging # of Log Messages # of Exceptions
51. Deployment Gone Bad
52. Test Environment Production Environment 8x slower 3x more SQL
53. Test Environment Production Environment That’s Normal: Having I/O for Web Request as main contributor Hibernate, Classloading, XML – The Key Hotspots I/O for Web Requests doesn’t even show up!
54. These calls all originate form thousands of calls to find item by code Top Contributor Class.getInterfaces Called from Hibernates FieldInterceptionHelper
55. Top Methods related to XML Processing Classloading is triggered through CustomMonkey and the Xalan Parser
56. • Plan enough time for proper testing • Anticipate changed user behavior during peak load • Only test what really ends up in Production Lessons Learned
57. Time Spent in API # Calls to API
61. Incorrect Sizing of Pools and Queues
62. Online Banking: Slow Balance Check 101s! To Check Balance! 600! SQL Executions 87% spent in IIS
63. #1 Time really spent in IIS? Tip: Elapsed Time tells us WHEN a Method was executed! Tip: Thread# gives us insight on Thread Queues / Switches Finding: Thread 32 in IIS waited 87s to pass control to Thread 30 in ASP.NET
64. #2 What about these SQL Executions? Finding: EVERY SQL statement is executed on ITS OWN Connection! Tip: Look at “GetConnection”
65. #2 SQL Executions! continued … #1: Same SQL is executed 67! times #2: NO PREPARATION because everything executed on new Connection
66. Lessons Learned! ASP.NET Worker Thread Pool Sizing! DB Connection Pools More Efficient SQL
67. Idle vs. Busy Threads # SQLs / Request # GetConnection
69. Do know what you Test
70. Natural (data) context ln ulc oIief do sbemwp itsTiu ea p.leyokefhtu'm rrcereit' euacfttat, ts Ch Theoretically I'm super fast Coherence, but it's difficult to tweak me up.
71. 23s for One click 22s $3-5M worth Data grid New Generation CRM: Angular.js / Coherence
72. New Generation CRM: Angular.js / Coherence 7s for filter execution Filter Value
73. Talk to Architects, and Trace argument’s values 4 performance sensitive methods # of unique invocations Response Time
74. # Images # Redirects # and Size of Resources # SQL Executions # of SAME SQLs # Items per Page # AJAX per Page Remember: New Metrics When Testing Apps Time Spent in API # Calls into API # Functional Errors # 3rd Party calls # of Domains Total Size Resource (W3C) Timings: PLT, DOM Processing/Ready, Page Interactive
75. Online Performance Clinics Every week @ bit.ly/onlineperfclinic apmblog.dynatrace.com event.apmchallenge.eu
76. Putting it into a Test Automation 12 0 120ms 3 1 68ms Build 20 testPurchase OK testSearch OK Build 17 testPurchase OK testSearch OK Build 18 testPurchase FAILED testSearch OK Build 19 testPurchase OK testSearch OK Build # Test Case Status # SQL # Excep CPU 12 0 120ms 3 1 68ms 12 5 60ms 3 1 68ms 75 0 230ms 3 1 68ms Test Framework Results Architectural Data We identified a regression Problem solved Exceptions probably reason for failed tests Problem fixed but now we have an architectural regression Problem fixed but now we have an architectural regression Now we have the functional and architectural confidence Let’s look behind the scenes
78. And don’t fail to deliver on Day #2 :-)