Avoiding software fails: how to improve application reliability

by Vladislav Samoylenko
DevOps Pro Vilnius 2016

Transcript

1. Confidential, Dynatrace LLC Avoiding software fails Few metrics to improve application reliability Vlad Samoylenko | Senior Sales Engineer vs@bakotech.com
2. Before we start Confidential, Dynatrace LLC
3. Confidential, Dynatrace LLC Get a Dron • Go to http://apmchallenge.eu/ • Enter your email, site and expected time in ms • Get a drone if you win  5 PM somewhere near dynaTrace booth
4. Now it’s time to start Confidential, Dynatrace LLC
5. What to do with the fastest car …
6. … if it fails to (deliver) reach the finish line
7. Failed delivery #1
9. Nobody wants this …
10. Unless you work for Microsoft or Google…
11. Because it often leads to this …
12. The “War Room” Facebook – December 2012
13. And to this …
16. I don’t always Test my code But when I do it I do it in Production
17. Performance issues increase costs 63% of IT organizations spend 20%+ of the time working on performance issues Inability to Innovate 40% of Developers’ time is wasted in triage, stealing a focus from activities that innovates
18. The good thing is 80:20
20. Lets start on the frontend 80/20 rule from Steve
21. But - then we have to focus on the backend
22. 5 Use Cases… & metrics that really pay off
26. Pushing without a Plan
27. Web Site: this shoudn’t happen Some Ad Company during American Super-Bowl Total size ~ 20MB 434 Resources in on that page
28. Web Site: this could be easily eliminated Obama Care 16 individual jQuery -related files that should be merged Most JavaScript files contains Dev documentation, which makes up to 80% of the file size
29. Web Site: this shoudn’t happen Fifa.com doring Woldcup Favicon the Largest element Some heavy CSS & JS +150kb
30. • Developers not using the browser built-in diagnostics tools • Testers not doing a sanity checks with the same tools • Some tools for you • Built-in Inspectors via Ctrl-Shift-I in Chrome and Firefox • YSlow, PageSpeed • Dynatrace Ajax Edition • Level-Up: Automate Testing & Diagnostics Check Lessons Learnt – NO Excuses for …
31. # Resources # of Domains Usage of CDNs Page Load & Size
35. Not every Architect makes good decisions
36. • Symptoms • HTML takes 60-120s to render • High GC Time • Developer Assumptions • Bad GC Tuning • Probably bad DB performance as rendering was simple • Resulted in: months of finger-pointing between Dev & DBA Project: Online Room Reservation System
37. Developers-built monitoring void roomreservationReport(int officeId) { long startTime = System.currentTimeMillis(); Object data = loadDataForOffice(officeId); long dataLoadTime = System.currentTimeMillis() - startTime; generateReport(data, officeId); } Result: Avg. Data Load Time: 41s! DB Tool says: Avg. SQL Query: <1ms!
39. #1: Loading too much data 24889! Calls to the DB API High CPU & High Memory Usage to keep all data in Memory
40. #2: On individual connections 12444! individual connections Individual SQL really fast <1ms Classical N+1 Query Problem
41. #3. Response time 142 sec o_O
43. • …You know what code is doing • Challenge the developers • Explore Tools that “might seem” out of your league! • Built-In Database Analysis Tools • “Logging” options of Frameworks such as Hibernate, … • JMX, Perf Counters, … of your Application Servers • APM (Performance Tracing) Tools: Dynatrace, Ruxit,… Lessons Learned – Don’t Assume …
44. # SQL Executions # of Same SQLs Conn. Acquisition Time
45. Root Cause: Deployment Considerations Log Service provides a Synchronized File across all JVMs 1M Log exceptions over 30 min
46. Production Deployment leads to Log SYNC Issues Log message Time In Sync Two calls comming from Customr coded methods
47. Enabled debug on Production • 70% of response time spent on Log4J
48. Before After
49. Time Spent in Sync & Logging # of Log Messages # of Exceptions
53. Deployment Gone Bad
54. Test Environment Production Environment 8x slower 3x more SQL
55. Test Environment Production Environment That’s Normal: Having I/O for Web Request as main contributor Hibernate, Classloading, XML – The Key Hotspots I/O for Web Requests doesn’t even show up!
56. These calls all originate from thousands of calls to find item by code Top Contributor Class.getInterfaces Called from Hibernates FieldInterceptionHelper
57. Top Methods related to XML Processing Classloading is triggered through CustomMonkey and the Xalan Parser
58. • Plan enough time for proper testing • Only test what really ends up in Production Lessons Learned
59. Time Spent in API # Calls to API
63. Incorrect Sizing of Pools and Queues
64. Online Banking: Slow Balance Check 101s! To Check Balance! 600! SQL Executions 87% spent in IIS
65. #1 Time really spent in IIS? Tip: Elapsed Time tells us WHEN a Method was executed! Tip: Thread# gives us insight on Thread Queues / Switches Finding: Thread 32 in IIS waited 87s to pass control to Thread 30 in ASP.NET
66. #2 What about these SQL Executions? Finding: EVERY SQL statement is executed on ITS OWN Connection! Tip: Look at “GetConnection”
67. #2 SQL Executions! continued … #1: Same SQL is executed 67! times #2: NO PREPARATION because everything executed on new Connection
68. Lessons Learned! ASP.NET Worker Thread Pool Sizing! DB Connection Pools More Efficient SQL
69. Connection Pool Saturated
70. Idle vs. Busy Threads # SQLs / Request # GetConnection
71. # Images # Redirects # and Size of Resources # SQL Executions # of SAME SQLs # Items per Page # AJAX per Page Remember: New Metrics When Testing Apps Time Spent in API # Calls into API # Functional Errors # 3rd Party calls # of Domains Total Size Resource (W3C) Timings: PLT, DOM Processing/Ready, Page Interactive
72. Online Performance Clinics Every week @ bit.ly/onlineperfclinic apmblog.dynatrace.com event.apmchallenge.eu
73. Don’t be afraid you’re the only one here
74. Confidential, Dynatrace LLC Get a Dron • Go to http://apmchallenge.eu/ • Enter your email, site and expected time in MS • Get a drone if you win  In a few minutes somewhere near dynaTrace booth
76. And don’t fail to deliver on Friday :-)
77. Confidential, Dynatrace LLC Performance management for the digital customer age