The story of a performance outage

Talk from DevOpsDays Wellington 2016. Covering a performance issue & how it could have been prevented in dev or test.

Transcript

1. The story of a performance outage And how we could have prevented it (before everything broke) © Datacom 2016
2. Who am I? © Datacom 2016
3. Who am I? © Datacom 2016 • Bridget Cowie • Application Performance Consultant at Datacom Wellington • @BridgetKCowie
4. 1023% © Datacom 2016
5. Tip #1 Friday night phone calls from your boss are never good © Datacom 2016
6. 0 –> 100 in 4 hours © Datacom 2016
7. “We know it’s a memory leak” © Datacom 2016
8. Tip #2 If someone tells you they know exactly what’s wrong but they can’t give you any details or fix it – be very suspicious. © Datacom 2016
9. Some light Sunday reading… © Datacom 2016
10. Things start to get interesting © Datacom 2016
11. © Datacom 2016
12. Technology Stack © Datacom 2016
13. Technology Stack © Datacom 2016 • Jboss JVM’s • Mulesoft ESB • Active – MQ • HornetQ • Oracle
14. © Datacom 2016
15. Ah ha moment © Datacom 2016
16. © Datacom 2016 2.3 million DB queries per half hour
17. Tip #3 Typically 80% of all issues can be detected in dev or test if you look for them © Datacom 2016
18. Development © Datacom 2016
19. © Datacom 2016 Mule A-MQ App DB 1 call 12 calls 12 calls 102 db queries
20. Testing © Datacom 2016
21. Automated Test Framework results Build 1 Batch Update Process Success Build 2 Batch Update Process Success Build 3 Batch Update Process Success © Datacom 2016 Test Results
22. Automated Test Framework results Build 1 Batch Update Process Success Build 2 Batch Update Process Success Build 3 Batch Update Process Success © Datacom 2016 Test Results Architectural Data #A-MQMsg #DB Queries Avg Rows 12 102 92 12 704 350 20 1054 400
23. © Datacom 2016 Image courtesy of
24. Tip #4 Know exactly how your code behaves, And track how this behaviour changes with each code change. © Datacom 2016
25. Frameworks © Datacom 2016
26. Frameworks © Datacom 2016 • Default Behaviour • Implementation
27. Conclusion © Datacom 2016