Telling Tales & Solving Crimes with New Relic

DevOps and Developers love New Relic, but how can you get your Sales and Project Management teams on board? This presentation tells the tale of two real-world experiences we had in Production and how New Relic helped us uncover just exactly what happened.

Transcript

1. Telling Tales & Solving Crimes uncovering the practical, business side of New Relic
2. “Just like automated deployments and unit tests, New Relic is going to change how we work.”
3. Reacting fast Application Performance Monitoring enables us to deal with issues quickly and definitively
4. Case Study 1 Javascript Errors on the Live site
5. Case Study 1 - ‘jQuery’ is undefined That’s bad. That’s very, very bad. jQuery is key to this application. It needs to work.
6. We tested this - how did that happen?
7. Scenario: • Nobody has reported an issue (yet). • We didn’t pick up on any issues in testing. • Critical issue - the website won’t work without it. Recent changes: • Loading jQuery from a Content Delivery Network. • Feature-detect based embedding of jQuery.
8. Data inspection time... It happens predominantly in Internet Explorer, but the browser version is not to blame, this time.
9. [manual testing] Repeating our test process to ensure that everything we’ve tested for is still working as expected.
10. When all else fails, Google it.
11. • Some Corporate (or Educational Institution) networks will be ‘protected’ by disallowing external resources from Content Delivery Networks. • This will result in 404 Errors when requesting files, which would explain the errors we see in New Relic. • Our solution: put a fallback version of the file locally to continue supporting these customers. Result!
12. Most importantly, we’ve identified issues that real users are experiencing, debugged and resolved them without the customer or the client having to report any issues or provide any details. Which also means we fixed the issue in a fraction of the time!
13. Case Study 2 Server Crash
14. End Users Technical Contact Product Owner DevOps Development Team Customer Support Channels of communication
15. End Users Technical Contact Product Owner DevOps Development Team Customer Support At the time it all kicks off… (11am) PO & Technical AWOL (everyone’s allowed a lunch break)
16. End Users Technical Contact Product Owner DevOps Development Team Customer Support 11:00am ! Warning: High load on server DevOps team gets advance warning of high load on server and begin investigating.
17. End Users Technical Contact Product Owner DevOps Development Team Customer Support 11:31am ! ! ! Alert: Server unavailable ! ! Alert: Downtime Server Crash! Email alerts for everyone! Customer Support team knows of the downtime instantly. 500 Error
18. Fortunately, DevOps have been aware of the issue for 30 minutes already and attempting to fix it. When the server finally dies, they talk to the the Development Team about the possibility of simply rebooting the server. Solution & implications agreed: Server is rebooted.
19. End Users Technical Contact Product Owner DevOps Development Team Customer Support 12:00am Rebooted server comes online and restores service.
20. Server Crash: Fallout • First instance of downtime since we started using New Relic. • Server ‘officially’ down for 29 minutes. (Customer Support were only aware of downtime for final 11 minutes) • Enhanced visibility of server health meant remediation steps were underway before the downtime started. • Downtime issue was resolved in the same time it took for meetings about the downtime to be arranged. • Once normal operation is resumed, we can use New Relic data & server logs to perform a ‘post mortem’ on the incident.
21. “Real user data is much, much better than artificially-created lab results”