Troubleshooting Node.js

what to do when things go wrong

Transcript

1. TROUBLESHOOTING NODE.JS what to do when things go wrong Igor Soarez igor@yld.io @igorsoarez Hassy Veldstra h@yld.io @hveldstra WDCNZ, July 2015
2. HELLO HELLO • YLD! • Node.js Consultancy & Training • Digital transformation for enterprise clients
3. FIRES Someone has to put them out
4. BE PREPARED
5. LOG EVERYTHING Make it easy to grep & visualise
6. TEST TEST TEST • Functional tests first • Good coverage for unit tests • Randomised testing & fuzz testing
7. MONITORING & ALERTING • Resource monitoring • CPU • Memory • I/O • Application monitoring • Errors • Real-user monitoring • Latency • Availability
8. AUTOMATED DEPLOYMENTS
9. PAIR PROGRAMMING
10. TAKE YOUR TIME WHEN BUILDING YOUR APPLICATION Tight deadlines are the root of all evil
11. LOAD TESTING
12. LOAD TESTING • You should do it
13. LOAD TESTING • You should do it • Find the bottlenecks
14. LOAD TESTING • You should do it • Find the bottlenecks • Understand your system
15. LOAD TESTING • You should do it • Find the bottlenecks • Understand your system
16. LOAD TESTING
17. LOAD TESTING
18. YOU SHOULD DO IT IN PRODUCTION
19. IN PRODUCTION??? I must have heard you wrong
20. WHY NOT? Why — Reason 1
21. STAGING IS NEVER THE SAME Why — Reason 1I
22. GAIN EXTRA CONFIDENCE Why — Reason 1II
23. IF YOU DO THIS • Analytics
24. IF YOU DO THIS • Analytics • Custom header • Cookie
25. IF YOU DO THIS • Analytics • Custom header • Cookie • Let Marketing know
26. IF YOU DO THIS • Analytics • Custom header • Cookie • Let Marketing know • Start outside of peak times
27. IF YOU DO THIS • Analytics • Custom header • Cookie • Let Marketing know • Start outside of peak times • Start slow & ramp up
28. LOAD TESTING TOOLS • JMeter
29. LOAD TESTING TOOLS • JMeter • Tsung
30. LOAD TESTING TOOLS • JMeter • Tsung • Minigun - https://artillery.io/minigun
31. AFTER THE FIRE STARTS how to approach it
32. PERFORMANCE PROBLEMS
33. ATTACK THE PROBLEM 1. Isolated test case * 2. Measure 3. Change / Tweak / Improve 4. Goto 2 * If possible
34. TYPES OF ISSUES • CPU-bound code • Memory leaks • GC pressure • I/O performance • Event loop lag (Node.js)
35. KNOW THE TOOLS AVAILABLE • OS-level • perf • sysstat, netstat, netmon, ps, top • sysdig
36. KNOW THE TOOLS AVAILABLE • Platform-specific (Node.js)
37. KNOW THE TOOLS AVAILABLE • Platform-specific (Node.js) • V8 profiler • find hot-path functions that cannot be optimized • trace GC activity
38. KNOW THE TOOLS AVAILABLE • Platform-specific (Node.js) • V8 profiler • find hot-path functions that cannot be optimized • trace GC activity • heapdump • find memory leaks
39. MEMORY • ps • top / htop • Heapdumps
40. KNOW THE TOOLS AVAILABLE • Platform-specific (Node.js) • V8 profiler • find hot-path functions that cannot be optimized • trace GC activity • heapdump • find memory leaks • flamegraphs • see what's hot on the CPU
41. CPU • ps • top / htop • Flamegraphs
42. KNOW THE TOOLS AVAILABLE • Platform-specific (Node.js) • V8 profiler • find hot-path functions that cannot be optimized • trace GC activity • heapdump • find memory leaks • flamegraphs • see what's hot on the CPU • node-toobusy-js • track the event loop lag
43. FLAMEGRAPHS
44. PERF aka Performance Counters
45. Collected stack frames h() c() h() c() c() z() c() b() f() b() b() f() b() y() b() b() a() a() a() m() a() m() x() m() a() main() main() main() main() main() main() main() main() main()
46. Collapsed stack frames h() c() h() c() z() b() f() b() y() a() m() x() main()
47. DEMO!
48. THANK YOU! @hveldstra @igorsoarez