Troubleshooting Node.js

Transcript

1. TROUBLESHOOTING NODE.JS what to do when things go wrong Igor

   Soarez igor@yld.io @igorsoarez Hassy Veldstra h@yld.io @hveldstra WDCNZ, July 2015

2. HELLO HELLO • YLD! • Node.js Consultancy & Training •

   Digital transformation for enterprise clients

3. FIRES Someone has to put them out

4. BE PREPARED

5. LOG EVERYTHING Make it easy to grep & visualise

6. TEST TEST TEST • Functional tests first • Good coverage

   for unit tests • Randomised testing & fuzz testing

7. MONITORING & ALERTING • Resource monitoring • CPU • Memory

   • I/O • Application monitoring • Errors • Real-user monitoring • Latency • Availability

8. AUTOMATED DEPLOYMENTS

9. PAIR PROGRAMMING

10. TAKE YOUR TIME WHEN BUILDING YOUR APPLICATION Tight deadlines are

    the root of all evil

11. LOAD TESTING

12. LOAD TESTING • You should do it

13. LOAD TESTING • You should do it • Find the

    bottlenecks

14. LOAD TESTING • You should do it • Find the

    bottlenecks • Understand your system

15. LOAD TESTING • You should do it • Find the

    bottlenecks • Understand your system

16. LOAD TESTING

17. LOAD TESTING

18. YOU SHOULD DO IT IN PRODUCTION

19. IN PRODUCTION??? I must have heard you wrong

20. WHY NOT? Why — Reason 1

21. STAGING IS NEVER THE SAME Why — Reason 1I

22. GAIN EXTRA CONFIDENCE Why — Reason 1II

23. IF YOU DO THIS • Analytics

24. IF YOU DO THIS • Analytics • Custom header •

    Cookie

25. IF YOU DO THIS • Analytics • Custom header •

    Cookie • Let Marketing know

26. IF YOU DO THIS • Analytics • Custom header •

    Cookie • Let Marketing know • Start outside of peak times

27. IF YOU DO THIS • Analytics • Custom header •

    Cookie • Let Marketing know • Start outside of peak times • Start slow & ramp up

28. LOAD TESTING TOOLS • JMeter

29. LOAD TESTING TOOLS • JMeter • Tsung

30. LOAD TESTING TOOLS • JMeter • Tsung • Minigun -

    https://artillery.io/minigun

31. AFTER THE FIRE STARTS how to approach it

32. PERFORMANCE PROBLEMS

33. ATTACK THE PROBLEM 1. Isolated test case * 2. Measure

    3. Change / Tweak / Improve 4. Goto 2 * If possible

34. TYPES OF ISSUES • CPU-bound code • Memory leaks •

    GC pressure • I/O performance • Event loop lag (Node.js)

35. KNOW THE TOOLS AVAILABLE • OS-level • perf • sysstat,

    netstat, netmon, ps, top • sysdig

36. KNOW THE TOOLS AVAILABLE • Platform-specific (Node.js)

37. KNOW THE TOOLS AVAILABLE • Platform-specific (Node.js) • V8 profiler

    • find hot-path functions that cannot be optimized • trace GC activity

38. KNOW THE TOOLS AVAILABLE • Platform-specific (Node.js) • V8 profiler

    • find hot-path functions that cannot be optimized • trace GC activity • heapdump • find memory leaks

39. MEMORY • ps • top / htop • Heapdumps

40. KNOW THE TOOLS AVAILABLE • Platform-specific (Node.js) • V8 profiler

    • find hot-path functions that cannot be optimized • trace GC activity • heapdump • find memory leaks • flamegraphs • see what's hot on the CPU

41. CPU • ps • top / htop • Flamegraphs

42. KNOW THE TOOLS AVAILABLE • Platform-specific (Node.js) • V8 profiler

    • find hot-path functions that cannot be optimized • trace GC activity • heapdump • find memory leaks • flamegraphs • see what's hot on the CPU • node-toobusy-js • track the event loop lag

43. FLAMEGRAPHS

44. PERF aka Performance Counters

45. Collected stack frames h() c() h() c() c() z() c()

    b() f() b() b() f() b() y() b() b() a() a() a() m() a() m() x() m() a() main() main() main() main() main() main() main() main() main()

46. Collapsed stack frames h() c() h() c() z() b() f()

    b() y() a() m() x() main()

47. DEMO!

48. THANK YOU! @hveldstra @igorsoarez