Profiling Node.js apps on production

A talk about Node.js profiling on production using Linux perf_events and FlameGraph/FlameScope, and a couple of findings

See https://shuheikagawa.com/blog/2018/09/16/node-js-under-a-microscope/ for more details

Transcript

1. 1 Profiling Node.js apps on production
2. 2 Shuhei Kagawa @shuheikagawa Software Engineer at Zalando Hi, I’m...
3. 3 Microservices
4. 4 Node.js servers with React SSR
5. 5 Performance issue
6. 6 Mysterious Gap API server API client 500 milliseconds
7. 7 test env production env
9. 9 Linux perf
10. 10 Small overhead
11. 11 JS & native
12. 12 node --perf-basic-prof-only-functions
13. 13 # Install dependencies for `perf` command sudo apt-get install linux-tools-common sudo apt-get install linux-tools-$(uname -r)
14. 14 # Record stack traces 99 times per second for 30 seconds sudo perf record -F 99 -p ${pid} -g -- sleep 30s # Generate human readable stack traces sudo perf script > stacks.${pid}.out
15. 15 • Image from http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html 2,970 Stack Traces!!!
17. 17 Image from https://github.com/brendangregg/FlameGraph CPU Flame Graph by Brendan Gregg
18. 18 CPU Flame Graph GZIP JSON.parse JSON.parse React
19. 19 Nothing looks so wrong…?
20. 20 https://github.com/Netflix/flamescope FlameScope by Netflix cloud performance team
21. 21 Finding 1: Metrics Collection from Histograms Busy for ~1.5s!
22. 22 Finding 1: Metrics Collection from Histograms JSON.stringify() JSON.parse() in a metrics library
23. 23 Finding 2: Garbage Collection Busy for ~400ms once in ~10s
24. 24 Finding 2: Garbage Collection Unused fallback cache was causing slow GCs
26. 26 p99 response time: 50% ⬇
28. 28 Be Bold
29. 29 Thank you!
30. 30 • CPU Flame Graphs • FlameScope • A sample project • How to fix wrong symbols • Node.js under a Microscope Links