Profiling Node.js apps on production

Profiling Node.js apps on production

A talk about Node.js profiling on production using Linux perf_events and FlameGraph/FlameScope, and a couple of findings

See https://shuheikagawa.com/blog/2018/09/16/node-js-under-a-microscope/ for more details

Ca0ab6e450f894e06652ee257df9d647?s=128

Shuhei Kagawa

June 06, 2019
Tweet

Transcript

  1. 1 Profiling Node.js apps on production

  2. 2 Shuhei Kagawa @shuheikagawa Software Engineer at Zalando Hi, I’m...

  3. 3 Microservices

  4. 4 Node.js servers with React SSR

  5. 5 Performance issue

  6. 6 Mysterious Gap API server API client 500 milliseconds

  7. 7 test env production env

  8. 8

  9. 9 Linux perf

  10. 10 Small overhead

  11. 11 JS & native

  12. 12 node --perf-basic-prof-only-functions

  13. 13 # Install dependencies for `perf` command sudo apt-get install

    linux-tools-common sudo apt-get install linux-tools-$(uname -r)
  14. 14 # Record stack traces 99 times per second for

    30 seconds sudo perf record -F 99 -p ${pid} -g -- sleep 30s # Generate human readable stack traces sudo perf script > stacks.${pid}.out
  15. 15 • Image from http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html 2,970 Stack Traces!!!

  16. 16

  17. 17 Image from https://github.com/brendangregg/FlameGraph CPU Flame Graph by Brendan Gregg

  18. 18 CPU Flame Graph GZIP JSON.parse JSON.parse React

  19. 19 Nothing looks so wrong…?

  20. 20 https://github.com/Netflix/flamescope FlameScope by Netflix cloud performance team

  21. 21 Finding 1: Metrics Collection from Histograms Busy for ~1.5s!

  22. 22 Finding 1: Metrics Collection from Histograms JSON.stringify() JSON.parse() in

    a metrics library
  23. 23 Finding 2: Garbage Collection Busy for ~400ms once in

    ~10s
  24. 24 Finding 2: Garbage Collection Unused fallback cache was causing

    slow GCs
  25. 25

  26. 26 p99 response time: 50% ⬇

  27. 27

  28. 28 Be Bold

  29. 29 Thank you!

  30. 30 • CPU Flame Graphs • FlameScope • A sample

    project • How to fix wrong symbols • Node.js under a Microscope Links