Slide 1

Slide 1 text

1 Profiling Node.js apps on production

Slide 2

Slide 2 text

2 Shuhei Kagawa @shuheikagawa Software Engineer at Zalando Hi, I’m...

Slide 3

Slide 3 text

3 Microservices

Slide 4

Slide 4 text

4 Node.js servers with React SSR

Slide 5

Slide 5 text

5 Performance issue

Slide 6

Slide 6 text

6 Mysterious Gap API server API client 500 milliseconds

Slide 7

Slide 7 text

7 test env production env

Slide 8

Slide 8 text

8

Slide 9

Slide 9 text

9 Linux perf

Slide 10

Slide 10 text

10 Small overhead

Slide 11

Slide 11 text

11 JS & native

Slide 12

Slide 12 text

12 node --perf-basic-prof-only-functions

Slide 13

Slide 13 text

13 # Install dependencies for `perf` command sudo apt-get install linux-tools-common sudo apt-get install linux-tools-$(uname -r)

Slide 14

Slide 14 text

14 # Record stack traces 99 times per second for 30 seconds sudo perf record -F 99 -p ${pid} -g -- sleep 30s # Generate human readable stack traces sudo perf script > stacks.${pid}.out

Slide 15

Slide 15 text

15 • Image from http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html 2,970 Stack Traces!!!

Slide 16

Slide 16 text

16

Slide 17

Slide 17 text

17 Image from https://github.com/brendangregg/FlameGraph CPU Flame Graph by Brendan Gregg

Slide 18

Slide 18 text

18 CPU Flame Graph GZIP JSON.parse JSON.parse React

Slide 19

Slide 19 text

19 Nothing looks so wrong…?

Slide 20

Slide 20 text

20 https://github.com/Netflix/flamescope FlameScope by Netflix cloud performance team

Slide 21

Slide 21 text

21 Finding 1: Metrics Collection from Histograms Busy for ~1.5s!

Slide 22

Slide 22 text

22 Finding 1: Metrics Collection from Histograms JSON.stringify() JSON.parse() in a metrics library

Slide 23

Slide 23 text

23 Finding 2: Garbage Collection Busy for ~400ms once in ~10s

Slide 24

Slide 24 text

24 Finding 2: Garbage Collection Unused fallback cache was causing slow GCs

Slide 25

Slide 25 text

25

Slide 26

Slide 26 text

26 p99 response time: 50% ⬇

Slide 27

Slide 27 text

27

Slide 28

Slide 28 text

28 Be Bold

Slide 29

Slide 29 text

29 Thank you!

Slide 30

Slide 30 text

30 • CPU Flame Graphs • FlameScope • A sample project • How to fix wrong symbols • Node.js under a Microscope Links