Tweet
Tweet

Slide 1

Slide 1 text

J U LY 2 3 , 2 0 1 8 C O N F I D E N T I A L Leak Hunting Finding and debugging a memory leak in Node.js

Slide 2

Slide 2 text

© 2018 NodeSource C O N F I D E N T I A L We do a lot of Node! (crazy right?) • N|Solid • NCM • ~18 companies trust our support team for inquiries and problems around node. • …expect more

Slide 3

Slide 3 text

C O N F I D E N T I A L © 2018 NodeSource Developers/Companies (not all) don’t consider how difficult could be to track and fix a memory leak until it happens

Slide 4

Slide 4 text

C O N F I D E N T I A L © 2018 NodeSource Developers/Companies (not all) don’t consider how difficult could be to track and fix a memory leak until it happens … a day before of a new version to be released

Slide 5

Slide 5 text

C O N F I D E N T I A L © 2018 NodeSource Developers/Companies (not all) don’t consider how difficult could be to track and fix a memory leak until it happens … after a new cool feature is pushed

Slide 6

Slide 6 text

C O N F I D E N T I A L © 2018 NodeSource Developers/Companies (not all) don’t consider how difficult could be to track and fix a memory leak until it happens … in production on Friday night

Slide 7

Slide 7 text

C O N F I D E N T I A L © 2018 NodeSource Developers/Companies (not all) don’t consider how difficult could be to track and fix a memory leak until it happens … prod gets more client request than dev tests

Slide 8

Slide 8 text

© 2018 NodeSource C O N F I D E N T I A L Types of production issues

Slide 9

Slide 9 text

© 2018 NodeSource C O N F I D E N T I A L Runtime performance: • The website/app is slow. • Not giving a great/fast experience to the user.

Slide 10

Slide 10 text

© 2018 NodeSource C O N F I D E N T I A L Things you can do on Runtime performance issues: • --perf-basic-prof-only-functions. • A system tool like Perf Events in Linux. • Spare the traces that are really important in your situation.

Slide 11

Slide 11 text

© 2018 NodeSource C O N F I D E N T I A L Runtime crashes: • Uptime is critical. • Not easily to reproduce. • Resume the process asap.

Slide 12

Slide 12 text

© 2018 NodeSource C O N F I D E N T I A L Things you can do on Runtime crashes: • --abort-on-uncaught-exception. • Use some system tooling llbd, gbd, mdb.

Slide 13

Slide 13 text

© 2018 NodeSource C O N F I D E N T I A L Memory leaks: • This will be our focus today.

Slide 14

Slide 14 text

© 2018 NodeSource C O N F I D E N T I A L Quick brief around Memory in Node.js

Slide 15

Slide 15 text

© 2018 NodeSource C O N F I D E N T I A L Garbage Collection Every program that consumes memory requires a mechanism for reserving and freeing space.

Slide 16

Slide 16 text

© 2018 NodeSource C O N F I D E N T I A L How GC works: • Root R

Slide 17

Slide 17 text

© 2018 NodeSource C O N F I D E N T I A L How GC works: • Root. • Object. R O O O O

Slide 18

Slide 18 text

© 2018 NodeSource C O N F I D E N T I A L How GC works: • Root. • Object. • Primitives. R O O O O P P

Slide 19

Slide 19 text

© 2018 NodeSource C O N F I D E N T I A L How GC works: • Root • Object. • Primitives. • Non referenced object. R O O O O P P O P

Slide 20

Slide 20 text

© 2018 NodeSource C O N F I D E N T I A L Memory scheme (v8): • Resident Set R E S ID E N T SE T

Slide 21

Slide 21 text

© 2018 NodeSource C O N F I D E N T I A L Memory scheme (v8): • Resident Set • Code Segment R E S ID E N T SE T CO DE S E GM E NT

Slide 22

Slide 22 text

© 2018 NodeSource C O N F I D E N T I A L Memory scheme (v8): • Resident Set • Code Segment • Stack R E S ID E N T SE T CO DE S E GM E NT STACK

Slide 23

Slide 23 text

© 2018 NodeSource C O N F I D E N T I A L Memory scheme (v8): • Resident Set • Code Segment • Stack • Heap R E S ID E N T SE T CO DE S E GM E NT STACK H E AP

Slide 24

Slide 24 text

© 2018 NodeSource C O N F I D E N T I A L Memory scheme (v8): • Resident Set • Code Segment • Stack • Heap • Used heap R E S ID E N T SE T CO DE S E GM E NT STACK H E AP RSS 
 H E APTOTA L 
 H E AP US E D
 E XT E R N AL

Slide 25

Slide 25 text

© 2018 NodeSource C O N F I D E N T I A L Relation between GC and a memory leak? • Impossible to remove non referenced data. • Portion of memory allocated not released when no longer needed.

Slide 26

Slide 26 text

© 2018 NodeSource C O N F I D E N T I A L Memory Leak causes: • Registered event handlers • Function closures • Leaking connections

Slide 27

Slide 27 text

© 2018 NodeSource C O N F I D E N T I A L Process to investigate an issue: • Think and construct an hypothesis • Collect data • Analyze the data (duh) • Draw a conclusion • Repeat (profit?)

Slide 28

Slide 28 text

© 2018 NodeSource C O N F I D E N T I A L Strategy for Memory Leaks detection: • Inspect objects for more context. • Look for weird objects on heap • Growing object counts are likely leaking • Walk reverse references to find the root object • Compare object counts !important

Slide 29

Slide 29 text

© 2018 NodeSource C O N F I D E N T I A L Let’s do it!

Slide 30

Slide 30 text

© 2018 NodeSource C O N F I D E N T I A L Node.js Using V8 Inspector & Chrome Dev Tools • Run your node process with --inspect • Check chrome://inspect • Run it!

Slide 31

Slide 31 text

© 2018 NodeSource C O N F I D E N T I A L Taking Snapshots of the V8 Memory • Check the Memory tab • Take heap snapshot • Profit!

Slide 32

Slide 32 text

© 2018 NodeSource C O N F I D E N T I A L What is happening? • On the first snapshot, there are already/almost 5MB allocated before any request is processed. • Memory keeps growing while the server is running • Exactly # Date objects and # Objects have been allocated between the two load sessions.

Slide 33

Slide 33 text

© 2018 NodeSource C O N F I D E N T I A L Possible solution: • Reduce Date • The solution for this example is to store the logs not in memory, but on the filesystem

Slide 34

Slide 34 text

© 2018 NodeSource C O N F I D E N T I A L • As you can see, the memory growth is far slower! • This said, the API takes more time to respond. Reading and writing to the disk comes with a cost, so do other API calls or database requests.

Slide 35

Slide 35 text

© 2018 NodeSource C O N F I D E N T I A L Last considerations?

Slide 36

Slide 36 text

© 2018 NodeSource C O N F I D E N T I A L Name you functions

Slide 37

Slide 37 text

© 2018 NodeSource C O N F I D E N T I A L • Var test = function () {}

Slide 38

Slide 38 text

© 2018 NodeSource C O N F I D E N T I A L • Var test = function () {} • test.protory.biz = function () {}

Slide 39

Slide 39 text

© 2018 NodeSource C O N F I D E N T I A L • Var test = function () {} • test.protory.biz = function () {} • fs.writeFile(chunck, function(err) {})

Slide 40

Slide 40 text

© 2018 NodeSource C O N F I D E N T I A L • Var test = function () {} • test.protory.biz = function () {} • fs.writeFile(chunck, function(err) {}) • What about arrow functions?

Slide 41

Slide 41 text

© 2018 NodeSource C O N F I D E N T I A L Don’t block on I/O

Slide 42

Slide 42 text

© 2018 NodeSource C O N F I D E N T I A L I/O operations are… • Expensive. • A waste of resources if your program is unable to do other work. • Usually a hidden operation.

Slide 43

Slide 43 text

© 2018 NodeSource C O N F I D E N T I A L Blocking the Event Loop • Totally possible • Happens more times we want them to happen • Iterations and complex calculations would prevent the process from handling other events

Slide 44

Slide 44 text

© 2018 NodeSource C O N F I D E N T I A L Succeed doing I/O operations: • Choose asynchronous over synchronous • Opt for parallel I/O wherever possible • Don't hog the JavaScript thread with long-running calculations and iterations

Slide 45

Slide 45 text

© 2018 NodeSource C O N F I D E N T I A L Async FTW! Modern applications spend most of their time waiting for I/O

Slide 46

Slide 46 text

© 2018 NodeSource C O N F I D E N T I A L • Better work scheduling • Reduces thread CPU overhead • V8 can only do one thing at a time • Spend as little time in V8 as possible

Slide 47

Slide 47 text

© 2018 NodeSource C O N F I D E N T I A L What we can learn from this: • A running Node.js application uses asynchronous I/O for performance. • Node.js runs JavaScript in a single thread. Each JavaScript function runs until completion. • I/O operations can be performed synchronously, but will occupy the main thread and prevent events from being processed

Slide 48

Slide 48 text

C O N F I D E N T I A L Thank you. Giovanny Gongora [email protected] @Gioyik