Upgrade to PRO for Only $50/Year—Limited-Time Offer! 🔥

Leak Hunting - Finding and debugging a memory l...

Leak Hunting - Finding and debugging a memory leak in Nodejs.pdf

Presented: NodeFest Japan

Giovanny Gongora

November 23, 2018
Tweet

More Decks by Giovanny Gongora

Other Decks in Technology

Transcript

  1. N O V E M B E R 2 3

    , 2 0 1 8 C O N F I D E N T I A L Leak Hunting Finding and debugging a memory leak in Node.js
  2. C O N F I D E N T I

    A L © 2018 NodeSource Developers/Companies (not all) don’t consider how difficult could be to track and fix a memory leak until it happens
  3. C O N F I D E N T I

    A L © 2018 NodeSource Developers/Companies (not all) don’t consider how difficult could be to track and fix a memory leak until it happens … a day before the release date
  4. C O N F I D E N T I

    A L © 2018 NodeSource Developers/Companies (not all) don’t consider how difficult could be to track and fix a memory leak until it happens … after a new cool feature is pushed
  5. C O N F I D E N T I

    A L © 2018 NodeSource Developers/Companies (not all) don’t consider how difficult could be to track and fix a memory leak until it happens … in production on Friday night
  6. C O N F I D E N T I

    A L © 2018 NodeSource Developers/Companies (not all) don’t consider how difficult could be to track and fix a memory leak until it happens … prod gets more client request than dev tests
  7. © 2018 NodeSource C O N F I D E

    N T I A L Quick brief around Memory in Node.js
  8. © 2018 NodeSource C O N F I D E

    N T I A L Garbage Collection Every program that consumes memory requires a mechanism for reserving and freeing space.
  9. © 2018 NodeSource C O N F I D E

    N T I A L How GC works: • Root R
  10. © 2018 NodeSource C O N F I D E

    N T I A L How GC works: • Root. • Object. R O O O O
  11. © 2018 NodeSource C O N F I D E

    N T I A L How GC works: • Root. • Object. • Primitives. R O O O O P P
  12. © 2018 NodeSource C O N F I D E

    N T I A L How GC works: • Root • Object. • Primitives. • Non referenced object. R O O O O P P O P
  13. © 2018 NodeSource C O N F I D E

    N T I A L Memory scheme (v8): • Resident Set R E S ID E N T SE T
  14. © 2018 NodeSource C O N F I D E

    N T I A L Memory scheme (v8): • Resident Set • Code Segment R E S ID E N T SE T CO DE S E GM E NT
  15. © 2018 NodeSource C O N F I D E

    N T I A L Memory scheme (v8): • Resident Set • Code Segment • Stack R E S ID E N T SE T CO DE S E GM E NT STACK
  16. © 2018 NodeSource C O N F I D E

    N T I A L Memory scheme (v8): • Resident Set • Code Segment • Stack • Heap R E S ID E N T SE T CO DE S E GM E NT STACK H E AP
  17. © 2018 NodeSource C O N F I D E

    N T I A L Memory scheme (v8): • Resident Set • Code Segment • Stack • Heap • Used heap R E S ID E N T SE T CO DE S E GM E NT STACK H E AP RSS 
 H E APTOTA L 
 H E AP US E D
 E XT E R N AL
  18. © 2018 NodeSource C O N F I D E

    N T I A L Relation between GC and a memory leak? • Impossible to remove non referenced data. • Portion of memory allocated not released when no longer needed.
  19. © 2018 NodeSource C O N F I D E

    N T I A L Memory Leak causes: • Registered event handlers • Function closures • Leaking connections
  20. © 2018 NodeSource C O N F I D E

    N T I A L Types of production issues
  21. © 2018 NodeSource C O N F I D E

    N T I A L 1. Runtime performance: • The website/app is slow. • Not giving a great/fast experience to the user.
  22. © 2018 NodeSource C O N F I D E

    N T I A L Things you can do on Runtime performance issues: • --perf-basic-prof-only-functions. • A system tool like Perf Events in Linux. • Spare the traces that are really important in your situation.
  23. © 2018 NodeSource C O N F I D E

    N T I A L 2. Runtime crashes: • Uptime is critical. • Not easily to reproduce. • Resume the process asap.
  24. © 2018 NodeSource C O N F I D E

    N T I A L Things you can do on Runtime crashes: • --abort-on-uncaught-exception. • Use some system tooling llbd, gbd, mdb.
  25. © 2018 NodeSource C O N F I D E

    N T I A L 3. Memory leaks: • This will be our focus today.
  26. © 2018 NodeSource C O N F I D E

    N T I A L Let’s do it!
  27. © 2018 NodeSource C O N F I D E

    N T I A L 27 Process to investigate an issue: • Think and construct an hypothesis • Collect data • Analyze the data (duh) • Draw a conclusion • Repeat (profit?)
  28. © 2018 NodeSource C O N F I D E

    N T I A L 28 Strategy for Memory Leaks detection: • Inspect objects for more context. • Look for weird objects on heap • Growing object counts are likely leaking • Walk reverse references to find the root object • Compare object counts !important
  29. © 2018 NodeSource C O N F I D E

    N T I A L Node.js Using V8 Inspector & Chrome Dev Tools • Run your node process with --inspect • Check chrome://inspect • Run it!
  30. © 2018 NodeSource C O N F I D E

    N T I A L Taking Snapshots of the V8 Memory • Check the Memory tab • Take heap snapshot • Profit!
  31. © 2018 NodeSource C O N F I D E

    N T I A L What is happening? • On the first snapshot, there are already/almost 7-8MB allocated before any request is processed. • Memory keeps growing while the server is running • Date and Objects have been allocated between the two load sessions.
  32. © 2018 NodeSource C O N F I D E

    N T I A L Possible solution: • Reduce Date • The solution for this example is to store the logs not in memory, but on the filesystem
  33. © 2018 NodeSource C O N F I D E

    N T I A L • As you can see, the memory growth is far slower! • This said, the API takes more time to respond. Reading and writing to the disk comes with a cost, so do other API calls or database requests.
  34. © 2018 NodeSource C O N F I D E

    N T I A L Do something before it happens 34
  35. © 2018 NodeSource C O N F I D E

    N T I A L Endurance Testing Primary goal of Endurance testing is to check for memory leaks.
  36. © 2018 NodeSource C O N F I D E

    N T I A L 36 Goals • Discover how the system performs under sustained usage • Ensure that after a long period, the system response time will remain the same or better than the start of the test. • Manage the future loads, we need to understand how many additional resources are necessary to support usage in future
  37. © 2018 NodeSource C O N F I D E

    N T I A L k6 is your friend • Can output more granular metrics than what is printed on stdout • Testing tool for testing the performance of 
 your backend infrastructure • Written in Go and JavaScript
  38. © 2018 NodeSource C O N F I D E

    N T I A L 38 N|Solid to the rescue! • Monitoring and diagnostic tools to help identify and tackle tricky issues. • Production friendly. • Shows performance insights. • Helps you build highly reliable applications and minimize infrastructure costs.
  39. © 2018 NodeSource C O N F I D E

    N T I A L Name you functions.. or not?
  40. © 2018 NodeSource C O N F I D E

    N T I A L • Var test = function () {}
  41. © 2018 NodeSource C O N F I D E

    N T I A L • Var test = function () {} • test.protory.biz = function () {}
  42. © 2018 NodeSource C O N F I D E

    N T I A L • Var test = function () {} • test.protory.biz = function () {} • fs.writeFile(chunck, function(err) {})
  43. © 2018 NodeSource C O N F I D E

    N T I A L • Var test = function () {} • test.protory.biz = function () {} • fs.writeFile(chunck, function(err) {}) • What about arrow functions?
  44. © 2018 NodeSource C O N F I D E

    N T I A L Don’t block on I/O
  45. © 2018 NodeSource C O N F I D E

    N T I A L I/O operations are… • Expensive. • A waste of resources if your program is unable to do other work. • Usually a hidden operation.
  46. © 2018 NodeSource C O N F I D E

    N T I A L Blocking the Event Loop • Totally possible • Happens more times we want it to happen (who wants to block it in first place?) • Iterations and complex calculations would prevent the process from handling other events
  47. © 2018 NodeSource C O N F I D E

    N T I A L Succeed doing I/O operations: • Choose asynchronous over synchronous • Opt for parallel I/O wherever possible • Don't hog the JavaScript thread with long-running calculations and iterations
  48. © 2018 NodeSource C O N F I D E

    N T I A L Async FTW! Modern applications spend most of their time waiting for I/O
  49. © 2018 NodeSource C O N F I D E

    N T I A L • Better work scheduling • Reduces thread CPU overhead • V8 can only do one thing at a time • Spend as little time in V8 as possible
  50. C O N F I D E N T I

    A L © 2018 NodeSource “Measure first before doing any performance change” Franziska Hinkelmann
  51. © 2018 NodeSource C O N F I D E

    N T I A L What we can learn from this: • Is never a good idea wait for Friday at night to learn about memory in Node.js
  52. C O N F I D E N T I

    A L Thank you. Giovanny Gongora [email protected] @Gioyik