Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Node Interactive US 2016: Real-life node.js troubleshooting

Node Interactive US 2016: Real-life node.js troubleshooting

When building a large enough set of services using node.js, there will be a point when you find that your application is suffering from performance or memory issues. When this happens, you have to roll up your sleeves, get your tools and start digging. This talk explains how you can use tools such as ab, flame graphs, heap snapshots and Chrome's memory inspector to find the cause of these. We will go over two real life issues, a CPU bottleneck and a memory leak, we found while building our services at Auth0, and also explain how we fixed them.

Damian Schenkelman

November 30, 2016
Tweet

More Decks by Damian Schenkelman

Other Decks in Programming

Transcript

  1. –Sebastian Peyrott (a co-worker) “The main cause for leaks in

    garbage collected languages are unwanted references”
  2. Memory Leak A B C F E D G Things

    we don’t need and are taking up a LOT of space GC
  3. Memory Leak A B C F E D G Things

    we don’t need and are taking up a LOT of space GC More things we don’t need
  4. v8-profiler https://github.com/node-inspector/v8-profiler var profiler = require('v8-profiler'); var snapshot = profiler.takeSnapshot();

    snapshot.export(function(error, result) { fs.writeFileSync('snapshot.json', result); snapshot.delete(); });
  5. Mental Picture { 'kinesis.us-west-1.amazonaws.co:443' : } { sockets : }

    [ …, …, …, …, …, …, …, …,…, TLSSocket ] Something related to logging…
  6. forever-agent https://github.com/request/forever-agent/blob/ ece900a6e8dfac734186db3080c00875c0300450/index.js#L70 if (this.freeSockets[name] && this.freeSockets[name].length > 0 &&

    !req.useChunkedEncodingByDefault) { var idleSocket = this.freeSockets[name].pop() idleSocket.removeListener('error', idleSocket._onIdleError) delete idleSocket._onIdleError req._reusedSocket = true req.onSocket(idleSocket) } else { this.addRequestNoreuse(req, host, port) }
  7. BaaS https://github.com/auth0/node-baas // compare bcrypt for req.password to db hash

    baas.compare(req.body.password, user.passwordHash, (err, success) => { res.send(err || !success ? 401 : 200); });
  8. Cost comparison Price / (1M req) #req per sec /

    vCPU t2-micro $0.36 10.00 t2-medium $0.76 9.50 c4-large $1.53 10.00 c3-8xlarge $1.64 8.88
  9. Memory Leaks • Memory terminology: https://developers.google.com/web/tools/ chrome-devtools/memory-problems/memory-101 • Memory leaks

    in JS: https://auth0.com/blog/four-types-of-leaks- in-your-javascript-code-and-how-to-get-rid-of-them/ • Roots: https://stackoverflow.com/questions/9748358/when- does-the-js-engine-create-a-garbage-collection-root • Increase heap size: https://twitter.com/tjholowaychuk/status/ 480753206301966336 • v8 GC logs speaking to you: https://github.com/joyeecheung/ v8-gc-talk
  10. Memory Leaks (2) • Connection draining: https://cloud.google.com/ compute/docs/load-balancing/enabling-connection- draining •

    master-process: https://github.com/jfromaniello/ master-process • Express connection draining: https://github.com/ expressjs/express/issues/1366 • v8-profiler: https://github.com/node-inspector/v8- profiler
  11. Memory Leaks (3) • AWS SDK issues: https://github.com/aws/aws-sdk-js/issues/329, https://github.com/aws/aws-sdk-js/issues/855 •

    Kinesis event fix: https://github.com/auth0/kinesis-writable/pull/6/files • Forever agent: https://github.com/request/forever-agent/blob/ ece900a6e8dfac734186db3080c00875c0300450/index.js#L70 • Memory profiling for mere mortals: http://thlorenz.com/talks/memory- profiling/book/memory_profiling_for_mere_mortals_/ memory_profiling_for_mere_mortals__0.html • @thlorenz notes on mem profiling: https://github.com/thlorenz/v8- perf/blob/master/memory-profiling.md
  12. Performance • http://security.stackexchange.com/a/83382 • http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html • https://github.com/davidmarkclements/0x • https://github.com/thlorenz/v8-perf/issues/4 •

    https://github.com/auth0/node-baas • http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/ US_SetUpASLBApp.html • https://gist.github.com/trevnorris/9616784 • https://github.com/dschenkelman/bcrypt-sample