Node Interactive US 2016: Real-life node.js troubleshooting

Slide 1

Slide 1 text

Real-Life Node.js Troubleshooting Damian Schenkelman, Auth0

Slide 2

Slide 2 text

4 years

Slide 3

Slide 3 text

Usual suspects

Slide 4

Slide 4 text

Memory Leaks

Slide 5

Slide 5 text

–Sebastian Peyrott (a co-worker) “The main cause for leaks in garbage collected languages are unwanted references”

Slide 6

Slide 6 text

Memory Model https://developers.google.com/web/tools/chrome- devtools/memory-problems/memory-101 A B C F E D G GC H I

Slide 7

Slide 7 text

b.d = null A B C F E D G GC H I

Slide 8

Slide 8 text

D not referenced A B C F E D G GC H I

Slide 9

Slide 9 text

D is GCed A B C F E G GC H I

Slide 10

Slide 10 text

Sawtooth

Slide 11

Slide 11 text

Memory Leak A B C F E D G Things we don’t need and are taking up a LOT of space GC

Slide 12

Slide 12 text

Memory Leak A B C F E D G Things we don’t need and are taking up a LOT of space GC More things we don’t need

Slide 13

Slide 13 text

Crashes

Slide 14

Slide 14 text

Memory leak busting

Slide 15

Slide 15 text

Take Control

Slide 16

Slide 16 text

--max-old-space-size is your friend

Slide 17

Slide 17 text

Drain connections

Slide 18

Slide 18 text

Restarts

Slide 19

Slide 19 text

Heap Snapshot

Slide 20

Slide 20 text

v8-proﬁler https://github.com/node-inspector/v8-proﬁler var profiler = require('v8-profiler'); var snapshot = profiler.takeSnapshot(); snapshot.export(function(error, result) { fs.writeFileSync('snapshot.json', result); snapshot.delete(); });

Slide 21

Slide 21 text

Demo

Slide 22

Slide 22 text

No content

Slide 23

Slide 23 text

No content

Slide 24

Slide 24 text

No content

Slide 25

Slide 25 text

No content

Slide 26

Slide 26 text

Mental Picture { 'kinesis.us-west-1.amazonaws.co:443' : } { sockets : } [ …, …, …, …, …, …, …, …,…, TLSSocket ] Something related to logging…

Slide 27

Slide 27 text

Not Easy https://github.com/auth0/kinesis-writable/pull/6/ﬁles

Slide 28

Slide 28 text

forever-agent https://github.com/request/forever-agent/blob/ ece900a6e8dfac734186db3080c00875c0300450/index.js#L70 if (this.freeSockets[name] && this.freeSockets[name].length > 0 && !req.useChunkedEncodingByDefault) { var idleSocket = this.freeSockets[name].pop() idleSocket.removeListener('error', idleSocket._onIdleError) delete idleSocket._onIdleError req._reusedSocket = true req.onSocket(idleSocket) } else { this.addRequestNoreuse(req, host, port) }

Slide 29

Slide 29 text

Fixed

Slide 30

Slide 30 text

CPU bottlenecks

Slide 31

Slide 31 text

Auth Service DB Scenario Client

Slide 32

Slide 32 text

Expected response times

Slide 33

Slide 33 text

Actual response times

Slide 34

Slide 34 text

Flame graphs a() b() c() d() h() i() e() f() g()

Slide 35

Slide 35 text

Flame graphs

Slide 36

Slide 36 text

Demo

Slide 37

Slide 37 text

The problem

Slide 38

Slide 38 text

Scale

Slide 39

Slide 39 text

Faster Hash

Slide 40

Slide 40 text

Caching

Slide 41

Slide 41 text

Scale up

Slide 42

Slide 42 text

Multiple Auth Services LB Auth Service Auth Service Auth Service Auth Service

Slide 43

Slide 43 text

BaaS https://github.com/auth0/node-baas // compare bcrypt for req.password to db hash baas.compare(req.body.password, user.passwordHash, (err, success) => { res.send(err || !success ? 401 : 200); });

Slide 44

Slide 44 text

Auto Scaling Auth Service LB BaaS BaaS BaaS BaaS

Slide 45

Slide 45 text

Cost comparison Price / (1M req) #req per sec / vCPU t2-micro $0.36 10.00 t2-medium $0.76 9.50 c4-large $1.53 10.00 c3-8xlarge $1.64 8.88

Slide 46

Slide 46 text

Fail gracefully

Slide 47

Slide 47 text

Cat picture

Slide 48

Slide 48 text

Memory Leaks • Memory terminology: https://developers.google.com/web/tools/ chrome-devtools/memory-problems/memory-101 • Memory leaks in JS: https://auth0.com/blog/four-types-of-leaks- in-your-javascript-code-and-how-to-get-rid-of-them/ • Roots: https://stackoverﬂow.com/questions/9748358/when- does-the-js-engine-create-a-garbage-collection-root • Increase heap size: https://twitter.com/tjholowaychuk/status/ 480753206301966336 • v8 GC logs speaking to you: https://github.com/joyeecheung/ v8-gc-talk

Slide 49

Slide 49 text

Memory Leaks (2) • Connection draining: https://cloud.google.com/ compute/docs/load-balancing/enabling-connection- draining • master-process: https://github.com/jfromaniello/ master-process • Express connection draining: https://github.com/ expressjs/express/issues/1366 • v8-proﬁler: https://github.com/node-inspector/v8- proﬁler

Slide 50

Slide 50 text

Memory Leaks (3) • AWS SDK issues: https://github.com/aws/aws-sdk-js/issues/329, https://github.com/aws/aws-sdk-js/issues/855 • Kinesis event fix: https://github.com/auth0/kinesis-writable/pull/6/files • Forever agent: https://github.com/request/forever-agent/blob/ ece900a6e8dfac734186db3080c00875c0300450/index.js#L70 • Memory profiling for mere mortals: http://thlorenz.com/talks/memory- profiling/book/memory_profiling_for_mere_mortals_/ memory_profiling_for_mere_mortals__0.html • @thlorenz notes on mem profiling: https://github.com/thlorenz/v8- perf/blob/master/memory-profiling.md

Slide 51

Slide 51 text

Performance • http://security.stackexchange.com/a/83382 • http://www.brendangregg.com/FlameGraphs/cpuﬂamegraphs.html • https://github.com/davidmarkclements/0x • https://github.com/thlorenz/v8-perf/issues/4 • https://github.com/auth0/node-baas • http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/ US_SetUpASLBApp.html • https://gist.github.com/trevnorris/9616784 • https://github.com/dschenkelman/bcrypt-sample

Slide 52

Slide 52 text

Thanks https://github.com/dschenkelman/node-troubleshooting @dschenkelman npm i dschenkelman