Node Interactive US 2016: Real-life node.js troubleshooting

When building a large enough set of services using node.js, there will be a point when you find that your application is suffering from performance or memory issues. When this happens, you have to roll up your sleeves, get your tools and start digging. This talk explains how you can use tools such as ab, flame graphs, heap snapshots and Chrome's memory inspector to find the cause of these. We will go over two real life issues, a CPU bottleneck and a memory leak, we found while building our services at Auth0, and also explain how we fixed them.

Transcript

1. Real-Life Node.js
   Troubleshooting
   Damian Schenkelman, Auth0
   

   View Slide

2. 4 years
   

   View Slide

3. Usual suspects
   

   View Slide

4. Memory Leaks
   

   View Slide

5. –Sebastian Peyrott (a co-worker)
   “The main cause for leaks in garbage collected
   languages are unwanted references”
   

   View Slide

6. Memory Model
   https://developers.google.com/web/tools/chrome-
   devtools/memory-problems/memory-101
   A
   B
   C
   F
   E
   D
   G
   GC
   H I
   

   View Slide

7. b.d = null
   A
   B
   C
   F
   E
   D
   G
   GC
   H I
   

   View Slide

8. D not referenced
   A
   B
   C
   F
   E
   D
   G
   GC
   H I
   

   View Slide

9. D is GCed
   A
   B
   C
   F
   E G
   GC
   H I
   

   View Slide

10. Sawtooth
    

    View Slide

11. Memory Leak
    A
    B
    C
    F
    E
    D
    G
    Things we don’t need and are
    taking up a LOT of space
    GC
    

    View Slide

12. Memory Leak
    A
    B
    C
    F
    E
    D
    G
    Things we don’t need and are
    taking up a LOT of space
    GC
    More things we don’t need
    

    View Slide

13. Crashes
    

    View Slide

14. Memory leak busting
    

    View Slide

15. Take Control
    

    View Slide

16. --max-old-space-size
    is your friend
    

    View Slide

17. Drain connections
    

    View Slide

18. Restarts
    

    View Slide

19. Heap Snapshot
    

    View Slide

20. v8-profiler
    https://github.com/node-inspector/v8-profiler
    var profiler = require('v8-profiler');
    var snapshot = profiler.takeSnapshot();
    snapshot.export(function(error, result) {
    fs.writeFileSync('snapshot.json', result);
    snapshot.delete();
    });
    

    View Slide

21. Demo
    

    View Slide

22. View Slide

23. View Slide

24. View Slide

25. View Slide

26. Mental Picture
    {
    'kinesis.us-west-1.amazonaws.co:443' :
    }
    {
    sockets :
    }
    [
    …, …, …, …, …, …, …, …,…, TLSSocket
    ]
    Something related to
    logging…
    

    View Slide

27. Not Easy
    https://github.com/auth0/kinesis-writable/pull/6/files
    

    View Slide

28. forever-agent
    https://github.com/request/forever-agent/blob/
    ece900a6e8dfac734186db3080c00875c0300450/index.js#L70
    if (this.freeSockets[name] && this.freeSockets[name].length > 0
    && !req.useChunkedEncodingByDefault) {
    var idleSocket = this.freeSockets[name].pop()
    idleSocket.removeListener('error', idleSocket._onIdleError)
    delete idleSocket._onIdleError
    req._reusedSocket = true
    req.onSocket(idleSocket)
    } else {
    this.addRequestNoreuse(req, host, port)
    }
    

    View Slide

29. Fixed
    

    View Slide

30. CPU bottlenecks
    

    View Slide

31. Auth Service
    DB
    Scenario
    Client
    

    View Slide

32. Expected response times
    

    View Slide

33. Actual response times
    

    View Slide

34. Flame graphs
    a()
    b()
    c()
    d()
    h()
    i()
    e() f()
    g()
    

    View Slide

35. Flame graphs
    

    View Slide

36. Demo
    

    View Slide

37. The problem
    

    View Slide

38. Scale
    

    View Slide

39. Faster Hash
    

    View Slide

40. Caching
    

    View Slide

41. Scale up
    

    View Slide

42. Multiple Auth Services
    LB
    Auth Service
    Auth Service
    Auth Service
    Auth Service
    

    View Slide

43. BaaS
    https://github.com/auth0/node-baas
    // compare bcrypt for req.password to db hash
    baas.compare(req.body.password, user.passwordHash,
    (err, success) => {
    res.send(err || !success ? 401 : 200);
    });
    

    View Slide

44. Auto Scaling
    Auth Service LB
    BaaS
    BaaS
    BaaS
    BaaS
    

    View Slide

45. Cost comparison
    Price / (1M req)
    #req per sec /
    vCPU
    t2-micro $0.36 10.00
    t2-medium $0.76 9.50
    c4-large $1.53 10.00
    c3-8xlarge $1.64 8.88
    

    View Slide

46. Fail gracefully
    

    View Slide

47. Cat picture
    

    View Slide

48. Memory Leaks
    • Memory terminology: https://developers.google.com/web/tools/
    chrome-devtools/memory-problems/memory-101
    • Memory leaks in JS: https://auth0.com/blog/four-types-of-leaks-
    in-your-javascript-code-and-how-to-get-rid-of-them/
    • Roots: https://stackoverflow.com/questions/9748358/when-
    does-the-js-engine-create-a-garbage-collection-root
    • Increase heap size: https://twitter.com/tjholowaychuk/status/
    480753206301966336
    • v8 GC logs speaking to you: https://github.com/joyeecheung/
    v8-gc-talk
    

    View Slide

49. Memory Leaks (2)
    • Connection draining: https://cloud.google.com/
    compute/docs/load-balancing/enabling-connection-
    draining
    • master-process: https://github.com/jfromaniello/
    master-process
    • Express connection draining: https://github.com/
    expressjs/express/issues/1366
    • v8-profiler: https://github.com/node-inspector/v8-
    profiler
    

    View Slide

50. Memory Leaks (3)
    • AWS SDK issues: https://github.com/aws/aws-sdk-js/issues/329,
    https://github.com/aws/aws-sdk-js/issues/855
    • Kinesis event fix: https://github.com/auth0/kinesis-writable/pull/6/files
    • Forever agent: https://github.com/request/forever-agent/blob/
    ece900a6e8dfac734186db3080c00875c0300450/index.js#L70
    • Memory profiling for mere mortals: http://thlorenz.com/talks/memory-
    profiling/book/memory_profiling_for_mere_mortals_/
    memory_profiling_for_mere_mortals__0.html
    • @thlorenz notes on mem profiling: https://github.com/thlorenz/v8-
    perf/blob/master/memory-profiling.md
    

    View Slide

51. Performance
    • http://security.stackexchange.com/a/83382
    • http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html
    • https://github.com/davidmarkclements/0x
    • https://github.com/thlorenz/v8-perf/issues/4
    • https://github.com/auth0/node-baas
    • http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/
    US_SetUpASLBApp.html
    • https://gist.github.com/trevnorris/9616784
    • https://github.com/dschenkelman/bcrypt-sample
    

    View Slide

52. Thanks
    https://github.com/dschenkelman/node-troubleshooting
    @dschenkelman
    npm i dschenkelman
    

    View Slide