After Years of Node in Prod, We Still Block the Event Loop

Transcript

1. After Years of Node in Prod, We Still Block the Event Loop Yamil Asusta
2. /usr/bin/whoami • I do stuff for Auth0 • Been using Node.js since ~2011 • From Puerto Rico , fluent in Spanglish • @elbuo8
3. What is the event loop again? • Pipeline of operations ◦ Timers: setTimeout/setInterval ◦ I/O: most of your callbacks ◦ Poll: looks for new events to schedule (ej. new HTTP connection) ◦ Check: setImmediate ◦ Close: server.on(‘close’, cb) • Runs our JS code and delegates operations to give us “async”
4. Why should we event loop? “@coreinfra: customer reported had another spike of failed requests (502 and timeouts) at 13:00 UTC”
5. How can we monitor it? • npm install blocked (new npm --save by default ) • Hard to find which part of the code actually caused the delay • Complements: graphs (cpu), heapdumps (memory) & custom metrics (apm)
6. Bad Case: Lazy require • require is a blocking operation • Invoked mostly when starting the process • Avoid require outside the initialization logic
7. Bad Case: Lazy require • 58 directories, 2166 files
8. Bad Case: JSON parse
9. Bad Case: JSON parse • Complexity (nested levels) are proportional to time • Used for serializing communication across services • Enforce limits that prevent unexpected payloads
10. Bad Case: Garbage Collection • “max_old_space_size” helps memory leaks ( yenkel) • “MarkSweepCompact” gets slower as heap increases • GC operations are less frequent and have higher latency
13. Bad Case: Arbitrary Size Data Structures • Add “request.id” in RateLimitMap when new request is received • Listen for “onPreResponse” to be triggered • Remove “request.id” from RateLimitMap when request is processed
14. Bad Case: Arbitrary Size Data Structures
16. Bad Case: npm install innocent-package • We install after we determine a module is “good” • Reviewers often trust a selected module ☑ • Once merged, the module is “trusted” by the whole organization
17. Bad Case: npm install innocent-package
18. Bad Case: npm install innocent-package
19. Bad Case: npm install innocent-package
20. Bad Case: npm install innocent-package
25. Special Thanks • The real hero of this talk is @jfroma from Cordoba
26. References 1. https://nodejs.org/en/docs/guides/event-loop-timers-and-nexttick/ 2. https://medium.com/the-node-js-collection/what-you-should-know-to-really-und erstand-the-node-js-event-loop-and-its-metrics-c4907b19da4c 3. https://github.com/dschenkelman/patova/pull/31/files 4. https://github.com/zaggino/z-schema/pull/187/files
27. Thanks!