Architect at Scale - My musings as a Node.js developer

When faced with the inevitable question of choosing a framework, people often times go for the most popular and hardly do the research on why others may serve their purpose. Deployment is another issue; microservices? Monilithic? Docker? What about testing? And security? They are a lot of questions. With this conversation (rather than a "talk"), I will try to justify the good and the bad use-case of each of the technologies.

By the end of this talk, the attendees should have a nice grasp on the pros and the cons of some of the most popular libraries, design patterns and mechanisms in the Node.js/JS world and when should they use which.

Transcript

1. musings of the a frustrated developer
2. i am shreyansh; and I write code, (obviously) i really don’t know what to put here… /labsvisual /acodingpanda /acodingpanda
3. Why this talk?
4. because no one can write bug free code
5. however, there is a way. }1. sudo rm -rf / 2. burn your computer 3. live as a monk
6. to err is to human, to err in prod is JS
7. i forgot how memory works i was infatuated by JSON my crypto degree failed me logging > response config explosion express ftw
8. episode 1 the one where i* forgot how memory works
9. at 2:57 AM, i got this
10. what the…?!
11. it restarted. no biggie. > 15 nodes
12. every 20 minutes i got a notification of a failing node restart
13. first thought?
14. after 10 minutes of solid contemplation i decided to…
15. finish rick and morty
18. Who can debug this?
20. There is coredump
21. mdb?
22. Node debug?
23. What about perfmon?
24. Are you lazy? “Back in my day we used to read memory fragments directly from the RAM.”
25. So what did you choose?
26. Flamegraphs!
27. flamegraphs 101 stack frame (y-axis) ? no meaning (x-axis) box.width = time on cpu
28. our flamegraph analyzeFile() processFile() Controller call
29. // routers/classifier/index.js // ... classifierRouter.post( '/', [ ...Controller.middlewares.common, ], Controller.sendForAnalysis ); // ...
30. // classifiers/ruben/index.js // At the top of the file const parsed = {}; // Parse the PDF file into a list of strings // Go through these strings one by one for ( let i = 0; i < splitLines.length; ++i ) { const currentLine = splitLines[ i ]; parsed[ getHash( fileName ) ][ i ] = analyze( currentLine ); } // ...
31. ++i? pre-increment undefined leak
32. Monitor? How?
33. Flamegraph? How?
34. Create heapdump. Throw it in DevTools.
35. Create it in Prod. Lol.
36. // ... process.on( 'SIGUSR2', function generateHeapDump() { // heapdump package } ); kill -s SIGUSR2 <PID>
38. Re-start the process before it locks you out
39. Minimal? // ... const { rss, heapTotal } = process.memoryUsage(); Total memory by all packages. Total memory for objects, etc.
40. episode 2 the one where i logged more than i responded
41. Response Latency
42. // ... baseRouter.get( '/', function handleBaseRoute( req, res, next ) { // Fetch something from Mongo // Validate it // Send it back res.status( 200 ).json( Helpers.convertToJSONApi( ... ) ); }
44. Winston, anyone?
45. Average Latency 400 ms/req 328 ms/req logs
46. What is the point of logging? Data Payload Stack Replay
47. Why it’s an overkill.
48. We found out that Winston is slow as…
49. Why? Because it also processes these logs through transports.
50. Application should only be responsible for sending logs.
51. Write logs to STDOUT Pipe it to another process Achieve nirvana
52. Control verbosity
53. What about debugging? Logs FTW!
54. Controlling verbosity. in prod; in real-time.
55. Shreyansh, How will you interface this? yours sincerely, everyone.
56. Management routes POST :/management/log POST :/management/_health
57. Great. Okay, but this does not scale.
58. Wish I had a… some method to automate this. (automate any task which takes > 10 seconds)
59. Introducing Consul
60. Reactive Config
61. // index.js process.on( 'SIGHUP', function handleHangup() { ConfigStore.reloadConfig(); } );
62. // config/config-store.js const { EventEmitter } = require( 'events' ); const ConfigStore = Object.create( EventEmitter.prototype ); ConfigStore.reloadConfig = function reloadConfig() { /* Some logic to fetch the latest changes from the Consul * service. */ delete require.cache[ require.resolve( ... ) ]; this.config = require( `./${ environment }.json` ); this.emit( 'config-change' ); } module.exports = ConfigStore;o
63. Benefits.
64. episode 3 the one where my crypto knowledge failed me
65. legacy system had insecure auth code
66. function login(){ $username = $this->request->data['username']; $password = $this->request->data['password']; $admins = TableRegistry::get('Admins'); $row = $admins ->find() ->where(['username' => $username]) ->where(['password' => md5($password)]) ->first(); ...
67. the bigger problem Most of the credential storage tutorials don’t cover password reset… which is a worry.
68. Usual Flow Request Email Magic
69. Predictable Tokens Attacker can figure out the token, generate one and replay the password reset.
70. Non-volatile tokens (do not expire after use)
71. Better way.
72. HMAC a random string with the user’s current password
73. No need to delete. Upon successful reset, the password in the db changes making the token invalid.
74. Why did I bring this up?
75. y u do dis?!
82. Questions? https://github.com/labsvisual https://linkedin.com/in/acodingpanda/ https://isomr.co/
83. We’re hiring, btw!
84. Very aggressively. If you want to see real aggression, look at Recro’s HR effort. - Shreyansh Pandey