Pro Yearly is on sale from $80 to $50! »

Architect at Scale - My musings as a Node.js developer

Architect at Scale - My musings as a Node.js developer

When faced with the inevitable question of choosing a framework, people often times go for the most popular and hardly do the research on why others may serve their purpose. Deployment is another issue; microservices? Monilithic? Docker? What about testing? And security? They are a lot of questions. With this conversation (rather than a "talk"), I will try to justify the good and the bad use-case of each of the technologies.

By the end of this talk, the attendees should have a nice grasp on the pros and the cons of some of the most popular libraries, design patterns and mechanisms in the Node.js/JS world and when should they use which.

D02cb020cc161b9d0bb16d1c041a2cc5?s=128

Shreyansh Pandey

March 16, 2019
Tweet

Transcript

  1. musings of the a frustrated developer

  2. i am shreyansh; and I write code, (obviously) i really

    don’t know what to put here… /labsvisual /acodingpanda /acodingpanda
  3. Why this talk?

  4. because no one can write bug free code

  5. however, there is a way. }1. sudo rm -rf /

    2. burn your computer 3. live as a monk
  6. to err is to human, to err in prod is

    JS
  7. i forgot how memory works i was infatuated by JSON

    my crypto degree failed me logging > response config explosion express ftw
  8. episode 1 the one where i* forgot how memory works

  9. at 2:57 AM, i got this

  10. what the…?!

  11. it restarted. no biggie. > 15 nodes

  12. every 20 minutes i got a notification of a failing

    node restart
  13. first thought?

  14. after 10 minutes of solid contemplation i decided to…

  15. finish rick and morty

  16. None
  17. None
  18. Who can debug this?

  19. None
  20. There is coredump

  21. mdb?

  22. Node debug?

  23. What about perfmon?

  24. Are you lazy? “Back in my day we used to

    read memory fragments directly from the RAM.”
  25. So what did you choose?

  26. Flamegraphs!

  27. flamegraphs 101 stack frame (y-axis) ? no meaning (x-axis) box.width

    = time on cpu
  28. our flamegraph analyzeFile() processFile() Controller call

  29. // routers/classifier/index.js // ... classifierRouter.post( '/', [ ...Controller.middlewares.common, ], Controller.sendForAnalysis

    ); // ...
  30. // classifiers/ruben/index.js // At the top of the file const

    parsed = {}; // Parse the PDF file into a list of strings // Go through these strings one by one for ( let i = 0; i < splitLines.length; ++i ) { const currentLine = splitLines[ i ]; parsed[ getHash( fileName ) ][ i ] = analyze( currentLine ); } // ...
  31. ++i? pre-increment undefined leak

  32. Monitor? How?

  33. Flamegraph? How?

  34. Create heapdump. Throw it in DevTools.

  35. Create it in Prod. Lol.

  36. // ... process.on( 'SIGUSR2', function generateHeapDump() { // heapdump package

    } ); kill -s SIGUSR2 <PID>
  37. None
  38. Re-start the process before it locks you out

  39. Minimal? // ... const { rss, heapTotal } = process.memoryUsage();

    Total memory by all packages. Total memory for objects, etc.
  40. episode 2 the one where i logged more than i

    responded
  41. Response Latency

  42. // ... baseRouter.get( '/', function handleBaseRoute( req, res, next )

    { // Fetch something from Mongo // Validate it // Send it back res.status( 200 ).json( Helpers.convertToJSONApi( ... ) ); }
  43. None
  44. Winston, anyone?

  45. Average Latency 400 ms/req 328 ms/req logs

  46. What is the point of logging? Data Payload Stack Replay

  47. Why it’s an overkill.

  48. We found out that Winston is slow as…

  49. Why? Because it also processes these logs through transports.

  50. Application should only be responsible for sending logs.

  51. Write logs to STDOUT Pipe it to another process Achieve

    nirvana
  52. Control verbosity

  53. What about debugging? Logs FTW!

  54. Controlling verbosity. in prod; in real-time.

  55. Shreyansh, How will you interface this? yours sincerely, everyone.

  56. Management routes POST :/management/log POST :/management/_health

  57. Great. Okay, but this does not scale.

  58. Wish I had a… some method to automate this. (automate

    any task which takes > 10 seconds)
  59. Introducing Consul

  60. Reactive Config

  61. // index.js process.on( 'SIGHUP', function handleHangup() { ConfigStore.reloadConfig(); } );

  62. // config/config-store.js const { EventEmitter } = require( 'events' );

    const ConfigStore = Object.create( EventEmitter.prototype ); ConfigStore.reloadConfig = function reloadConfig() { /* Some logic to fetch the latest changes from the Consul * service. */ delete require.cache[ require.resolve( ... ) ]; this.config = require( `./${ environment }.json` ); this.emit( 'config-change' ); } module.exports = ConfigStore;o
  63. Benefits.

  64. episode 3 the one where my crypto knowledge failed me

  65. legacy system had insecure auth code

  66. function login(){ $username = $this->request->data['username']; $password = $this->request->data['password']; $admins =

    TableRegistry::get('Admins'); $row = $admins ->find() ->where(['username' => $username]) ->where(['password' => md5($password)]) ->first(); ...
  67. the bigger problem Most of the credential storage tutorials don’t

    cover password reset… which is a worry.
  68. Usual Flow Request Email Magic

  69. Predictable Tokens Attacker can figure out the token, generate one

    and replay the password reset.
  70. Non-volatile tokens (do not expire after use)

  71. Better way.

  72. HMAC a random string with the user’s current password

  73. No need to delete. Upon successful reset, the password in

    the db changes making the token invalid.
  74. Why did I bring this up?

  75. y u do dis?!

  76. None
  77. None
  78. None
  79. None
  80. None
  81. None
  82. Questions? https://github.com/labsvisual https://linkedin.com/in/acodingpanda/ https://isomr.co/

  83. We’re hiring, btw!

  84. Very aggressively. If you want to see real aggression, look

    at Recro’s HR effort. - Shreyansh Pandey