Tweet
Tweet

Slide 1

Slide 1 text

musings of the a frustrated developer

Slide 2

Slide 2 text

i am shreyansh; and I write code, (obviously) i really don’t know what to put here… /labsvisual /acodingpanda /acodingpanda

Slide 3

Slide 3 text

Why this talk?

Slide 4

Slide 4 text

because no one can write bug free code

Slide 5

Slide 5 text

however, there is a way. }1. sudo rm -rf / 2. burn your computer 3. live as a monk

Slide 6

Slide 6 text

to err is to human, to err in prod is JS

Slide 7

Slide 7 text

i forgot how memory works i was infatuated by JSON my crypto degree failed me logging > response config explosion express ftw

Slide 8

Slide 8 text

episode 1 the one where i* forgot how memory works

Slide 9

Slide 9 text

at 2:57 AM, i got this

Slide 10

Slide 10 text

what the…?!

Slide 11

Slide 11 text

it restarted. no biggie. > 15 nodes

Slide 12

Slide 12 text

every 20 minutes i got a notification of a failing node restart

Slide 13

Slide 13 text

first thought?

Slide 14

Slide 14 text

after 10 minutes of solid contemplation i decided to…

Slide 15

Slide 15 text

finish rick and morty

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

No content

Slide 18

Slide 18 text

Who can debug this?

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

There is coredump

Slide 21

Slide 21 text

mdb?

Slide 22

Slide 22 text

Node debug?

Slide 23

Slide 23 text

What about perfmon?

Slide 24

Slide 24 text

Are you lazy? “Back in my day we used to read memory fragments directly from the RAM.”

Slide 25

Slide 25 text

So what did you choose?

Slide 26

Slide 26 text

Flamegraphs!

Slide 27

Slide 27 text

flamegraphs 101 stack frame (y-axis) ? no meaning (x-axis) box.width = time on cpu

Slide 28

Slide 28 text

our flamegraph analyzeFile() processFile() Controller call

Slide 29

Slide 29 text

// routers/classifier/index.js // ... classifierRouter.post( '/', [ ...Controller.middlewares.common, ], Controller.sendForAnalysis ); // ...

Slide 30

Slide 30 text

// classifiers/ruben/index.js // At the top of the file const parsed = {}; // Parse the PDF file into a list of strings // Go through these strings one by one for ( let i = 0; i < splitLines.length; ++i ) { const currentLine = splitLines[ i ]; parsed[ getHash( fileName ) ][ i ] = analyze( currentLine ); } // ...

Slide 31

Slide 31 text

++i? pre-increment undefined leak

Slide 32

Slide 32 text

Monitor? How?

Slide 33

Slide 33 text

Flamegraph? How?

Slide 34

Slide 34 text

Create heapdump. Throw it in DevTools.

Slide 35

Slide 35 text

Create it in Prod. Lol.

Slide 36

Slide 36 text

// ... process.on( 'SIGUSR2', function generateHeapDump() { // heapdump package } ); kill -s SIGUSR2

Slide 37

Slide 37 text

No content

Slide 38

Slide 38 text

Re-start the process before it locks you out

Slide 39

Slide 39 text

Minimal? // ... const { rss, heapTotal } = process.memoryUsage(); Total memory by all packages. Total memory for objects, etc.

Slide 40

Slide 40 text

episode 2 the one where i logged more than i responded

Slide 41

Slide 41 text

Response Latency

Slide 42

Slide 42 text

// ... baseRouter.get( '/', function handleBaseRoute( req, res, next ) { // Fetch something from Mongo // Validate it // Send it back res.status( 200 ).json( Helpers.convertToJSONApi( ... ) ); }

Slide 43

Slide 43 text

No content

Slide 44

Slide 44 text

Winston, anyone?

Slide 45

Slide 45 text

Average Latency 400 ms/req 328 ms/req logs

Slide 46

Slide 46 text

What is the point of logging? Data Payload Stack Replay

Slide 47

Slide 47 text

Why it’s an overkill.

Slide 48

Slide 48 text

We found out that Winston is slow as…

Slide 49

Slide 49 text

Why? Because it also processes these logs through transports.

Slide 50

Slide 50 text

Application should only be responsible for sending logs.

Slide 51

Slide 51 text

Write logs to STDOUT Pipe it to another process Achieve nirvana

Slide 52

Slide 52 text

Control verbosity

Slide 53

Slide 53 text

What about debugging? Logs FTW!

Slide 54

Slide 54 text

Controlling verbosity. in prod; in real-time.

Slide 55

Slide 55 text

Shreyansh, How will you interface this? yours sincerely, everyone.

Slide 56

Slide 56 text

Management routes POST :/management/log POST :/management/_health

Slide 57

Slide 57 text

Great. Okay, but this does not scale.

Slide 58

Slide 58 text

Wish I had a… some method to automate this. (automate any task which takes > 10 seconds)

Slide 59

Slide 59 text

Introducing Consul

Slide 60

Slide 60 text

Reactive Config

Slide 61

Slide 61 text

// index.js process.on( 'SIGHUP', function handleHangup() { ConfigStore.reloadConfig(); } );

Slide 62

Slide 62 text

// config/config-store.js const { EventEmitter } = require( 'events' ); const ConfigStore = Object.create( EventEmitter.prototype ); ConfigStore.reloadConfig = function reloadConfig() { /* Some logic to fetch the latest changes from the Consul * service. */ delete require.cache[ require.resolve( ... ) ]; this.config = require( `./${ environment }.json` ); this.emit( 'config-change' ); } module.exports = ConfigStore;o

Slide 63

Slide 63 text

Benefits.

Slide 64

Slide 64 text

episode 3 the one where my crypto knowledge failed me

Slide 65

Slide 65 text

legacy system had insecure auth code

Slide 66

Slide 66 text

function login(){ $username = $this->request->data['username']; $password = $this->request->data['password']; $admins = TableRegistry::get('Admins'); $row = $admins ->find() ->where(['username' => $username]) ->where(['password' => md5($password)]) ->first(); ...

Slide 67

Slide 67 text

the bigger problem Most of the credential storage tutorials don’t cover password reset… which is a worry.

Slide 68

Slide 68 text

Usual Flow Request Email Magic

Slide 69

Slide 69 text

Predictable Tokens Attacker can figure out the token, generate one and replay the password reset.

Slide 70

Slide 70 text

Non-volatile tokens (do not expire after use)

Slide 71

Slide 71 text

Better way.

Slide 72

Slide 72 text

HMAC a random string with the user’s current password

Slide 73

Slide 73 text

No need to delete. Upon successful reset, the password in the db changes making the token invalid.

Slide 74

Slide 74 text

Why did I bring this up?

Slide 75

Slide 75 text

y u do dis?!

Slide 76

Slide 76 text

No content

Slide 77

Slide 77 text

No content

Slide 78

Slide 78 text

No content

Slide 79

Slide 79 text

No content

Slide 80

Slide 80 text

No content

Slide 81

Slide 81 text

No content

Slide 82

Slide 82 text

Questions? https://github.com/labsvisual https://linkedin.com/in/acodingpanda/ https://isomr.co/

Slide 83

Slide 83 text

We’re hiring, btw!

Slide 84

Slide 84 text

Very aggressively. If you want to see real aggression, look at Recro’s HR effort. - Shreyansh Pandey