Link
Embed
Share
Beginning
This slide
Copy link URL
Copy link URL
Copy iframe embed code
Copy iframe embed code
Copy javascript embed code
Copy javascript embed code
Share
Tweet
Share
Tweet
Slide 1
Slide 1 text
design patterns & modularity in the npm registry
Slide 2
Slide 2 text
C J Silverio, CTO @ceejbot
Slide 3
Slide 3 text
human brains are pattern-detection machines
Slide 4
Slide 4 text
the process of writing software is abstraction & pattern extraction
Slide 5
Slide 5 text
patterns in code patterns in systems
Slide 6
Slide 6 text
emergent patterns in npm's registry
Slide 7
Slide 7 text
Let's analyze them not just for how they scale but how they promote modularity
Slide 8
Slide 8 text
registry?
Slide 9
Slide 9 text
registry, n: the system of services that manage package tar archives + metadata
Slide 10
Slide 10 text
361,263 packages 2,278,817 million tarballs medium data (fits on 1 disk)
Slide 11
Slide 11 text
npm's largest engineering project & its most obvious scaling challenge
Slide 12
Slide 12 text
No content
Slide 13
Slide 13 text
monoliths microservices transaction logs message queues
Slide 14
Slide 14 text
monoliths: everything in one big process
Slide 15
Slide 15 text
monoliths are okay easy to write & change perf more than good enough
Slide 16
Slide 16 text
No content
Slide 17
Slide 17 text
when you scale perf & team size monoliths are less okay
Slide 18
Slide 18 text
No content
Slide 19
Slide 19 text
it's easy to write highly-coupled code inside a non-modular monolith
Slide 20
Slide 20 text
modularity: let's be less vague
Slide 21
Slide 21 text
Q: where does modularity come from? A: information hiding
Slide 22
Slide 22 text
"On the Criteria To Be Used in Decomposing Systems into Modules" — D. L. Parnas, 1972
Slide 23
Slide 23 text
information data, its structures, the algorithms
Slide 24
Slide 24 text
hide info behind an interface so you can change it
Slide 25
Slide 25 text
No content
Slide 26
Slide 26 text
the hot trend is rewriting monoliths as microservices
Slide 27
Slide 27 text
you're forced to design an API & put implementation inside a service
Slide 28
Slide 28 text
microservices can still mess up modularity
Slide 29
Slide 29 text
can scatter a task across services making retries & failure hard to cope with
Slide 30
Slide 30 text
No content
Slide 31
Slide 31 text
No content
Slide 32
Slide 32 text
auth sets up package access on a publish as a side effect
Slide 33
Slide 33 text
what happens if a service crashes? or if validation rejects a publish?
Slide 34
Slide 34 text
No content
Slide 35
Slide 35 text
after publication, it's a different pattern: the transaction log
Slide 36
Slide 36 text
transaction log write-ahead log (WAL) commit log
Slide 37
Slide 37 text
The Log: What every software engineer should know about real-time data's unifying abstraction
Slide 38
Slide 38 text
couchdb's super power the changes feed
Slide 39
Slide 39 text
registry followers: consumers of couchdb's commit logs
Slide 40
Slide 40 text
No content
Slide 41
Slide 41 text
→ distribute tarballs → invalidate our CDN's cache → populate postgresdb to drive the website → index data in ElasticSearch → scan packages for security leaks → populate our registry mirror → "re webhooks
Slide 42
Slide 42 text
each log consumer does one thing well
Slide 43
Slide 43 text
No content
Slide 44
Slide 44 text
Estragon: Let's fix publication. Vladimir: Fine. But how?
Slide 45
Slide 45 text
No content
Slide 46
Slide 46 text
message queues
Slide 47
Slide 47 text
message queues inversion of control
Slide 48
Slide 48 text
workers consume messages & retry or unwind on failure
Slide 49
Slide 49 text
a worker does one thing puts a new message back into the queue
Slide 50
Slide 50 text
you scale by adding more workers
Slide 51
Slide 51 text
queue has to be reliable workers can crash
Slide 52
Slide 52 text
http://queues.io
Slide 53
Slide 53 text
No content
Slide 54
Slide 54 text
queue disadvantages? we don't have them in production so ∞ disadvantages!
Slide 55
Slide 55 text
No content
Slide 56
Slide 56 text
what's the pattern that emerges from this discussion of patterns?
Slide 57
Slide 57 text
there is no silver bullet
Slide 58
Slide 58 text
it's tradeoffs all the way down
Slide 59
Slide 59 text
what problem are you solving? what tools do you have to hand? what is your team experienced with?
Slide 60
Slide 60 text
you'll need to fight for modularity no matter what you pick
Slide 61
Slide 61 text
make your users happy first because that's the hard part
Slide 62
Slide 62 text
but know that you can change your systems
Slide 63
Slide 63 text
we'll be changing ours check back next year to hear how it turned out
Slide 64
Slide 64 text
No content