Slide 1

Slide 1 text

design patterns & modularity in the npm registry

Slide 2

Slide 2 text

C J Silverio, CTO @ceejbot

Slide 3

Slide 3 text

human brains are pattern-detection machines

Slide 4

Slide 4 text

the process of writing software is abstraction & pattern extraction

Slide 5

Slide 5 text

patterns in code patterns in systems

Slide 6

Slide 6 text

emergent patterns in npm's registry

Slide 7

Slide 7 text

Let's analyze them not just for how they scale but how they promote modularity

Slide 8

Slide 8 text

registry?

Slide 9

Slide 9 text

registry, n: the system of services that manage package tar archives + metadata

Slide 10

Slide 10 text

361,263 packages 2,278,817 million tarballs medium data (fits on 1 disk)

Slide 11

Slide 11 text

npm's largest engineering project & its most obvious scaling challenge

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

monoliths microservices transaction logs message queues

Slide 14

Slide 14 text

monoliths: everything in one big process

Slide 15

Slide 15 text

monoliths are okay easy to write & change perf more than good enough

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

when you scale perf & team size monoliths are less okay

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

it's easy to write highly-coupled code inside a non-modular monolith

Slide 20

Slide 20 text

modularity: let's be less vague

Slide 21

Slide 21 text

Q: where does modularity come from? A: information hiding

Slide 22

Slide 22 text

"On the Criteria To Be Used in Decomposing Systems into Modules" — D. L. Parnas, 1972

Slide 23

Slide 23 text

information data, its structures, the algorithms

Slide 24

Slide 24 text

hide info behind an interface so you can change it

Slide 25

Slide 25 text

No content

Slide 26

Slide 26 text

the hot trend is rewriting monoliths as microservices

Slide 27

Slide 27 text

you're forced to design an API & put implementation inside a service

Slide 28

Slide 28 text

microservices can still mess up modularity

Slide 29

Slide 29 text

can scatter a task across services making retries & failure hard to cope with

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

No content

Slide 32

Slide 32 text

auth sets up package access on a publish as a side effect

Slide 33

Slide 33 text

what happens if a service crashes? or if validation rejects a publish?

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

after publication, it's a different pattern: the transaction log

Slide 36

Slide 36 text

transaction log write-ahead log (WAL) commit log

Slide 37

Slide 37 text

The Log: What every software engineer should know about real-time data's unifying abstraction

Slide 38

Slide 38 text

couchdb's super power the changes feed

Slide 39

Slide 39 text

registry followers: consumers of couchdb's commit logs

Slide 40

Slide 40 text

No content

Slide 41

Slide 41 text

→ distribute tarballs → invalidate our CDN's cache → populate postgresdb to drive the website → index data in ElasticSearch → scan packages for security leaks → populate our registry mirror → "re webhooks

Slide 42

Slide 42 text

each log consumer does one thing well

Slide 43

Slide 43 text

No content

Slide 44

Slide 44 text

Estragon: Let's fix publication. Vladimir: Fine. But how?

Slide 45

Slide 45 text

No content

Slide 46

Slide 46 text

message queues

Slide 47

Slide 47 text

message queues inversion of control

Slide 48

Slide 48 text

workers consume messages & retry or unwind on failure

Slide 49

Slide 49 text

a worker does one thing puts a new message back into the queue

Slide 50

Slide 50 text

you scale by adding more workers

Slide 51

Slide 51 text

queue has to be reliable workers can crash

Slide 52

Slide 52 text

http://queues.io

Slide 53

Slide 53 text

No content

Slide 54

Slide 54 text

queue disadvantages? we don't have them in production so ∞ disadvantages!

Slide 55

Slide 55 text

No content

Slide 56

Slide 56 text

what's the pattern that emerges from this discussion of patterns?

Slide 57

Slide 57 text

there is no silver bullet

Slide 58

Slide 58 text

it's tradeoffs all the way down

Slide 59

Slide 59 text

what problem are you solving? what tools do you have to hand? what is your team experienced with?

Slide 60

Slide 60 text

you'll need to fight for modularity no matter what you pick

Slide 61

Slide 61 text

make your users happy first because that's the hard part

Slide 62

Slide 62 text

but know that you can change your systems

Slide 63

Slide 63 text

we'll be changing ours check back next year to hear how it turned out

Slide 64

Slide 64 text

No content