Design Patterns & Modularity in the npm Registry

A discussion of some distributed systems design patterns and how they support modularity or fight against it, as learned from trying to make the npm registry work.

Transcript

1. design patterns & modularity in the npm registry
2. C J Silverio, CTO @ceejbot
3. human brains are pattern-detection machines
4. the process of writing software is abstraction & pattern extraction
5. patterns in code patterns in systems
6. emergent patterns in npm's registry
7. Let's analyze them not just for how they scale but how they promote modularity
8. registry?
9. registry, n: the system of services that manage package tar archives + metadata
10. 361,263 packages 2,278,817 million tarballs medium data (fits on 1 disk)
11. npm's largest engineering project & its most obvious scaling challenge
13. monoliths microservices transaction logs message queues
14. monoliths: everything in one big process
15. monoliths are okay easy to write & change perf more than good enough
17. when you scale perf & team size monoliths are less okay
19. it's easy to write highly-coupled code inside a non-modular monolith
20. modularity: let's be less vague
21. Q: where does modularity come from? A: information hiding
22. "On the Criteria To Be Used in Decomposing Systems into Modules" — D. L. Parnas, 1972
23. information data, its structures, the algorithms
24. hide info behind an interface so you can change it
26. the hot trend is rewriting monoliths as microservices
27. you're forced to design an API & put implementation inside a service
28. microservices can still mess up modularity
29. can scatter a task across services making retries & failure hard to cope with
32. auth sets up package access on a publish as a side effect
33. what happens if a service crashes? or if validation rejects a publish?
35. after publication, it's a different pattern: the transaction log
36. transaction log write-ahead log (WAL) commit log
37. The Log: What every software engineer should know about real-time data's unifying abstraction
38. couchdb's super power the changes feed
39. registry followers: consumers of couchdb's commit logs
41. → distribute tarballs → invalidate our CDN's cache → populate postgresdb to drive the website → index data in ElasticSearch → scan packages for security leaks → populate our registry mirror → "re webhooks
42. each log consumer does one thing well
44. Estragon: Let's fix publication. Vladimir: Fine. But how?
46. message queues
47. message queues inversion of control
48. workers consume messages & retry or unwind on failure
49. a worker does one thing puts a new message back into the queue
50. you scale by adding more workers
51. queue has to be reliable workers can crash
52. http://queues.io
54. queue disadvantages? we don't have them in production so ∞ disadvantages!
56. what's the pattern that emerges from this discussion of patterns?
57. there is no silver bullet
58. it's tradeoffs all the way down
59. what problem are you solving? what tools do you have to hand? what is your team experienced with?
60. you'll need to fight for modularity no matter what you pick
61. make your users happy first because that's the hard part
62. but know that you can change your systems
63. we'll be changing ours check back next year to hear how it turned out