Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Stabilizing the npm registry
Search
C J Silverio
February 11, 2015
Programming
2
330
Stabilizing the npm registry
How npm went from a car fire to a very boring service with a very boring uptime.
C J Silverio
February 11, 2015
Tweet
Share
More Decks by C J Silverio
See All by C J Silverio
The economics of package management
ceejbot
4
1.4k
The future of (javascript) modules (in node)
ceejbot
1
210
Keeping JavaScript safe
ceejbot
3
370
ceej's how to solve it
ceejbot
6
730
work-life balance at npm
ceejbot
5
760
hash functions and you!
ceejbot
2
340
The accidental noder
ceejbot
2
130
Design Patterns & Modularity in the npm Registry
ceejbot
3
170
Monitoring on a budget
ceejbot
2
270
Other Decks in Programming
See All in Programming
プロンプトエンジニアリング入門
tomokusaba
2
980
App Router への移行は「改善」となり得るのか?/ Can migration to App Router be an improvement
takefumiyoshii
1
130
RubyVM を PHP で実装する 〜Hello World を出力するまで〜
memory1994
PRO
1
490
TDDと今まで
kanayannet
0
140
WebComponentsで フレームワークを1ページに共存させる
webuilder240
0
150
Migrating to Signals: A Practical Workshop
manfredsteyer
PRO
0
280
ファイル先頭の use の意味、説明できますか? 〜PHP の namespace と autoloading の関係を正しく理解しよう〜 / namespace and autoloading in php
okashoi
2
470
ISUCONってなんだか難しそう……!!でも、初めてのISUCONにPHPで挑戦してきました!
kotomin_m
1
300
Deep Dive 大規模システムアーキテクチャ/開発組織エンジニアリング / Deep Dive Large-Scale System Architecture, Development Organization Engineering
nrslib
15
2.9k
Why 1 + 1 = 2 in Swift?
1plus4
1
250
せっかくモデル図描くのなら、嬉しいことが多い方がいいよね!
kuboaki
1
1.8k
IntelliJ IDEA を知らなかった 自分に教えたい小ネタ集 / IntelliJ IDEA Hints for My Past Self
mackey0225
3
180
Featured
See All Featured
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
12
1.4k
Faster Mobile Websites
deanohume
296
30k
Scaling GitHub
holman
456
140k
Fantastic passwords and where to find them - at NoRuKo
philnash
35
2.4k
KATA
mclloyd
14
11k
Designing the Hi-DPI Web
ddemaree
275
33k
It's Worth the Effort
3n
180
27k
Imperfection Machines: The Place of Print at Facebook
scottboms
257
12k
10 Git Anti Patterns You Should be Aware of
lemiorhan
644
57k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
6
950
Large-scale JavaScript Application Architecture
addyosmani
501
110k
For a Future-Friendly Web
brad_frost
170
8.9k
Transcript
stabilizing the registry
C J Silverio director of engineering, npm @ceejbot
This is the story of a plucky package registry named
npm
scaling problem manifesting itself as a stability problem
"scaling" capacity to meet growing demands
"At scale" huge demand & lots of data
"stability" not falling over under normal demand
What's normal demand?
129K packages 239 GB package tarballs 40 million pkg dls/day
1500 req/sec, peak 3200
"Legacy" Anything you've put into production
this is the story of a legacy system becoming more
flexible
None
January 2013 20K packages .5 million dls/day
Oct 2013 44K packages 108 million dls/month 3.6 million dls/day
None
our plucky little registry had to change
step 1: CDN Put Fastly.com in front of the registry
cache rules everything around me
step 2: tarballs get them out of couchdb
tarballs are huge! couch runs better without them base64 decoding
is work.
None
January 2014 60K packages 6+ million dls/day
step 3: visibility are things going wrong? what's going wrong?
reactive monitoring monitor deeply fix things quickly
proactive monitoring self-healing (also things don't break)
monitoring is unit testing Add monitoring after every outage
visibility is a prerequisite but not a solution
act on what monitoring and metrics reveal
step 4: redundancy several CouchDBs! reads, writes, & replication
fewer responsibilities for each piece isolates errors
step 5: automation ansible no server is special
June 2014 Superficially similar.
June 2014 80K packages 10 million dls/day
step 6: simplification now that it's not on fire we
can modify at leisure
None
Nov 2014 105K packages 28 million dls/day peak
50/50 AWS region split no AWS-specific magic Ubuntu 14.04 Trusty
Fastly: geoloc + varnish haproxy + CouchDB nginx + a
filesystem
where's the node?
registry 2 electric boogaloo with 500% more node
None
haproxy + node services couchdb ➜ postgres redis for caching
nginx + filesystem
more complicated more flexible & redundant more scaling dials to
turn
excited about postgres ad-hoc queries are fun
scaling node is exactly like scaling everything else
Understand system get visibility cool down hot spots add redundancy
npm client <3 npm install -g npm@latest
npm loves you