Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Stabilizing the npm registry
Search
C J Silverio
February 11, 2015
Programming
2
330
Stabilizing the npm registry
How npm went from a car fire to a very boring service with a very boring uptime.
C J Silverio
February 11, 2015
Tweet
Share
More Decks by C J Silverio
See All by C J Silverio
The economics of package management
ceejbot
4
1.4k
The future of (javascript) modules (in node)
ceejbot
1
220
Keeping JavaScript safe
ceejbot
3
380
ceej's how to solve it
ceejbot
6
740
work-life balance at npm
ceejbot
5
760
hash functions and you!
ceejbot
2
340
The accidental noder
ceejbot
2
130
Design Patterns & Modularity in the npm Registry
ceejbot
3
170
Monitoring on a budget
ceejbot
2
270
Other Decks in Programming
See All in Programming
RubyGems on ruby.wasm
kateinoigakukun
0
110
Using "modern" Ruby to build a better, faster Homebrew
mikemcquaid
2
240
欠陥を早期に発見するための Software Engineer in Test とその重要性 / What is Software Engineer in Test and How they works
orgachem
PRO
17
2.3k
GitLab CI/CD で C#/WPFアプリケーションのテストとインストーラーのビルド・デプロイを自動化する
hacarus
0
610
Findy - エンジニア向け会社紹介 / Findy Letter for Engineers
findyinc
2
74k
Introducing Kotlin Multiplatform in an existing mobile app - Workshop Edition | AndroidMakers Paris
prof18
0
180
Embedding it into Ruby code
soutaro
1
300
Deep Dive into React Stream/Serialize
mugi_uno
4
860
An adventure of Happy Eyeballs
coe401_
1
140
Let's learn code review
riofujimon
2
640
WinActorの勉強を継続する方法
tamai_63
0
130
Documentation testsの恩恵 / Documentation testing benefits
ssssota
1
560
Featured
See All Featured
The Power of CSS Pseudo Elements
geoffreycrofte
62
5k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
82
45k
Fashionably flexible responsive web design (full day workshop)
malarkey
398
65k
How to name files
jennybc
65
94k
Web Components: a chance to create the future
zenorocha
306
41k
Bash Introduction
62gerente
605
210k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
1
120
Atom: Resistance is Futile
akmur
260
25k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
22
1.4k
Robots, Beer and Maslow
schacon
PRO
155
8k
Designing for humans not robots
tammielis
247
25k
No one is an island. Learnings from fostering a developers community.
thoeni
16
2.1k
Transcript
stabilizing the registry
C J Silverio director of engineering, npm @ceejbot
This is the story of a plucky package registry named
npm
scaling problem manifesting itself as a stability problem
"scaling" capacity to meet growing demands
"At scale" huge demand & lots of data
"stability" not falling over under normal demand
What's normal demand?
129K packages 239 GB package tarballs 40 million pkg dls/day
1500 req/sec, peak 3200
"Legacy" Anything you've put into production
this is the story of a legacy system becoming more
flexible
None
January 2013 20K packages .5 million dls/day
Oct 2013 44K packages 108 million dls/month 3.6 million dls/day
None
our plucky little registry had to change
step 1: CDN Put Fastly.com in front of the registry
cache rules everything around me
step 2: tarballs get them out of couchdb
tarballs are huge! couch runs better without them base64 decoding
is work.
None
January 2014 60K packages 6+ million dls/day
step 3: visibility are things going wrong? what's going wrong?
reactive monitoring monitor deeply fix things quickly
proactive monitoring self-healing (also things don't break)
monitoring is unit testing Add monitoring after every outage
visibility is a prerequisite but not a solution
act on what monitoring and metrics reveal
step 4: redundancy several CouchDBs! reads, writes, & replication
fewer responsibilities for each piece isolates errors
step 5: automation ansible no server is special
June 2014 Superficially similar.
June 2014 80K packages 10 million dls/day
step 6: simplification now that it's not on fire we
can modify at leisure
None
Nov 2014 105K packages 28 million dls/day peak
50/50 AWS region split no AWS-specific magic Ubuntu 14.04 Trusty
Fastly: geoloc + varnish haproxy + CouchDB nginx + a
filesystem
where's the node?
registry 2 electric boogaloo with 500% more node
None
haproxy + node services couchdb ➜ postgres redis for caching
nginx + filesystem
more complicated more flexible & redundant more scaling dials to
turn
excited about postgres ad-hoc queries are fun
scaling node is exactly like scaling everything else
Understand system get visibility cool down hot spots add redundancy
npm client <3 npm install -g npm@latest
npm loves you