Stabilizing npm

How the npm, Inc team stabilized the registry early in 2014. Video from the presentation is here: https://www.youtube.com/watch?v=3ivx2RsZ1yA

1. stabilizing the registry
2. C J Silverio devops at npmjs.com @ceejbot
3. side project 100% couchdb donated hosting IrisCouch
6. December 2013
7. January 2014
8. February 2014 » company founded & funded » 100% hosted on Joyent » several skimdbs load-balanced by Fastly » hand-built CouchDB + Spidermonkey » automation by bash » Twitter tells us when we're down
9. This is when I arrive. (funding means you can hire!) » PagerDuty account: first thing I did » Nagios all hooked up & monitoring basic host health » we have maybe 10 hosts total driving the registry
10. Funding also means attention from bounty-hunters.
11. security audit
12. Stabilization stage 1 reactive » monitor everything more deeply » methodically identify & monitor causes of outages » react quickly to fix problems » Twitter is no longer telling us when we're down
13. Stabilization stage 2 proactive » our second devops person: Ben Coe » recurring problems fixed in the apps » monitoring checks self-heal » redundancy everywhere » automation! » our night shift is bored!
14. June 2013 Superficially similar.
15. major changes 100% on AWS Ubuntu Trusty 70/30 split between us-west-2 & us-east-1 100% automated with ansible 52 running instances, variable
16. the stack » Fastly CDN for Varnish cache & geolocality » nginx to serve static files » pound to terminate TLS » CouchDB for package metadata & app logic » nagios + PagerDuty for monitoring » InfluxDB + Grafana for metrics » Tarsnap for backups
19. weak points » single points of failure: Fastly, write primary » still looking for an off-AWS backup » expensive to run: too many couchdbs » too entangled with couchdb » complex in odd places: the skimworker, for example
20. I now praise CouchDB
21. my next goal: make it cheap
22. by next week haproxy 50-50 region balance cheaper by far
23. my long-term goal: npm as reliable utility