devops at npm: scaling the registry

How we scaled the registry from a handful of packages to 28 million downloads a day, with general lessons learned.

Transcript

1. scaling the registry
2. C J Silverio devops at npmjs.com @ceejbot
3. What we did lessons learned generalizations
5. Jacques Marneweck Benjamin Coe Laurie Voss
7. January 2013 20K packages .5 million dls/day
8. January 2014 60K packages 8 million dls/day
9. Nov 2014 > 100K packages 28 million dls/day peak
10. side project 100% couchdb donated hosting IrisCouch
11. October 2013
12. General lesson #1 Put a cache on it
13. Re-architecture 1 move tarballs out of poor couchdb
15. February 2014 company founded
16. hosted on Joyent/SmartOS hand-built CouchDB + Spidermonkey bash scripts to deploy
17. Twitter tells us when we're down
18. Re-architecture 2 Many couchdbs
19. General lesson #2 understand your db deeply
20. Monitoring & alerts
21. General lesson #3 Add monitoring after every outage
22. 1: reactive monitor deeply fix things quickly
23. 2: proactive self-healing monitoring (also things don't break)
24. June 2014 Superficially similar.
25. AWS / Ubuntu 70/30 west/east split 52 running instances, variable
27. 50/50 AWS region split haproxy to load balance no AWS-specific magic
28. Fastly: geoloc + cache haproxy / CouchDB nginx + a filesystem
29. behind the scenes ansible / nagios InfluxDB+Grafana
30. General lesson #4 metrics for everything
31. memory & cpu use request latency event counts
32. metrics == visibility
33. metrics drive monitoring
34. General lesson #5 automate
35. no special snowflakes every instance can be replaced
36. General lesson #6 the goal is to be BORING
37. if operations are boring you can do the dev
38. Goal: to be the most boring part of your node experience
39. npm client <3 npm install -g npm@latest
40. npm loves you