Scaling an API: From Reboots to Redis, how (not) to do it

My company that has grown from a small B2C app written in PHP/MySQL to a B2B advertising company doing 15 Billion API calls per month. Along the way there have been some big technology changes and decisions that have both helped and hindered us. Along the way we have had 3 day outages, several AWS outages, massive traffic spikes especially early days when we signed customers like WordPress.com, and many many improvements, re-factors and even successes. Today and for the last year we are highly modern engineering team, using many of the latest technologies and methodologies. We code in PHP and Python with C, Go, CoffeeScript and more, store data in MongoDB, Redis, memcache, Solr and MySQL, we have a full CI environment, do deploys using IRC (and Hubot) and Puppet everything. We have 100% (not 99.999%) up time for over 12 months now.

Transcript

1. Scaling an API From Reboots to Redis, how (not) to do it
2. { "fullName": "Ciarán Rooney", "workLife": { "company": "Skimlinks", "position": "CTO", "website": "http://skimlinks.com" }, "homeLife": { "email": "ciaran.rooney@gmail.com", "twitter": "@CiaranR", "website": "http://ciaran.ie", "hobby": "http://phplondon.org" } }
4. { "dataYear": "2013", "dataMonth": "april", "dataPeriod": "30 days", "dataRequests": { "apiRequests": "18,256,654,541", "impressions": “5,773,142,857", "uniqueUsers": "145,137,679" }, "customers": { "websitesPowered": "1,087,348", "customers": "26,379" } }
15. BIND / DNS imageStorage eMail
16. imageStorage eMail
17. imageStorage
24. APIs
25. Database Master/ Slave
26. Client App
27. Client App Database Master/ Slave APIs
29. WordPress
30. WordPress Pinterest
33. Client App Database Master/ Slave APIs
34. WordPress
37. SkimWords™
39. APIxx
40. APIxx APIxx APIxx APIxx APIxx APIxx APIxx APIxx CLUSTERxx ELB ELB
41. CLUSTERxx CLUSTERxx CLUSTERxx CLUSTERxx CLUSTERxx US-WEST US-EAST EU-WEST
44. Pinterest
47. JAYxx Apache Flume
48. JAYxx JAYxx JAYxx JAYxx JAYxx JAYxx JAYxx JAYxx ELB ELB CLUSTERxx Apache Flume Collector C Prefix Tree C NLP Filtering
49. CLUSTERxx CLUSTERxx CLUSTERxx CLUSTERxx CLUSTERxx US-WEST US-EAST EU-WEST Apache Flume Master
51. Cacti
53. Conclusions  Small Scalable units  Horizontal is always better  Eliminate Single points of failure  Scale writes and reads separately  Automate everything  Monitor everything
54. Scaling an API From Reboots to Redis, how (not) to do it Questions?
55. Scaling an API From Reboots to Redis, how (not) to do it Feedback! https://joind.in/8639
56. { "fullName": "Ciarán Rooney", "workLife": { "company": "Skimlinks", "position": "CTO", "website": "http://skimlinks.com" }, "homeLife": { "email": "ciaran.rooney@gmail.com", "twitter": "@CiaranR", "website": "http://ciaran.ie", "hobby": "http://phplondon.org" } }