Deathmatch: Bundler vs. Rubygems.org (Scottish Ruby 2013)

The story of the quest to make `bundle install` faster; in which Rubyists around the world inadvertently DDoS rubygems.org, witness its ignominious death, and vow to rebuild it from the ashes stronger than it was before. Then, a tour of the changes; why is Redis so much slower than Postgres? Marvel at the gorgeous metrics and graphs used to measure and optimize; gasp in delight as we track, live, exactly how many Heroku dynos are needed. Finally, a happy ending: today, the server responds to requests TWO ORDERS OF MAGNITUDE faster than it did before.

Transcript

1. Deathmatch: vs
2. André Arko @indirect    
5. vs
6. !
7. Bundler 1.0
8. Bundler 1.0 30 Aug 2010
9. Bundler 1.0 30 Aug 2010 !
10. bundle install
11. bundle install ...and install
12. bundle install ...and install ...and install
13. bundle install ...and install ...and install ...and install ...and install
14. !
15. (what seemed like) 10 minutes later…
17. bundler API
18. bundler API just the gems you need
19. bundler API so. much. faster.
20. 32s → 8s so. much. faster.
21. Bundler 1.1
22. Bundler 1.1 07 March 2012
23. Bundler 1.1 07 March 2012 !
24. API requests are lots of work
25. pride cometh etc., etc…
26. dramatic reenactment
27. ! !
28. ! ! !
29. ! ! ! ! !
30. ! ! ! ! ! ! !
31. no more API
35. crap.
36. replacement a new bundler api
37. heroku sinatra postgres
38. questions
39. questions will it work, this time?
40. questions how is it bad?
41. questions how can it be good?
42. answers need measurements
43. we meausure a lot of things
44. librato tracks all the metrics
46. papertrail logs all the lines
48. measuring leads to knowledge
49. discovery 1 small postgres is fast
50. discovery 1 big postgres is faster
51. discovery 2 redis caching was slow
52. discovery 3 usage is hard to measure
53. discovery 3 usage is hard to measure
54. discovery 3 usage is hard to measure
55. discovery 3 usage is hard to measure
56. discovery 4 threading is great!
57. discovery 4 threading is great! …but deadlocks aren’t
58. discovery 5 “response time” isn’t
59. discovery 5 “response time” isn’t so check from outside
60. then the bugs came rolling in
61. synchronizing is really, really hard
62. background processing is slow
63. webooks are unreliable
64. both is actually pretty good
65. after all that bundling is fast again
66. 2s → 20ms two entire orders of magnitude faster
67. it’s okay, I guess
68. we can do way better
69. and we have a plan