$30 off During Our Annual Pro Sale. View Details »

Deathmatch: Bundler vs. Rubygems.org (Scottish Ruby 2013)

Deathmatch: Bundler vs. Rubygems.org (Scottish Ruby 2013)

The story of the quest to make `bundle install` faster; in which Rubyists around the world inadvertently DDoS rubygems.org, witness its ignominious death, and vow to rebuild it from the ashes stronger than it was before. Then, a tour of the changes; why is Redis so much slower than Postgres? Marvel at the gorgeous metrics and graphs used to measure and optimize; gasp in delight as we track, live, exactly how many Heroku dynos are needed. Finally, a happy ending: today, the server responds to requests TWO ORDERS OF MAGNITUDE faster than it did before.

André Arko

May 12, 2013
Tweet

More Decks by André Arko

Other Decks in Technology

Transcript

  1. Deathmatch:
    vs

    View Slide

  2. André Arko
    @indirect
     
     

    View Slide

  3. View Slide

  4. View Slide

  5. vs

    View Slide

  6. !

    View Slide

  7. Bundler 1.0

    View Slide

  8. Bundler 1.0
    30 Aug 2010

    View Slide

  9. Bundler 1.0
    30 Aug 2010
    !

    View Slide

  10. bundle install

    View Slide

  11. bundle install
    ...and install

    View Slide

  12. bundle install
    ...and install
    ...and install

    View Slide

  13. bundle install
    ...and install
    ...and install
    ...and install
    ...and install

    View Slide

  14. !

    View Slide

  15. (what seemed like)
    10 minutes later…

    View Slide

  16. View Slide

  17. bundler API

    View Slide

  18. bundler API
    just the gems you need

    View Slide

  19. bundler API
    so. much. faster.

    View Slide

  20. 32s → 8s
    so. much. faster.

    View Slide

  21. Bundler 1.1

    View Slide

  22. Bundler 1.1
    07 March 2012

    View Slide

  23. Bundler 1.1
    07 March 2012
    !

    View Slide

  24. API requests
    are lots of work

    View Slide

  25. pride cometh
    etc., etc…

    View Slide

  26. dramatic
    reenactment

    View Slide

  27. !
    !

    View Slide

  28. !
    !
    !

    View Slide

  29. !
    !
    !
    ! !

    View Slide

  30. !
    !
    !
    ! !
    ! !

    View Slide

  31. no more API

    View Slide

  32. View Slide

  33. View Slide

  34. View Slide

  35. crap.

    View Slide

  36. replacement
    a new bundler api

    View Slide

  37. heroku
    sinatra
    postgres

    View Slide

  38. questions

    View Slide

  39. questions
    will it work, this time?

    View Slide

  40. questions
    how is it bad?

    View Slide

  41. questions
    how can it be good?

    View Slide

  42. answers
    need measurements

    View Slide

  43. we meausure
    a lot of things

    View Slide

  44. librato
    tracks all the metrics

    View Slide

  45. View Slide

  46. papertrail
    logs all the lines

    View Slide

  47. View Slide

  48. measuring
    leads to knowledge

    View Slide

  49. discovery 1
    small postgres is fast

    View Slide

  50. discovery 1
    big postgres is faster

    View Slide

  51. discovery 2
    redis caching was slow

    View Slide

  52. discovery 3
    usage is hard to measure

    View Slide

  53. discovery 3
    usage is hard to measure

    View Slide

  54. discovery 3
    usage is hard to measure

    View Slide

  55. discovery 3
    usage is hard to measure

    View Slide

  56. discovery 4
    threading is great!

    View Slide

  57. discovery 4
    threading is great!
    …but deadlocks aren’t

    View Slide

  58. discovery 5
    “response time” isn’t

    View Slide

  59. discovery 5
    “response time” isn’t
    so check from outside

    View Slide

  60. then the bugs
    came rolling in

    View Slide

  61. synchronizing
    is really, really hard

    View Slide

  62. background
    processing is slow

    View Slide

  63. webooks
    are unreliable

    View Slide

  64. both is
    actually pretty good

    View Slide

  65. after all that
    bundling is fast again

    View Slide

  66. 2s → 20ms
    two entire orders of
    magnitude faster

    View Slide

  67. it’s okay,
    I guess

    View Slide

  68. we can do
    way better

    View Slide

  69. and we
    have a plan

    View Slide