Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Extreme Makeover: Rubygems Edition (RubyKaigi 2014)

André Arko
September 19, 2014

Extreme Makeover: Rubygems Edition (RubyKaigi 2014)

Rubygems.org provides every Rubyist with an amazing service: all the libraries in the Ruby world. As amazing as that is, installing gems can be a time-consuming and even error-prone process. (Just ask the Travis guys.) In this talk, you'll learn about the recent dramatic changes in Rubygems and Bundler to improve speed and reliability by rewriting the Rubygems client/server architecture. I'll show how the new system caches more information, makes fewer requests, and takes less time to install gems. Finally, I'll cover how the changes allow worldwide mirrors of rubygems.org, improving things for Rubyists around the globe.

André Arko

September 19, 2014
Tweet

More Decks by André Arko

Other Decks in Technology

Transcript

  1. Extreme Makeover
    Rubygems Edition

    View full-size slide

  2. André Arko
    @indirect

    View full-size slide

  3. State of the Rubygems
    Year in Review

    View full-size slide

  4. Bundler DDoS
    !
    !
    !
    ! !

    View full-size slide

  5. Bundler DDoS

    View full-size slide

  6. Security breach

    View full-size slide

  7. Rebuilt rubygems.org

    View full-size slide

  8. Travis network issues
    ☁️

    View full-size slide

  9. Travis DNS issues
    ☁️

    View full-size slide

  10. SSL failures
    ⚠️

    View full-size slide

  11. Rebuilt rubygems.org

    View full-size slide

  12. Bundler Source CVE

    View full-size slide

  13. Bundler 1.7 released

    View full-size slide

  14. Bundler 1.7 released

    View full-size slide

  15. Bundler 1.7.2 released

    View full-size slide

  16. How it works today

    View full-size slide

  17. How it works today
    with a fast connection, at least

    View full-size slide

  18. /specs/rails-4.0.1.gemspec
    /specs/sinatra-1.4.2.gemspec
    !
    Gemspecs

    View full-size slide

  19. /specs/rails-4.0.1.gemspec
    /specs/sinatra-1.4.2.gemspec
    /Marshal.4.8.gz
    Marshalled Gemspecs

    View full-size slide

  20. /specs/rails-4.0.1.gemspec
    /specs/sinatra-1.4.2.gemspec
    /Marshal.4.8.gz
    /specs.4.8.gz
    List of Gemspecs

    View full-size slide

  21. /api/v1/dependencies?gems=rack
    /specs/gemname-version.gemspec
    !
    The Bundler API

    View full-size slide

  22. Could be better

    View full-size slide

  23. We Have A Plan

    View full-size slide

  24. We Have A Plan
    simple plaintext indexes

    View full-size slide

  25. We Have A Plan
    clients cache the indexes

    View full-size slide

  26. We Have A Plan
    put a CDN on it

    View full-size slide

  27. We Have A Plan
    easy local mirrors

    View full-size slide

  28. The New Index
    /versions
    /deps/gemname

    View full-size slide

  29. /versions
    rack 1.0.0,1.0.1,1.0.2
    sinatra 0.9.6,1.0,1.1.4
    rack 2.0.0

    View full-size slide

  30. /deps/sinatra
    0.9.6 rack:>= 0.9.1
    1.0 rack:>= 1.0
    1.1.4 rack:~> 1.1,tilt:< 2.0&>= 1.2.2

    View full-size slide

  31. Where we’re at
    designed the new index
    new index from bundler-api
    new index in bundler

    View full-size slide

  32. What’s left to do
    new index from rubygems.org
    !
    new index in rubygems
    !

    View full-size slide

  33. What’s left to do
    new index from rubygems.org
    new index in rubygems
    fastly-hosted index files

    View full-size slide

  34. The Present
    gem install bundler

    View full-size slide

  35. bundle install -j4
    Parallel gem installs
    export BUNDLE_JOBS=4

    View full-size slide

  36. Gem mirrors
    bundle config
    mirror.https://rubygems.org
    https://tokyo-m.rubygems.org

    View full-size slide

  37. New resolver!
    better errors
    faster installs
    many bugfixes

    View full-size slide

  38. New index
    super fast installs!
    !

    View full-size slide

  39. you can help!
    github.com/bundler/bundler
    CONTRIBUTING.md

    View full-size slide

  40. you can help!
    gittip.com/bundler

    View full-size slide