Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Extreme Makeover: Rubygems Edition

Extreme Makeover: Rubygems Edition

Rubygems.org provides every Rubyist with an amazing service: all the libraries in the Ruby world. As amazing as that is, installing gems can be a time-consuming and even error-prone process. (Just ask the Travis guys.) In this talk, you'll learn about the recent dramatic changes in Rubygems and Bundler to improve speed and reliability by rewriting the Rubygems client/server architecture. I'll show how the new system caches more information, makes fewer requests, and takes less time to install gems. Finally, I'll cover how the changes allow worldwide mirrors of rubygems.org, improving things for Rubyists around the globe.

André Arko

March 28, 2014
Tweet

More Decks by André Arko

Other Decks in Technology

Transcript

  1. Extreme Makeover
    Rubygems Edition

    View Slide

  2. André Arko
    @indirect

    View Slide

  3. View Slide

  4. State of the Rubygems
    Year in Review

    View Slide

  5. Bundler DDoS
    !
    !
    !
    ! !

    View Slide

  6. Bundler DDoS

    View Slide

  7. Security breach

    View Slide

  8. Rebuilt rubygems.org

    View Slide

  9. Travis network issues
    ☁️

    View Slide

  10. Travis DNS issues
    ☁️

    View Slide

  11. SSL failures
    ⚠️

    View Slide

  12. SSL fixes

    View Slide

  13. How it works today

    View Slide

  14. How it works today
    with a fast connection, at least

    View Slide

  15. /specs/rails-4.0.1.gemspec
    /specs/sinatra-1.4.2.gemspec
    !
    Gemspecs

    View Slide

  16. /specs/rails-4.0.1.gemspec
    /specs/sinatra-1.4.2.gemspec
    /Marshal.4.8.gz
    Marshalled Gemspecs

    View Slide

  17. /specs/rails-4.0.1.gemspec
    /specs/sinatra-1.4.2.gemspec
    /Marshal.4.8.gz
    /specs.4.8.gz
    List of Gemspecs

    View Slide

  18. /api/v1/dependencies?gems=rack
    /specs/gemname-version.gemspec
    !
    The Bundler API

    View Slide

  19. Could be better

    View Slide

  20. We Have A Plan

    View Slide

  21. We Have A Plan
    simple plaintext indexes

    View Slide

  22. We Have A Plan
    clients cache the indexes

    View Slide

  23. We Have A Plan
    put a CDN on it

    View Slide

  24. We Have A Plan
    easy local mirrors

    View Slide

  25. The New Index
    /versions
    /deps/gemname

    View Slide

  26. /versions
    rack 1.0.0,1.0.1,1.0.2
    sinatra 0.9.6,1.0,1.1.4
    rack 2.0.0

    View Slide

  27. /deps/sinatra
    0.9.6 rack:>= 0.9.1
    1.0 rack:>= 1.0
    1.1.4 rack:~> 1.1,tilt:< 2.0&>= 1.2.2

    View Slide

  28. !
    +

    View Slide

  29. Where we’re at
    designed the new index
    new index from bundler-api
    new index in bundler

    View Slide

  30. What’s left to do
    new index from rubygems.org
    !
    new index in rubygems
    !

    View Slide

  31. What’s left to do
    new index from rubygems.org
    new index in rubygems
    fastly-hosted index files

    View Slide

  32. The Present
    gem install bundler

    View Slide

  33. bundle install -j4
    Parallel gem installs
    export BUNDLE_JOBS=4

    View Slide

  34. Gem mirrors
    bundle config
    mirror.https://rubygems.org
    https://tokyo-m.rubygems.org

    View Slide

  35. New resolver!
    better errors
    faster installs
    many bugfixes

    View Slide

  36. The Future

    View Slide

  37. New index
    super fast installs!
    !

    View Slide

  38. you can help!
    [email protected]
    @indirect

    View Slide