Extreme Makeover: Rubygems Edition

Rubygems.org provides every Rubyist with an amazing service: all the libraries in the Ruby world. As amazing as that is, installing gems can be a time-consuming and even error-prone process. (Just ask the Travis guys.) In this talk, you'll learn about the recent dramatic changes in Rubygems and Bundler to improve speed and reliability by rewriting the Rubygems client/server architecture. I'll show how the new system caches more information, makes fewer requests, and takes less time to install gems. Finally, I'll cover how the changes allow worldwide mirrors of rubygems.org, improving things for Rubyists around the globe.

Transcript

1. Extreme Makeover
   Rubygems Edition
   

   View Slide

2. André Arko
   @indirect
   

   View Slide

3. View Slide

4. State of the Rubygems
   Year in Review
   

   View Slide

5. Bundler DDoS
   !
   !
   !
   ! !
   

   View Slide

6. Bundler DDoS
   

   View Slide

7. Security breach
   
   

   View Slide

8. Rebuilt rubygems.org
   
   

   View Slide

9. Travis network issues
   ☁️
   

   View Slide

10. Travis DNS issues
    ☁️
    

    View Slide

11. SSL failures
    ⚠️
    

    View Slide

12. SSL fixes
    
    

    View Slide

13. How it works today
    

    View Slide

14. How it works today
    with a fast connection, at least
    

    View Slide

15. /specs/rails-4.0.1.gemspec
    /specs/sinatra-1.4.2.gemspec
    !
    Gemspecs
    

    View Slide

16. /specs/rails-4.0.1.gemspec
    /specs/sinatra-1.4.2.gemspec
    /Marshal.4.8.gz
    Marshalled Gemspecs
    

    View Slide

17. /specs/rails-4.0.1.gemspec
    /specs/sinatra-1.4.2.gemspec
    /Marshal.4.8.gz
    /specs.4.8.gz
    List of Gemspecs
    

    View Slide

18. /api/v1/dependencies?gems=rack
    /specs/gemname-version.gemspec
    !
    The Bundler API
    

    View Slide

19. Could be better
    
    

    View Slide

20. We Have A Plan
    
    

    View Slide

21. We Have A Plan
    simple plaintext indexes
    

    View Slide

22. We Have A Plan
    clients cache the indexes
    

    View Slide

23. We Have A Plan
    put a CDN on it
    

    View Slide

24. We Have A Plan
    easy local mirrors
    

    View Slide

25. The New Index
    /versions
    /deps/gemname
    

    View Slide

26. /versions
    rack 1.0.0,1.0.1,1.0.2
    sinatra 0.9.6,1.0,1.1.4
    rack 2.0.0
    

    View Slide

27. /deps/sinatra
    0.9.6 rack:>= 0.9.1
    1.0 rack:>= 1.0
    1.1.4 rack:~> 1.1,tilt:< 2.0&>= 1.2.2
    

    View Slide

28. !
    +
    

    View Slide

29. Where we’re at
    designed the new index
    new index from bundler-api
    new index in bundler
    

    View Slide

30. What’s left to do
    new index from rubygems.org
    !
    new index in rubygems
    !
    

    View Slide

31. What’s left to do
    new index from rubygems.org
    new index in rubygems
    fastly-hosted index files
    

    View Slide

32. The Present
    gem install bundler
    

    View Slide

33. bundle install -j4
    Parallel gem installs
    export BUNDLE_JOBS=4
    

    View Slide

34. Gem mirrors
    bundle config
    mirror.https://rubygems.org
    https://tokyo-m.rubygems.org
    

    View Slide

35. New resolver!
    better errors
    faster installs
    many bugfixes
    

    View Slide

36. The Future
    

    View Slide

37. New index
    super fast installs!
    !
    

    View Slide

38. you can help!
    andre@arko.net
    @indirect
    

    View Slide