Slide 1

Slide 1 text

A History of Fetching Specs Terence Lee

Slide 2

Slide 2 text

R Ruby T Task F Force M Manager

Slide 3

Slide 3 text

#emumarch

Slide 4

Slide 4 text

Inspiration

Slide 5

Slide 5 text

Friday Hugs

Slide 6

Slide 6 text

Frozen Rails

Slide 7

Slide 7 text

Red Dot Ruby Conf

Slide 8

Slide 8 text

Euruko

Slide 9

Slide 9 text

Ruby Lugdunum

Slide 10

Slide 10 text

Cascadia RubyConf

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

It's Friday...

Slide 13

Slide 13 text

A History of Fetching Specs

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

$ gem install sinatra

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

$ cat Gemfile source 'https://rubygems.org' gem 'sinatra'

Slide 18

Slide 18 text

$ bundle install

Slide 19

Slide 19 text

Agenda Modern Index Dependency API New Index Format

Slide 20

Slide 20 text

Modern Index

Slide 21

Slide 21 text

$ wget --quiet http://rubygems.org/specs.4.8.gz $ du -h * 1.5M specs.4.8.gz

Slide 22

Slide 22 text

Marshal.load(Gem.gunzip(File.read('specs.4.8.gz'))) => [ ["sinatra", #, "ruby"], ... ]

Slide 23

Slide 23 text

Problem #1: No Dependency Info

Slide 24

Slide 24 text

Gem::RemoteFetcher.fetcher.download(spec, uri, path) #=> sinatra-1.3.2.gemspec.rz

Slide 25

Slide 25 text

spec = Marshal.load(Gem.inflate( File.read('sinatra-1.3.2.gemspec.rz'))) => # spec.dependencies => [

Slide 26

Slide 26 text

Problem #2: MOAR RAM

Slide 27

Slide 27 text

Problem #3: Sloooow

Slide 28

Slide 28 text

real 0m17.918s user 0m7.404s sys 0m0.136s

Slide 29

Slide 29 text

Dependency API

Slide 30

Slide 30 text

API Endpoint http://rubygems.org/api/v1/dependencies?gems=sinatra,foo,bar

Slide 31

Slide 31 text

Marshal.load(open('...')) => [ {:name=>"sinatra", :number=>"1.3.2", :platform=>"ruby", :dependencies=> [["tilt", ">= 1.3.3, ~> 1.3"], ["rack-protection", "~> 1.2"], ["rack", ">= 1.3.6, ~> 1.3"]]}, ... ]

Slide 32

Slide 32 text

Modern Index Problems

Slide 33

Slide 33 text

No Dependency Info - Solved

Slide 34

Slide 34 text

MOAR RAM - Solved

Slide 35

Slide 35 text

Sloooow - Faster real 0m3.337s user 0m0.456s sys 0m0.056s

Slide 36

Slide 36 text

Problem #1: Not cacheable

Slide 37

Slide 37 text

every user has different query params http://rubygems.org/api/v1/dependencies?gems=foo,bar,baz http://rubygems.org/api/v1/dependencies?gems=foo,bar,boo

Slide 38

Slide 38 text

new gem version every 5 mins

Slide 39

Slide 39 text

Not cacheable in HTTP Datastore (Redis)

Slide 40

Slide 40 text

Problem #2: Large Queries

Slide 41

Slide 41 text

SELECT rv.name, rv.number, rv.platform, d.requirements, for_dep_name.name dep_name FROM (SELECT r.name, v.number, v.platform, v.id AS version_id FROM rubygems AS r, versions AS v WHERE v.rubygem_id = r.id AND v.indexed is true AND r.name IN ?) AS rv LEFT JOIN dependencies AS d ON d.version_id = rv.version_id LEFT JOIN rubygems AS for_dep_name ON d.rubygem_id = for_dep_name.id AND d.scope = 'runtime';

Slide 42

Slide 42 text

Problem #3: Operational Burdens

Slide 43

Slide 43 text

Run by volunteers (Thanks Larry Marburger)

Slide 44

Slide 44 text

Local to US East

Slide 45

Slide 45 text

New Index Format

Slide 46

Slide 46 text

names.list a_gem bees_in_my_eyes foo

Slide 47

Slide 47 text

versions.list foo 0.0.1 bees_in_my_eyes 1.0,1.0.1,1.1,2.0 nokogiri 1.0.0-x86-mswin32-60,1.0.0,1.0.1,1.0.2,1.0.3

Slide 48

Slide 48 text

deps/* deps/nokogiri 1.0.0 1.1.6 rake:>= 0.7.1,activesupport:= 1.3.1,... 1.5.7.rc2 |ruby:>= 1.8.7,rubygems:> 1.3.1 1.5.7.rc3 |rubygems:> 1.3.1 1.6.0-java mini_portile:~> 0.5.0

Slide 49

Slide 49 text

Local Performance

Slide 50

Slide 50 text

Sizes Deps (202mb) vs all marshalled gemspecs (1.1gb) Gzipped (2.7mb) vs (20mb)

Slide 51

Slide 51 text

Performance of Whole Index (new) 0.70s user 0.13s system 99% cpu 0.838 total (old) 0.70s user 0.12s system 95% cpu 0.858 total (just rails) 0.08s user 0.04s system 96% cpu 0.126 total

Slide 52

Slide 52 text

HTTP Cacheable

Slide 53

Slide 53 text

CDN Currently Trying Fastly Fastly (< 1s) vs S3 (> 2s) Global Reach

Slide 54

Slide 54 text

Local Caching

Slide 55

Slide 55 text

Partial Cache Busting (ETags)

Slide 56

Slide 56 text

GZIP

Slide 57

Slide 57 text

HTTP Pipelining

Slide 58

Slide 58 text

Coming soon in a Bundler pre near you

Slide 59

Slide 59 text

Parallel Install $ gem install bundler --pre $ bundle install -j4

Slide 60

Slide 60 text

Thanks to Bundler Core Andre Arko Larry Marburger Jessica Lynn Suttles

Slide 61

Slide 61 text

Thank You @hone02 [email protected]

Slide 62

Slide 62 text

No content