A History of Fetching Specs

Transcript

1. A History of Fetching Specs Terence Lee
2. R Ruby T Task F Force M Manager
3. #emumarch
4. Inspiration
5. Friday Hugs
6. Frozen Rails
7. Red Dot Ruby Conf
8. Euruko
9. Ruby Lugdunum
10. Cascadia RubyConf
12. It's Friday...
13. A History of Fetching Specs
15. $ gem install sinatra
17. $ cat Gemfile source 'https://rubygems.org' gem 'sinatra'
18. $ bundle install
19. Agenda Modern Index Dependency API New Index Format
20. Modern Index
21. $ wget --quiet http://rubygems.org/specs.4.8.gz $ du -h * 1.5M specs.4.8.gz
22. Marshal.load(Gem.gunzip(File.read('specs.4.8.gz'))) => [ ["sinatra", #<Gem::Version "1.3.2">, "ruby"], ... ]
23. Problem #1: No Dependency Info
24. Gem::RemoteFetcher.fetcher.download(spec, uri, path) #=> sinatra-1.3.2.gemspec.rz
25. spec = Marshal.load(Gem.inflate( File.read('sinatra-1.3.2.gemspec.rz'))) => #<Gem::Specification name=sinatra version=1.3.2> spec.dependencies => [<Gem::Dependency type=:runtime name="rack" requirements=">= 1.3.6, ~> 1.3", ...]
26. Problem #2: MOAR RAM
27. Problem #3: Sloooow
28. real 0m17.918s user 0m7.404s sys 0m0.136s
29. Dependency API
30. API Endpoint http://rubygems.org/api/v1/dependencies?gems=sinatra,foo,bar
31. Marshal.load(open('...')) => [ {:name=>"sinatra", :number=>"1.3.2", :platform=>"ruby", :dependencies=> [["tilt", ">= 1.3.3, ~> 1.3"], ["rack-protection", "~> 1.2"], ["rack", ">= 1.3.6, ~> 1.3"]]}, ... ]
32. Modern Index Problems
33. No Dependency Info - Solved
34. MOAR RAM - Solved
35. Sloooow - Faster real 0m3.337s user 0m0.456s sys 0m0.056s
36. Problem #1: Not cacheable
37. every user has different query params http://rubygems.org/api/v1/dependencies?gems=foo,bar,baz http://rubygems.org/api/v1/dependencies?gems=foo,bar,boo
38. new gem version every 5 mins
39. Not cacheable in HTTP Datastore (Redis)
40. Problem #2: Large Queries
41. SELECT rv.name, rv.number, rv.platform, d.requirements, for_dep_name.name dep_name FROM (SELECT r.name, v.number, v.platform, v.id AS version_id FROM rubygems AS r, versions AS v WHERE v.rubygem_id = r.id AND v.indexed is true AND r.name IN ?) AS rv LEFT JOIN dependencies AS d ON d.version_id = rv.version_id LEFT JOIN rubygems AS for_dep_name ON d.rubygem_id = for_dep_name.id AND d.scope = 'runtime';
42. Problem #3: Operational Burdens
43. Run by volunteers (Thanks Larry Marburger)
44. Local to US East
45. New Index Format
46. names.list a_gem bees_in_my_eyes foo
47. versions.list foo 0.0.1 bees_in_my_eyes 1.0,1.0.1,1.1,2.0 nokogiri 1.0.0-x86-mswin32-60,1.0.0,1.0.1,1.0.2,1.0.3
48. deps/* deps/nokogiri 1.0.0 1.1.6 rake:>= 0.7.1,activesupport:= 1.3.1,... 1.5.7.rc2 |ruby:>= 1.8.7,rubygems:> 1.3.1 1.5.7.rc3 |rubygems:> 1.3.1 1.6.0-java mini_portile:~> 0.5.0
49. Local Performance
50. Sizes Deps (202mb) vs all marshalled gemspecs (1.1gb) Gzipped (2.7mb) vs (20mb)
51. Performance of Whole Index (new) 0.70s user 0.13s system 99% cpu 0.838 total (old) 0.70s user 0.12s system 95% cpu 0.858 total (just rails) 0.08s user 0.04s system 96% cpu 0.126 total
52. HTTP Cacheable
53. CDN Currently Trying Fastly Fastly (< 1s) vs S3 (> 2s) Global Reach
54. Local Caching
55. Partial Cache Busting (ETags)
56. GZIP
57. HTTP Pipelining
58. Coming soon in a Bundler pre near you
59. Parallel Install $ gem install bundler --pre $ bundle install -j4
60. Thanks to Bundler Core Andre Arko Larry Marburger Jessica Lynn Suttles
61. Thank You @hone02 terence@heroku.com