Dependencies, a boring, solved problem?

I gave this talk at Rulu 2013 in Lyon

Transcript

1. DEPENDENCIES: a solved, boring problem?
2. Greg Karékinian @gkarekinian Software developer Infrastructure developer Cofounder
3. Warning: This talk will open a gate to dependency hell
6. Ruby
7. Distro packages for Ruby are a mess.
8. $ sudo apt-get install ruby
9. No.
10. $ sudo apt-get install ruby1.9.1
11. $ sudo gem install nokogiri Fetching: mini_portile-0.5.0.gem (100%) Fetching: nokogiri-1.6.0.gem (100%) Building native extensions. This could take a while... ERROR: Error installing nokogiri: ERROR: Failed to build gem native extension. /usr/bin/ruby1.9.1 extconf.rb /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require': cannot load such file -- mkmf (LoadError) from /usr/lib/ruby/1.9.1/rubygems/ custom_require.rb:36:in `require' from extconf.rb:5:in `<main>' Gem files will remain installed in /var/lib/gems/1.9.1/ gems/nokogiri-1.6.0 for inspection. Results logged to /var/lib/gems/1.9.1/gems/ nokogiri-1.6.0/ext/nokogiri/gem_make.out
12. No.
13. $ sudo apt-get install ruby1.9.1-full
14. $ sudo apt-get install ruby1.9.1 ruby1.9.1-dev ri1.9.1
15. $ sudo gem install nokogiri Building native extensions. This could take a while... ERROR: Error installing nokogiri: ERROR: Failed to build gem native extension. /usr/bin/ruby1.9.1 extconf.rb Extracting libxml2-2.8.0.tar.gz into tmp//ports/libxml2/2.8.0... OK Running 'configure' for libxml2 2.8.0... ERROR, review 'tmp//ports/libxml2/2.8.0/configure.log' to see what happened. *** extconf.rb failed *** Could not create Makefile due to some reason, probably lack of necessary libraries and/or headers. Check the mkmf.log file for more details. You may need configuration options. Provided configuration options: --with-opt-dir --without-opt-dir --with-opt-include --without-opt-include=${opt-dir}/include --with-opt-lib --without-opt-lib=${opt-dir}/lib --with-make-prog --without-make-prog --srcdir=. --curdir --ruby=/usr/bin/ruby1.9.1 /var/lib/gems/1.9.1/gems/mini_portile-0.5.0/lib/mini_portile.rb:235:in `block in execute': Failed to complete configure task (RuntimeError) from /var/lib/gems/1.9.1/gems/mini_portile-0.5.0/lib/mini_portile.rb:227:in `chdir' from /var/lib/gems/1.9.1/gems/mini_portile-0.5.0/lib/mini_portile.rb:227:in `execute' from /var/lib/gems/1.9.1/gems/mini_portile-0.5.0/lib/mini_portile.rb:57:in `configure' from /var/lib/gems/1.9.1/gems/mini_portile-0.5.0/lib/mini_portile.rb:100:in `cook' from extconf.rb:101:in `block in <main>' from extconf.rb:119:in `call' from extconf.rb:119:in `block in <main>' from extconf.rb:109:in `tap' from extconf.rb:109:in `<main>' Gem files will remain installed in /var/lib/gems/1.9.1/gems/nokogiri-1.6.0 for inspection. Results logged to /var/lib/gems/1.9.1/gems/nokogiri-1.6.0/ext/nokogiri/gem_make.out
16. No.
17. $ sudo apt-get install ruby1.9.1 ruby1.9.1-dev ri1.9.1 build- essential libssl-dev zlib1g-dev libxml2-dev libxslt-dev
18. Gems
19. 1993: Ruby
20. 1995: 1st public version of Ruby
21. 1996: Ruby 1.0
22. 2003: RubyGems
24. Install, list, uninstall libs & apps
25. Dependencies!
26. RubyGems part of MRI 1.9+
27. Rails < 2.1
28. Plugins were not gems
29. We also used gems
30. Copy & paste dependencies from the README
31. When there's a README
32. 2008 Rails 2.1
33. http://railscasts.com/episodes/110-gem-dependencies # config/environment.rb config.gem "RedCloth", :version => ">= 3.0.4", :source => "http://code.whytheluckystiff.net/" config.gem 'aws-s3', :lib => 'aws/s3' $ gem install rails -v 2.1.0 $ rake gems:install
34. Almost solves the problem... for Rails apps
35. Bundler 0.3.0 July 29, 2009
36. source 'https://rubygems.org' gem 'rails', '~> 4.0.0.rc1' gem 'active_model_serializers', '~> 0.7.0'
37. $ bundle install
38. GEM remote: https://rubygems.org/ specs: actionmailer (4.0.0.rc1) actionpack (= 4.0.0.rc1) mail (~> 2.5.3) actionpack (4.0.0.rc1) activesupport (= 4.0.0.rc1) builder (~> 3.1.0) erubis (~> 2.7.0) rack (~> 1.5.2) rack-test (~> 0.6.2) active_model_serializers (0.7.0) activemodel (>= 3.0) activemodel (4.0.0.rc1)
39. Isolation
41. $ bundle package $ git add vendor/cache
42. $ bundle install --deployment
43. Requires up-to-date Gemfile.lock
44. Isolates dependencies
45. Problem solved.
46. Questions?
47. Not for apps distributed as gems
48. Apps as Gems
49. Command-line applications
50. 1. Conflicting versions 2. Unstable public API 3. Lack of isolation
51. 1. Conflicting versions
52. $ gem install chef -v 0.10.10 ERROR: While executing gem ... (Gem::DependencyError) Unable to resolve dependencies: chef requires net-ssh (~> 2.2.2); net-ssh- multi requires net-ssh (>= 2.1.4); net-ssh-gateway requires net-ssh (>= 2.6.5)
53. # chef 0.10.10 'net-ssh', '~> 2.2.2' 'net-ssh-multi', '~> 1.1.0' # net-ssh-multi 1.1 'net-ssh', '>= 2.1.4' 'net-ssh-gateway','>= 0.99.0' # net-ssh-gateway 1.2.0 'net-ssh', '>= 2.6.5'
54. # net-ssh-gateway 1.2.0 'net-ssh', '>= 2.6.5' Dependency resolution by hand # net-ssh-gateway 1.1.0 'net-ssh', '>= 1.99.1'
55. $ gem install chef -v 0.10.10 ERROR: While executing gem ... (Gem::DependencyError) Unable to resolve dependencies: net-ssh-gateway requires net-ssh (>= 2.6.5)
56. $ gem install net-ssh-gateway -v 1.1.0 Fetching: net-ssh-gateway-1.1.0.gem (100%) Successfully installed net-ssh- gateway-1.1.0 1 gem installed
57. $ gem install chef -v 0.10.10 Building native extensions. This could take a while... Fetching: [...] Successfully installed chef-0.10.10 20 gems installed
58. Yay! Done!
59. That would have been too easy
60. # chef 0.10.10 'moneta'
61. # chef 0.10.10 'moneta' That means ANY version
62. Moneta 0.7 Complete rewrite Different public API
63. $ gem uninstall moneta --ignore-dependencies $ gem install moneta -v 0.6.0 Dependency resolution by hand
64. $ gem install net-ssh-gateway -v 1.1.0 $ gem install moneta -v 0.6.0 $ gem install chef -v 0.10.10 Install chef 0.10.10:
65. blog.karekinian.com/ 2013/02/14/how-to- install-chef-0.10.4/
66. 2. Unstable public API
68. Pain
69. gembundler.com /v1.3/ rationale.html
70. “Constrain the dependency too much (rack =1.2.1) and you make it hard to use your project in other compatible projects.”
71. “Constrain it too little (rack >= 1.0) and a new release of Rack may break your code.”
72. “Using dependencies like rack ~> 1.2.1 and versioning code in a SemVer compliant way mostly solves this problem, but it assumes universal compliance.”
73. “Since Rubygems has over 100,000 packages, this assumption simply doesn't hold in practice.”
74. SemVer
75. Install a new gem Risk breaking another one
76. Example:
79. json gem 121 versions since Sept. 29, 2005
80. json gem 121 versions since Sept. 29, 2005
81. # chef 0.10.10 'json', '>= 1.4.4', '<= 1.6.1' # chef-handler-campfire 2.0.2 'tinder' # any version!
82. # tinder 1.9.1 'json', '~> 1.6' # tinder 1.9.2 'json', '~> 1.7.5'
83. # tinder 1.9.1 'json', '~> 1.6' >= 1.6.0, < 2.0.0 # tinder 1.9.2 'json', '~> 1.7.5'
84. # tinder 1.9.1 'json', '~> 1.6' >= 1.6.0, < 2.0.0 # tinder 1.9.2 'json', '~> 1.7.5' >= 1.7.5, < 1.8.0
85. different minor version of tinder = different json
86. Different json depending on the day you install chef-handler- campfire
87. New machine, different behavior
88. Fix dependencies by hand again!
89. # chef 11.4.4 # The JSON gem reliably releases breaking # changes as a patch release s.add_dependency "json", ">= 1.4.4", "<= 1.7.7"
90. Works for me
91. Works today
93. 3. Lack of isolation
95. Side-effects
96. Example:
97. # astrails-safe 0.2.7 'aws-s3' 'cloudfiles' 'net-sftp'
98. Your next backup
101. Tips
102. Keep your dependencies up to date
104. Expect things to break
105. Solutions
106. RVM, gemsets
107. RVM, gemsets Meh.
108. Use JRuby to deploy as jar/war
109. Pack Ruby scripts as a single binary
110. Build metagems just to fix dependencies
111. Hack something with Bundler
113. Vendor all the things as system packages
114. Vagrant Chef
115. Building distro packages: not my dream job
116. Omnibus Bunchr
117. github.com/ joemiller/ bunchr
118. DSL for packages & software
119. /usr/local/bin/ /usr/local/greenalto/bin/ /usr/local/greenalto/embedded/
121. Based on Rake!
122. It's pretty simple
123. fpm Fucking Package Manager
126. Bunchr::Software.new do |t| t.name = 'libxml2' t.version = '2.9.0' install_prefix = "#{Bunchr.install_dir}/embedded" t.download_commands << "curl -O http://xmlsoft.org/sources/"\ "libxml2-#{t.version}.tar.gz" t.download_commands << "tar xfvz libxml2-#{t.version}.tar.gz" t.work_dir = "libxml2-#{t.version}" t.build_commands << "./configure --prefix=#{install_prefix}" t.build_commands << "make" t.install_commands << "make install" CLEAN << install_prefix end
127. Bunchr::Packages.new do |t| t.name = 'greenalto_idk' t.version = '1' t.iteration = ENV['BUILD_NUMBER'] || '7' t.category = 'Development' t.license = 'All right reserved' t.vendor = 'Green Alto SAS' t.url = 'http://www.greenalto.com' t.description = 'Green Alto Infrastructure Developer Kit' t.include_software 'ruby' t.include_software 'chef' t.include_software 'knife-ec2' t.include_software 'knife-openstack' t.include_software 'greenalto_wrapper' t.files << Bunchr.install_dir t.files << '/usr/local/bin/greenalto' end
128. require 'bunchr' Bunchr.build_dir = '/tmp/build' Bunchr.install_dir = '/usr/local/greenalto' Bunchr.load_recipes Dir['recipes/**/*.rake'] desc 'Default task' task :default => ['packages:greenalto_idk']
129. Did I mention fpm is great?
130. github.com/ jordansissel/ fpm
131. deb rpm solaris tar
132. Go
133. $ go get github.com/mitchellh/go-vnc import "github.com/mitchellh/go-vnc"
134. No central package repo
135. No versions
136. !
137. No versions
138. NO
139. VERS IONS
140. master branch
141. No breaking API changes
142. Golang Dependency Management for Rubyists stovepipestudios.com/ blog/2013/02/go- dependency-
143. Does it work?
144. "Dependency my friends is someone elses’s piece of crap without which your piece of crap doesn’t work." areyoufuckingcoding.me/ 2012/04/16/dpendency- hell/
145. Questions?
146. Thank you.
147. fish shot at terrifyingly high aperture: @mortice Jason Seifer pics: http://jasonlint.org box-pile: http://www.flickr.com/photos/sillydog/ 287354869 Rake: http://www.flickr.com/photos/islespunkfan/ 2794069762 Broken eggs: http://www.flickr.com/photos/ peppysis/4870119233 Reggie Watts - If You're Fucking, you're Fucking Credits