Consistent Local Development with Vagrant & Chef

A high level overview of some of these fantastic tools, and some examples of how we've been using them to improve local development environments at the office.

Transcript

1. Consistent Local Development with Vagrant & Chef Lew Goettner @lewg #PhillyDevOps July 18, 2012
2. about me Also, these are additional notes where I felt the slide needed a little explanation for Speaker Deck.
3. LEWIS JOSEPH GOETTNER THE @lewg I’m shooting at manual processes.
4. I work for the Wharton School.
5. By JSquish (Own work) [CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons This may come to mind...
6. By TexasDex at en.wikipedia [GFDL (http://www.gnu.org/copyleft/fdl.html) or CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0/)], from Wikimedia Commons Or perhaps this, if you think about the University of Pennsylvania.
7. Wharton Computing Our group of ~130 employees probably lands somewhere in here.
8. PHP WordPress ColdFusion Windows Ubuntu/Red Hat MSSQL MySQL Ruby My primary focus, and our mixed bag of technologies.
9. Supporting non-technical users Slower adoption of new technology Legacy systems There is always room for a better way! When you’re stuck in the middle..
10. storytime “The Generic Developer’s Path to a Perfect Development Setup”
11. “Here’s a shared dev space.” - Sysadmin the progression...
12. “I’ll just run it locally.” - Developer the progression... Except, I don’t really want on this crap running my machine all the time.
13. “I’ll run it in a VM!” - Developer the progression... But configuring all this stuff isn’t all that fun.
14. “I’ll run it in a chef-configured VM!” - Developer the progression... Now my biggest gripe is the time it takes to install the OS.
15. “I’ll run it in a chef-configured VM, based on a post-install snapshot.” - Developer the progression... Now we’re talking!
16. “Why don’t we all start using this?” - Boss / Coworker / Intern the progression...
17. “Shit.” - Developer the progression...
18. vagrant
19. “Vagrant uses Oracle’s VirtualBox to build configurable, lightweight, and portable virtual machines dynamically.” - http://vagrantup.com
20. “I love the command line.” - Lew
21. % vagrant box add precise32 \ http://files.vagrantup.com/precise32.box % vagrant init precise32 % vagrant up 3 Lines = A Running VM
22. % vagrant box add precise32 \ http://files.vagrantup.com/precise32.box % vagrant init precise32 % vagrant up
23. Base OS Install Software for access & configuration VirtualBox Guest Additions Base Box
24. Built in tools for management: % vagrant box -h % vagrant package -h Base Box
25. VeeWee: tool for building your own https://github.com/jedi4ever/veewee Base Box
26. % vagrant box add precise32 \ http://files.vagrantup.com/precise32.box % vagrant init precise32 % vagrant up It’s a URL! So, create your own, and distribute via internet/intranet.
27. % vagrant box add precise32 \ http://files.vagrantup.com/precise32.box % vagrant init precise32 % vagrant up Drops a “Vagrantfile” into the current directory.
28. Vagrantfile = a Project Project = 1 to many VMs Vagrantfile
29. It’s just a text file Ruby DSL for configuring your VM(s) (no ruby skills required) Vagrantfile
30. % vagrant box add precise32 \ http://files.vagrantup.com/precise32.box % vagrant init precise32 % vagrant up
31. vagrant up http://www.flickr.com/photos/lightningjeff/7505934038/
32. “I’m not impressed.” -Developer Well, there’s some powerful stuff there, but maybe you’ve already been doing your own version of this with VMWare, snapshots, etc.
34. “Chef is an open-source systems integration framework built specifically for automating the cloud.” -http://www.opscode.com/chef/
35. “Chef means never hacking together shell scripts and/or dealing with config file hell again.” -Lew
36. Declarative: What, not how Idempotent: Only take action if required Convergent: Takes care of itself Configuration Management
37. You configure systems with Chef by writing self-documenting code. This code is lists of Resources that configure the system to do its job. Declarative Resources
38. package "bash" do action :install end
39. Chef Resources have Providers that take idempotent action to configure the resource, but only if it needs to change. Idempotent Actions
40. INFO: Processing package[apache2] action install (apache2::default line 20) DEBUG: package[apache2] checking package status for apache2 DEBUG: package[apache2] current version is 2.2.20-1ubuntu1.1 DEBUG: package[apache2] candidate version is 2.2.20-1ubuntu1.1 DEBUG: package[apache2] is already installed - nothing to do
41. Chef runs on the system, configuring the Node. In Chef, a single run should completely configure the system. Convergent Nodes
42. Chef Recipes are a pure Ruby domain specific language (DSL) Ruby Again!
43. Recipes are collected in Cookbooks along with associated components like config files or libraries. Continuing the Metaphor
44. A quick example of the “holy trinity” of configuration management. Example: Setup Apache
45. package "ntp" do action :install end template "/etc/ntp.conf" do source "ntp.conf.erb" owner "root" group "root" mode 0644 notifies :restart, resources(:service => node[:ntp][:service]) end service node[:ntp][:service] do action :start end
46. package "ntp" do action :install end template "/etc/ntp.conf" do source "ntp.conf.erb" owner "root" group "root" mode 0644 notifies :restart, resources(:service => node[:ntp][:service]) end service node[:ntp][:service] do action :start end Resource to Install Resource to Configure Resource to Start
47. package "ntp" do action :install end Install the Package
48. template "/etc/ntp.conf" do source "ntp.conf.erb" owner "root" group "root" mode 0644 notifies :restart, resources(:service => node[:ntp] [:service]) end Configure the Package
49. ntp.conf.erb <% node[:ntp][:servers].each do |ntpserver| -%> server <%= ntpserver %> iburst restrict <%= ntpserver %> nomodify notrap noquery <% end -%> A snippet from the ntp.conf.erb template
50. service node[:ntp][:service] do action :start end Start the Service
51. Roles are collections of Recipes (and roles) and Attributes that you assign to Nodes. Stacking them Up
52. name "ntp_client" description "Keep the clock in sync on these boxes" run_list( "recipe[ntp::default]" ) default_attributes( "ntp" => { "servers" => ["timeserver1.upenn.edu", "timeserver2.upenn.edu"] } ) ntp_client.rb
53. Server: “Test Box” Role: “ntp_client” Role: “ntp_client” Recipe: “ntp::default” Attributes: “ntp.servers” Recipe: “ntp::default” Resource: “package” Installs Software Resource: “template” Builds Config (using Attributes) Resource: “services” Starts Service
54. Server Role Recipe Resource Resource Resource Role Recipe Resource Resource Resource Resource Recipe Resource Recipe Recipe Resource Resource Resource Resource Resource Resource Resource Getting powerful, right?
55. http://www.flickr.com/photos/natalielucier/3620143737/ There’s so much more to Chef then I could possibly cover in 10 minutes!
56. % chef-client From 0 to configured.. And it’s likely that your nodes would be setup to run that for you!
57. + Vagrant has support for provisioning with Chef (and Puppet) out of the box.
58. config.vm.provision :chef_solo do |chef| chef.cookbooks_path = "~/my-cookbooks" chef.roles_path = "~/my-roles" chef.add_role "my_app_server" end Vagrantfile
59. “VIM is the bee’s knees!” - Developer A The Holy Wars
60. “Eclipse & Plugin X, Y, & Z are better then unicorn riding sharks!” - Developer B The Holy Wars Ok, maybe developers don’t actually talk like this..
61. “Let’s fight to the death!!” - Both Developers The Holy Wars Probably not likely..
62. “Let’s talk this thing to death in IRC!!” - Both Developers The Holy Wars Very likely.
63. Vagrantfile config.vm.share_folder "web", "/var/www", "~/my-site" config.vm.forward_port 80, 8080 Mapping folders into the running VM solves the problem. Use what you’d like. Additionally, you never actually have to log into the running VM.
64. “One more thing..” -Bill Gates
65. “One more thing..” -Steve Jobs Just making sure you’re still with me.
66. % vagrant destroy Despite all I’ve mentioned to this point, this may actually be my favorite vagrant command.
67. “vagrant destroy” is the hulk smash of virtual computing!
68. vagrant destroy Blow away the VM. But what’s left? Everything required to recreate it at will! Save space, fix problems, tinker at will.
69. Recap Powered by meme
71. Developer - Before Local Dev? Don’t want it on all the time. 1 VM? Mixing Libraries, Dependencies, etc. Multiple VMs: Painful to manage, large footprint
72. Developer - After A dedicated, disposable, consistent development environment... per project!
74. Organization - Before Fixed/shared dev? Developers like different tools Onboarding? Long time from hire to productivity Mismatched Envs: Leads to problems in production
75. Organization - After Providing consistent local development leads to smoother deployments, drastically shortens on-boarding time, and puts no regulations on developer tools.
77. cfstorytime I think that’s a ColdFusion joke.
78. How things fall apart.. “I’m ready to get started!” - New Hire
79. How things fall apart.. 1: Install the correct JDK. Note: Actual Directions!!
80. How things fall apart.. 2: Run the CF Installer Note: Actual Directions!!
81. How things fall apart.. 3: Run the 9.0.1 Installer Note: Actual Directions!!
82. How things fall apart.. 4: Download the two zip files that are part of "Cumulative Hot fix 2" Note: Actual Directions!!
83. How things fall apart.. 4a: Unzip one file, move a jar to a specific location, copy *.jar from the /lib folder to one place, copy *.properties from the /lib folder to another place Note: Actual Directions!!
84. How things fall apart.. 4b: Unzip the other file replacing some of your webroot contents Note: Actual Directions!!
85. How things fall apart.. 5: Download the latest hotfix, unzip, and move a .jar to a specific location Note: Actual Directions!!
86. How things fall apart.. “Ok, and that’s it?” - New Hire
87. How things fall apart.. Not pictured: 5-10 More Steps There’s SSL stuff, data sources, mapping shared libraries, etc.
88. How things fall apart.. “WTF?” - New Hire
89. Homer Liwag [CC-BY-SA-3.0-2.5-2.0-1.0 (www.creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons Ok, no actual magic, but when I was tasked with making some changes to one of our legacy systems, I wanted to do it without having to go through the previously described process.
90. Chef + Vagrant = ❤ Chef+Vagrant setup leads to Collaborative effort/forking leads to ColdFusion9 cookbook + internal repos Ended up working with coworkers from other groups to get this setup.
91. @Training: OS Consistency? 1/2 Windows 1/2 Mac It was time for a training, so I surveyed the room to get a feel for the state of people’s dev setups. I don’t think these results would be uncommon in many large organizations.
92. @Training: Software Consistency? Up to date Some hotfixes Base install
93. @Training: Frontend Consistency? IIS Apache Standalone
94. Tiny Victories! We’ve had some new employees use this process, and additionally, have been working toward getting some of the larger projects into this type of setup.
95. librarian
96. Bundler : Gems :: Librarian : _____ Remember These? Remember these from the SATs?
97. Bundler : Gems :: Librarian : Cookbooks Remember These? If you’re familiar with the ruby bundler, you’ll understand librarian immediately.
98. Cheffile site 'http://community.opscode.com/api/v1' cookbook 'apt' cookbook 'mysql' Can pull from the community site.
99. Cheffile # This branch looks nice! cookbook 'wordpress', :git => 'https://github.com/lewg/wordpress', :ref => 'all-the-things' Or from a specific branch of a git repo.
100. Distributing And a word of caution
101. “VCS or GTFU” - Mark Jaquith I saw Mark give a presentation a while back with a slide like this, and agreed with the sentiment, so I put it in a previous version of this presentation.
102. “DVCS or GTFU” - Lew But I wanted to enhance it slightly. Unfortunately when I mentioned it to him...
103. Whoops.
104. “VCS or GTFO” - Mark Jaquith Sorry about that! The correct quote. But I was curious, so I took it to the internet.
105. Grow the F*ck Up Used when adults are acting like kids. - urbandictionary.com GTFU Turns out it means something!
106. “DVCS or GTFU” - Lew So, it’s a little harsher then I would have gone with, but I feel like I’m stuck with it at this point. Also...
107. “Git or get out?” - Lew meh.. My other ideas were pretty terrible.
108. “Modern version control systems make these setups insanely easy to distribute and modify.” - Lew What I’m trying to say.. And you should be using them if you can.
109. % git clone http://my-setup % librarian-chef install % vagrant up How Easy? 3 Lines.
110. WordPress Another example from work. We’ve been doing a lot more work in WordPress lately.
111. The Famous 5-Minute Installation Which is great, if you’re running a blog. Once you start venturing into CMS territory, you’re adding a ton of stuff.
112. The Not-so-Famous Full Stack Install Web Server DB Server Plugins Custom Theme Matching Matching So it ends up looking more like this.
113. Install WordPress, plugins, themes, and map appropriate folders. http://github.com/lewg/wordpress-vagrant 1 Site = 1 Setup Repo
114. “Everybody is using it.” Goal: Provide easy to use directions to setup a dev environment for non- developers.
115. “You hacked the core?” Goal: Provide your setup to vendors with a clear acceptance path.
116. In Practice Consistent Local Development for our most complicated WordPress install.
117. What’s it do? Creates a new Ubuntu 12.04 VM Installs and configures all server software Apache, PHP, MySQL, Apt, Subversion Downloads & Installs WordPress
118. What’s it do? Creates a new Ubuntu 12.04 VM Installs and configures all server software Apache, PHP, MySQL, Apt, Subversion Downloads & Installs WordPress
119. What’s it do? Creates a new Ubuntu 12.04 VM Installs and configures all server software Apache, PHP, MySQL, Apt, Subversion Downloads & Installs WordPress
120. What’s it do? Installs 30 wordpress.org plugins Installs 3 paid plugins via mapped folder Maps in custom theme folder (own repo)
121. What’s it do? Installs 30 wordpress.org plugins Installs 3 paid plugins via mapped folders Maps in custom theme folder (own repo)
122. What’s it do? Installs 30 wordpress.org plugins Installs 3 paid plugins via mapped folder Maps in custom theme folder (own repo)
123. What does it REALLY do? Takes a complicated and time consuming process, and reduces it to a few minutes.
124. Observations About Vagrant, Chef, and Life in General
125. = This is an endorsement.
126. Build your Production Boxes with Chef Test your Production Cookbooks in Vagrant Vagrant to Demo New Features or Software Move to -dev with chef- solo or chef-server Once you start, at any point, you find yourself in this loop.
127. By Julius Schorzman (Own work) [CC-BY-SA-2.0 (http://creativecommons.org/licenses/by-sa/2.0)], via Wikimedia Commons This has nothing to do with technology, but we’re nearing the end.
128. 0 1 2 3 Number of Children Coffee Consuption Sleep Just an observation.
129. And the end result.. your desk looks like a meth lab!
130. Apt or Yum cache / Local or Datacenter Host your own base boxes Put files into your private Vagrant repos Cache all the things
131. Vagrant http://vagrantup.com @mitchellh VeeWee https://github.com/jedi4ever/veewee Git http://git-scm.com/ Ruby http://www.ruby-lang.org/ Chef http://opscode.com @opscode https://github.com/opscode https://github.com/opscode-cookbooks Librarian https://github.com/applicationsonline/librarian My Github Stuff https://github.com/lewg/