Save 37% off PRO during our Black Friday Sale! »

Consistent Local Development with Vagrant & Chef

Consistent Local Development with Vagrant & Chef

A high level overview of some of these fantastic tools, and some examples of how we've been using them to improve local development environments at the office.

9ca1a1f5c872115a582ae57743139447?s=128

Lew Goettner

July 18, 2012
Tweet

Transcript

  1. Consistent Local Development with Vagrant & Chef Lew Goettner @lewg

    #PhillyDevOps July 18, 2012
  2. about me Also, these are additional notes where I felt

    the slide needed a little explanation for Speaker Deck.
  3. LEWIS JOSEPH GOETTNER THE @lewg I’m shooting at manual processes.

  4. I work for the Wharton School.

  5. By JSquish (Own work) [CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons This

    may come to mind...
  6. By TexasDex at en.wikipedia [GFDL (http://www.gnu.org/copyleft/fdl.html) or CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0/)], from

    Wikimedia Commons Or perhaps this, if you think about the University of Pennsylvania.
  7. Wharton Computing Our group of ~130 employees probably lands somewhere

    in here.
  8. PHP WordPress ColdFusion Windows Ubuntu/Red Hat MSSQL MySQL Ruby My

    primary focus, and our mixed bag of technologies.
  9. Supporting non-technical users Slower adoption of new technology Legacy systems

    There is always room for a better way! When you’re stuck in the middle..
  10. storytime “The Generic Developer’s Path to a Perfect Development Setup”

  11. “Here’s a shared dev space.” - Sysadmin the progression...

  12. “I’ll just run it locally.” - Developer the progression... Except,

    I don’t really want on this crap running my machine all the time.
  13. “I’ll run it in a VM!” - Developer the progression...

    But configuring all this stuff isn’t all that fun.
  14. “I’ll run it in a chef-configured VM!” - Developer the

    progression... Now my biggest gripe is the time it takes to install the OS.
  15. “I’ll run it in a chef-configured VM, based on a

    post-install snapshot.” - Developer the progression... Now we’re talking!
  16. “Why don’t we all start using this?” - Boss /

    Coworker / Intern the progression...
  17. “Shit.” - Developer the progression...

  18. vagrant

  19. “Vagrant uses Oracle’s VirtualBox to build configurable, lightweight, and portable

    virtual machines dynamically.” - http://vagrantup.com
  20. “I love the command line.” - Lew

  21. % vagrant box add precise32 \ http://files.vagrantup.com/precise32.box % vagrant init

    precise32 % vagrant up 3 Lines = A Running VM
  22. % vagrant box add precise32 \ http://files.vagrantup.com/precise32.box % vagrant init

    precise32 % vagrant up
  23. Base OS Install Software for access & configuration VirtualBox Guest

    Additions Base Box
  24. Built in tools for management: % vagrant box -h %

    vagrant package -h Base Box
  25. VeeWee: tool for building your own https://github.com/jedi4ever/veewee Base Box

  26. % vagrant box add precise32 \ http://files.vagrantup.com/precise32.box % vagrant init

    precise32 % vagrant up It’s a URL! So, create your own, and distribute via internet/intranet.
  27. % vagrant box add precise32 \ http://files.vagrantup.com/precise32.box % vagrant init

    precise32 % vagrant up Drops a “Vagrantfile” into the current directory.
  28. Vagrantfile = a Project Project = 1 to many VMs

    Vagrantfile
  29. It’s just a text file Ruby DSL for configuring your

    VM(s) (no ruby skills required) Vagrantfile
  30. % vagrant box add precise32 \ http://files.vagrantup.com/precise32.box % vagrant init

    precise32 % vagrant up
  31. vagrant up http://www.flickr.com/photos/lightningjeff/7505934038/

  32. “I’m not impressed.” -Developer Well, there’s some powerful stuff there,

    but maybe you’ve already been doing your own version of this with VMWare, snapshots, etc.
  33. None
  34. “Chef is an open-source systems integration framework built specifically for

    automating the cloud.” -http://www.opscode.com/chef/
  35. “Chef means never hacking together shell scripts and/or dealing with

    config file hell again.” -Lew
  36. Declarative: What, not how Idempotent: Only take action if required

    Convergent: Takes care of itself Configuration Management
  37. You configure systems with Chef by writing self-documenting code. This

    code is lists of Resources that configure the system to do its job. Declarative Resources
  38. package "bash" do action :install end

  39. Chef Resources have Providers that take idempotent action to configure

    the resource, but only if it needs to change. Idempotent Actions
  40. INFO: Processing package[apache2] action install (apache2::default line 20) DEBUG: package[apache2]

    checking package status for apache2 DEBUG: package[apache2] current version is 2.2.20-1ubuntu1.1 DEBUG: package[apache2] candidate version is 2.2.20-1ubuntu1.1 DEBUG: package[apache2] is already installed - nothing to do
  41. Chef runs on the system, configuring the Node. In Chef,

    a single run should completely configure the system. Convergent Nodes
  42. Chef Recipes are a pure Ruby domain specific language (DSL)

    Ruby Again!
  43. Recipes are collected in Cookbooks along with associated components like

    config files or libraries. Continuing the Metaphor
  44. A quick example of the “holy trinity” of configuration management.

    Example: Setup Apache
  45. package "ntp" do action :install end template "/etc/ntp.conf" do source

    "ntp.conf.erb" owner "root" group "root" mode 0644 notifies :restart, resources(:service => node[:ntp][:service]) end service node[:ntp][:service] do action :start end
  46. package "ntp" do action :install end template "/etc/ntp.conf" do source

    "ntp.conf.erb" owner "root" group "root" mode 0644 notifies :restart, resources(:service => node[:ntp][:service]) end service node[:ntp][:service] do action :start end Resource to Install Resource to Configure Resource to Start
  47. package "ntp" do action :install end Install the Package

  48. template "/etc/ntp.conf" do source "ntp.conf.erb" owner "root" group "root" mode

    0644 notifies :restart, resources(:service => node[:ntp] [:service]) end Configure the Package
  49. ntp.conf.erb <% node[:ntp][:servers].each do |ntpserver| -%> server <%= ntpserver %>

    iburst restrict <%= ntpserver %> nomodify notrap noquery <% end -%> A snippet from the ntp.conf.erb template
  50. service node[:ntp][:service] do action :start end Start the Service

  51. Roles are collections of Recipes (and roles) and Attributes that

    you assign to Nodes. Stacking them Up
  52. name "ntp_client" description "Keep the clock in sync on these

    boxes" run_list( "recipe[ntp::default]" ) default_attributes( "ntp" => { "servers" => ["timeserver1.upenn.edu", "timeserver2.upenn.edu"] } ) ntp_client.rb
  53. Server: “Test Box” Role: “ntp_client” Role: “ntp_client” Recipe: “ntp::default” Attributes:

    “ntp.servers” Recipe: “ntp::default” Resource: “package” Installs Software Resource: “template” Builds Config (using Attributes) Resource: “services” Starts Service
  54. Server Role Recipe Resource Resource Resource Role Recipe Resource Resource

    Resource Resource Recipe Resource Recipe Recipe Resource Resource Resource Resource Resource Resource Resource Getting powerful, right?
  55. http://www.flickr.com/photos/natalielucier/3620143737/ There’s so much more to Chef then I could

    possibly cover in 10 minutes!
  56. % chef-client From 0 to configured.. And it’s likely that

    your nodes would be setup to run that for you!
  57. + Vagrant has support for provisioning with Chef (and Puppet)

    out of the box.
  58. config.vm.provision :chef_solo do |chef| chef.cookbooks_path = "~/my-cookbooks" chef.roles_path = "~/my-roles"

    chef.add_role "my_app_server" end Vagrantfile
  59. “VIM is the bee’s knees!” - Developer A The Holy

    Wars
  60. “Eclipse & Plugin X, Y, & Z are better then

    unicorn riding sharks!” - Developer B The Holy Wars Ok, maybe developers don’t actually talk like this..
  61. “Let’s fight to the death!!” - Both Developers The Holy

    Wars Probably not likely..
  62. “Let’s talk this thing to death in IRC!!” - Both

    Developers The Holy Wars Very likely.
  63. Vagrantfile config.vm.share_folder "web", "/var/www", "~/my-site" config.vm.forward_port 80, 8080 Mapping folders

    into the running VM solves the problem. Use what you’d like. Additionally, you never actually have to log into the running VM.
  64. “One more thing..” -Bill Gates

  65. “One more thing..” -Steve Jobs Just making sure you’re still

    with me.
  66. % vagrant destroy Despite all I’ve mentioned to this point,

    this may actually be my favorite vagrant command.
  67. “vagrant destroy” is the hulk smash of virtual computing!

  68. vagrant destroy Blow away the VM. But what’s left? Everything

    required to recreate it at will! Save space, fix problems, tinker at will.
  69. Recap Powered by meme

  70. None
  71. Developer - Before Local Dev? Don’t want it on all

    the time. 1 VM? Mixing Libraries, Dependencies, etc. Multiple VMs: Painful to manage, large footprint
  72. Developer - After A dedicated, disposable, consistent development environment... per

    project!
  73. None
  74. Organization - Before Fixed/shared dev? Developers like different tools Onboarding?

    Long time from hire to productivity Mismatched Envs: Leads to problems in production
  75. Organization - After Providing consistent local development leads to smoother

    deployments, drastically shortens on-boarding time, and puts no regulations on developer tools.
  76. None
  77. cfstorytime I think that’s a ColdFusion joke.

  78. How things fall apart.. “I’m ready to get started!” -

    New Hire
  79. How things fall apart.. 1: Install the correct JDK. Note:

    Actual Directions!!
  80. How things fall apart.. 2: Run the CF Installer Note:

    Actual Directions!!
  81. How things fall apart.. 3: Run the 9.0.1 Installer Note:

    Actual Directions!!
  82. How things fall apart.. 4: Download the two zip files

    that are part of "Cumulative Hot fix 2" Note: Actual Directions!!
  83. How things fall apart.. 4a: Unzip one file, move a

    jar to a specific location, copy *.jar from the /lib folder to one place, copy *.properties from the /lib folder to another place Note: Actual Directions!!
  84. How things fall apart.. 4b: Unzip the other file replacing

    some of your webroot contents Note: Actual Directions!!
  85. How things fall apart.. 5: Download the latest hotfix, unzip,

    and move a .jar to a specific location Note: Actual Directions!!
  86. How things fall apart.. “Ok, and that’s it?” - New

    Hire
  87. How things fall apart.. Not pictured: 5-10 More Steps There’s

    SSL stuff, data sources, mapping shared libraries, etc.
  88. How things fall apart.. “WTF?” - New Hire

  89. Homer Liwag [CC-BY-SA-3.0-2.5-2.0-1.0 (www.creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons Ok, no actual

    magic, but when I was tasked with making some changes to one of our legacy systems, I wanted to do it without having to go through the previously described process.
  90. Chef + Vagrant = ❤ Chef+Vagrant setup leads to Collaborative

    effort/forking leads to ColdFusion9 cookbook + internal repos Ended up working with coworkers from other groups to get this setup.
  91. @Training: OS Consistency? 1/2 Windows 1/2 Mac It was time

    for a training, so I surveyed the room to get a feel for the state of people’s dev setups. I don’t think these results would be uncommon in many large organizations.
  92. @Training: Software Consistency? Up to date Some hotfixes Base install

  93. @Training: Frontend Consistency? IIS Apache Standalone

  94. Tiny Victories! We’ve had some new employees use this process,

    and additionally, have been working toward getting some of the larger projects into this type of setup.
  95. librarian

  96. Bundler : Gems :: Librarian : _____ Remember These? Remember

    these from the SATs?
  97. Bundler : Gems :: Librarian : Cookbooks Remember These? If

    you’re familiar with the ruby bundler, you’ll understand librarian immediately.
  98. Cheffile site 'http://community.opscode.com/api/v1' cookbook 'apt' cookbook 'mysql' Can pull from

    the community site.
  99. Cheffile # This branch looks nice! cookbook 'wordpress', :git =>

    'https://github.com/lewg/wordpress', :ref => 'all-the-things' Or from a specific branch of a git repo.
  100. Distributing And a word of caution

  101. “VCS or GTFU” - Mark Jaquith I saw Mark give

    a presentation a while back with a slide like this, and agreed with the sentiment, so I put it in a previous version of this presentation.
  102. “DVCS or GTFU” - Lew But I wanted to enhance

    it slightly. Unfortunately when I mentioned it to him...
  103. Whoops.

  104. “VCS or GTFO” - Mark Jaquith Sorry about that! The

    correct quote. But I was curious, so I took it to the internet.
  105. Grow the F*ck Up Used when adults are acting like

    kids. - urbandictionary.com GTFU Turns out it means something!
  106. “DVCS or GTFU” - Lew So, it’s a little harsher

    then I would have gone with, but I feel like I’m stuck with it at this point. Also...
  107. “Git or get out?” - Lew meh.. My other ideas

    were pretty terrible.
  108. “Modern version control systems make these setups insanely easy to

    distribute and modify.” - Lew What I’m trying to say.. And you should be using them if you can.
  109. % git clone http://my-setup % librarian-chef install % vagrant up

    How Easy? 3 Lines.
  110. WordPress Another example from work. We’ve been doing a lot

    more work in WordPress lately.
  111. The Famous 5-Minute Installation Which is great, if you’re running

    a blog. Once you start venturing into CMS territory, you’re adding a ton of stuff.
  112. The Not-so-Famous Full Stack Install Web Server DB Server Plugins

    Custom Theme Matching Matching So it ends up looking more like this.
  113. Install WordPress, plugins, themes, and map appropriate folders. http://github.com/lewg/wordpress-vagrant 1

    Site = 1 Setup Repo
  114. “Everybody is using it.” Goal: Provide easy to use directions

    to setup a dev environment for non- developers.
  115. “You hacked the core?” Goal: Provide your setup to vendors

    with a clear acceptance path.
  116. In Practice Consistent Local Development for our most complicated WordPress

    install.
  117. What’s it do? Creates a new Ubuntu 12.04 VM Installs

    and configures all server software Apache, PHP, MySQL, Apt, Subversion Downloads & Installs WordPress
  118. What’s it do? Creates a new Ubuntu 12.04 VM Installs

    and configures all server software Apache, PHP, MySQL, Apt, Subversion Downloads & Installs WordPress
  119. What’s it do? Creates a new Ubuntu 12.04 VM Installs

    and configures all server software Apache, PHP, MySQL, Apt, Subversion Downloads & Installs WordPress
  120. What’s it do? Installs 30 wordpress.org plugins Installs 3 paid

    plugins via mapped folder Maps in custom theme folder (own repo)
  121. What’s it do? Installs 30 wordpress.org plugins Installs 3 paid

    plugins via mapped folders Maps in custom theme folder (own repo)
  122. What’s it do? Installs 30 wordpress.org plugins Installs 3 paid

    plugins via mapped folder Maps in custom theme folder (own repo)
  123. What does it REALLY do? Takes a complicated and time

    consuming process, and reduces it to a few minutes.
  124. Observations About Vagrant, Chef, and Life in General

  125. = This is an endorsement.

  126. Build your Production Boxes with Chef Test your Production Cookbooks

    in Vagrant Vagrant to Demo New Features or Software Move to -dev with chef- solo or chef-server Once you start, at any point, you find yourself in this loop.
  127. By Julius Schorzman (Own work) [CC-BY-SA-2.0 (http://creativecommons.org/licenses/by-sa/2.0)], via Wikimedia Commons

    This has nothing to do with technology, but we’re nearing the end.
  128. 0 1 2 3 Number of Children Coffee Consuption Sleep

    Just an observation.
  129. And the end result.. your desk looks like a meth

    lab!
  130. Apt or Yum cache / Local or Datacenter Host your

    own base boxes Put files into your private Vagrant repos Cache all the things
  131. Vagrant http://vagrantup.com @mitchellh VeeWee https://github.com/jedi4ever/veewee Git http://git-scm.com/ Ruby http://www.ruby-lang.org/ Chef

    http://opscode.com @opscode https://github.com/opscode https://github.com/opscode-cookbooks Librarian https://github.com/applicationsonline/librarian My Github Stuff https://github.com/lewg/