Consistent Local Development with Vagrant & Chef

Consistent Local Development with Vagrant & Chef Lew Goettner @lewg

#PhillyDevOps July 18, 2012

about me Also, these are additional notes where I felt

the slide needed a little explanation for Speaker Deck.

LEWIS JOSEPH GOETTNER THE @lewg I’m shooting at manual processes.

I work for the Wharton School.

By JSquish (Own work) [CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons This

may come to mind...

By TexasDex at en.wikipedia [GFDL (http://www.gnu.org/copyleft/fdl.html) or CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0/)], from

Wikimedia Commons Or perhaps this, if you think about the University of Pennsylvania.

Wharton Computing Our group of ~130 employees probably lands somewhere

in here.

PHP WordPress ColdFusion Windows Ubuntu/Red Hat MSSQL MySQL Ruby My

primary focus, and our mixed bag of technologies.

Supporting non-technical users Slower adoption of new technology Legacy systems

There is always room for a better way! When you’re stuck in the middle..

storytime “The Generic Developer’s Path to a Perfect Development Setup”

“Here’s a shared dev space.” - Sysadmin the progression...

“I’ll just run it locally.” - Developer the progression... Except,

I don’t really want on this crap running my machine all the time.

“I’ll run it in a VM!” - Developer the progression...

But configuring all this stuff isn’t all that fun.

“I’ll run it in a chef-configured VM!” - Developer the

progression... Now my biggest gripe is the time it takes to install the OS.

“I’ll run it in a chef-configured VM, based on a

post-install snapshot.” - Developer the progression... Now we’re talking!

“Why don’t we all start using this?” - Boss /

Coworker / Intern the progression...

“Shit.” - Developer the progression...

vagrant

“Vagrant uses Oracle’s VirtualBox to build configurable, lightweight, and portable

virtual machines dynamically.” - http://vagrantup.com

“I love the command line.” - Lew

% vagrant box add precise32 \ http://files.vagrantup.com/precise32.box % vagrant init

precise32 % vagrant up 3 Lines = A Running VM

% vagrant box add precise32 \ http://files.vagrantup.com/precise32.box % vagrant init

precise32 % vagrant up

Base OS Install Software for access & configuration VirtualBox Guest

Additions Base Box

Built in tools for management: % vagrant box -h %

vagrant package -h Base Box

VeeWee: tool for building your own https://github.com/jedi4ever/veewee Base Box

% vagrant box add precise32 \ http://files.vagrantup.com/precise32.box % vagrant init

precise32 % vagrant up It’s a URL! So, create your own, and distribute via internet/intranet.

% vagrant box add precise32 \ http://files.vagrantup.com/precise32.box % vagrant init

precise32 % vagrant up Drops a “Vagrantfile” into the current directory.

Vagrantfile = a Project Project = 1 to many VMs

Vagrantfile

It’s just a text file Ruby DSL for configuring your

VM(s) (no ruby skills required) Vagrantfile

% vagrant box add precise32 \ http://files.vagrantup.com/precise32.box % vagrant init

precise32 % vagrant up

vagrant up http://www.flickr.com/photos/lightningjeff/7505934038/

“I’m not impressed.” -Developer Well, there’s some powerful stuff there,

but maybe you’ve already been doing your own version of this with VMWare, snapshots, etc.

None

“Chef is an open-source systems integration framework built specifically for

automating the cloud.” -http://www.opscode.com/chef/

“Chef means never hacking together shell scripts and/or dealing with

config file hell again.” -Lew

Declarative: What, not how Idempotent: Only take action if required

Convergent: Takes care of itself Configuration Management

You configure systems with Chef by writing self-documenting code. This

code is lists of Resources that configure the system to do its job. Declarative Resources

package "bash" do action :install end

Chef Resources have Providers that take idempotent action to configure

the resource, but only if it needs to change. Idempotent Actions

INFO: Processing package[apache2] action install (apache2::default line 20) DEBUG: package[apache2]

checking package status for apache2 DEBUG: package[apache2] current version is 2.2.20-1ubuntu1.1 DEBUG: package[apache2] candidate version is 2.2.20-1ubuntu1.1 DEBUG: package[apache2] is already installed - nothing to do

Chef runs on the system, configuring the Node. In Chef,

a single run should completely configure the system. Convergent Nodes

Chef Recipes are a pure Ruby domain specific language (DSL)

Ruby Again!

Recipes are collected in Cookbooks along with associated components like

config files or libraries. Continuing the Metaphor

A quick example of the “holy trinity” of configuration management.

Example: Setup Apache

package "ntp" do action :install end template "/etc/ntp.conf" do source

"ntp.conf.erb" owner "root" group "root" mode 0644 notifies :restart, resources(:service => node[:ntp][:service]) end service node[:ntp][:service] do action :start end

package "ntp" do action :install end template "/etc/ntp.conf" do source

"ntp.conf.erb" owner "root" group "root" mode 0644 notifies :restart, resources(:service => node[:ntp][:service]) end service node[:ntp][:service] do action :start end Resource to Install Resource to Configure Resource to Start

package "ntp" do action :install end Install the Package

template "/etc/ntp.conf" do source "ntp.conf.erb" owner "root" group "root" mode

0644 notifies :restart, resources(:service => node[:ntp] [:service]) end Configure the Package

ntp.conf.erb <% node[:ntp][:servers].each do |ntpserver| -%> server <%= ntpserver %>

iburst restrict <%= ntpserver %> nomodify notrap noquery <% end -%> A snippet from the ntp.conf.erb template

service node[:ntp][:service] do action :start end Start the Service

Roles are collections of Recipes (and roles) and Attributes that

you assign to Nodes. Stacking them Up

name "ntp_client" description "Keep the clock in sync on these

boxes" run_list( "recipe[ntp::default]" ) default_attributes( "ntp" => { "servers" => ["timeserver1.upenn.edu", "timeserver2.upenn.edu"] } ) ntp_client.rb

Server: “Test Box” Role: “ntp_client” Role: “ntp_client” Recipe: “ntp::default” Attributes:

“ntp.servers” Recipe: “ntp::default” Resource: “package” Installs Software Resource: “template” Builds Config (using Attributes) Resource: “services” Starts Service

Server Role Recipe Resource Resource Resource Role Recipe Resource Resource

Resource Resource Recipe Resource Recipe Recipe Resource Resource Resource Resource Resource Resource Resource Getting powerful, right?

http://www.flickr.com/photos/natalielucier/3620143737/ There’s so much more to Chef then I could

possibly cover in 10 minutes!

% chef-client From 0 to configured.. And it’s likely that

your nodes would be setup to run that for you!

+ Vagrant has support for provisioning with Chef (and Puppet)

out of the box.

config.vm.provision :chef_solo do |chef| chef.cookbooks_path = "~/my-cookbooks" chef.roles_path = "~/my-roles"

chef.add_role "my_app_server" end Vagrantfile

“VIM is the bee’s knees!” - Developer A The Holy

Wars

“Eclipse & Plugin X, Y, & Z are better then

unicorn riding sharks!” - Developer B The Holy Wars Ok, maybe developers don’t actually talk like this..

“Let’s fight to the death!!” - Both Developers The Holy

Wars Probably not likely..

“Let’s talk this thing to death in IRC!!” - Both

Developers The Holy Wars Very likely.

Vagrantfile config.vm.share_folder "web", "/var/www", "~/my-site" config.vm.forward_port 80, 8080 Mapping folders

into the running VM solves the problem. Use what you’d like. Additionally, you never actually have to log into the running VM.

“One more thing..” -Bill Gates

“One more thing..” -Steve Jobs Just making sure you’re still

with me.

% vagrant destroy Despite all I’ve mentioned to this point,

this may actually be my favorite vagrant command.

“vagrant destroy” is the hulk smash of virtual computing!

vagrant destroy Blow away the VM. But what’s left? Everything

required to recreate it at will! Save space, fix problems, tinker at will.

Recap Powered by meme

None

Developer - Before Local Dev? Don’t want it on all

the time. 1 VM? Mixing Libraries, Dependencies, etc. Multiple VMs: Painful to manage, large footprint

Developer - After A dedicated, disposable, consistent development environment... per

project!

None

Organization - Before Fixed/shared dev? Developers like different tools Onboarding?

Long time from hire to productivity Mismatched Envs: Leads to problems in production

Organization - After Providing consistent local development leads to smoother

deployments, drastically shortens on-boarding time, and puts no regulations on developer tools.

None

cfstorytime I think that’s a ColdFusion joke.

How things fall apart.. “I’m ready to get started!” -

New Hire

How things fall apart.. 1: Install the correct JDK. Note:

Actual Directions!!

How things fall apart.. 2: Run the CF Installer Note:

Actual Directions!!

How things fall apart.. 3: Run the 9.0.1 Installer Note:

Actual Directions!!

How things fall apart.. 4: Download the two zip files

that are part of "Cumulative Hot fix 2" Note: Actual Directions!!

How things fall apart.. 4a: Unzip one file, move a

jar to a specific location, copy *.jar from the /lib folder to one place, copy *.properties from the /lib folder to another place Note: Actual Directions!!

How things fall apart.. 4b: Unzip the other file replacing

some of your webroot contents Note: Actual Directions!!

How things fall apart.. 5: Download the latest hotfix, unzip,

and move a .jar to a specific location Note: Actual Directions!!

How things fall apart.. “Ok, and that’s it?” - New

Hire

How things fall apart.. Not pictured: 5-10 More Steps There’s

SSL stuff, data sources, mapping shared libraries, etc.

How things fall apart.. “WTF?” - New Hire

Homer Liwag [CC-BY-SA-3.0-2.5-2.0-1.0 (www.creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons Ok, no actual

magic, but when I was tasked with making some changes to one of our legacy systems, I wanted to do it without having to go through the previously described process.

Chef + Vagrant = ❤ Chef+Vagrant setup leads to Collaborative

effort/forking leads to ColdFusion9 cookbook + internal repos Ended up working with coworkers from other groups to get this setup.

@Training: OS Consistency? 1/2 Windows 1/2 Mac It was time

for a training, so I surveyed the room to get a feel for the state of people’s dev setups. I don’t think these results would be uncommon in many large organizations.

@Training: Software Consistency? Up to date Some hotfixes Base install

@Training: Frontend Consistency? IIS Apache Standalone

Tiny Victories! We’ve had some new employees use this process,

and additionally, have been working toward getting some of the larger projects into this type of setup.

librarian

Bundler : Gems :: Librarian : _____ Remember These? Remember

these from the SATs?

Bundler : Gems :: Librarian : Cookbooks Remember These? If

you’re familiar with the ruby bundler, you’ll understand librarian immediately.

Cheffile site 'http://community.opscode.com/api/v1' cookbook 'apt' cookbook 'mysql' Can pull from

the community site.

Cheffile # This branch looks nice! cookbook 'wordpress', :git =>

'https://github.com/lewg/wordpress', :ref => 'all-the-things' Or from a specific branch of a git repo.

Distributing And a word of caution

“VCS or GTFU” - Mark Jaquith I saw Mark give

a presentation a while back with a slide like this, and agreed with the sentiment, so I put it in a previous version of this presentation.

“DVCS or GTFU” - Lew But I wanted to enhance

it slightly. Unfortunately when I mentioned it to him...

Whoops.

“VCS or GTFO” - Mark Jaquith Sorry about that! The

correct quote. But I was curious, so I took it to the internet.

Grow the F*ck Up Used when adults are acting like

kids. - urbandictionary.com GTFU Turns out it means something!

“DVCS or GTFU” - Lew So, it’s a little harsher

then I would have gone with, but I feel like I’m stuck with it at this point. Also...

“Git or get out?” - Lew meh.. My other ideas

were pretty terrible.

“Modern version control systems make these setups insanely easy to

distribute and modify.” - Lew What I’m trying to say.. And you should be using them if you can.

% git clone http://my-setup % librarian-chef install % vagrant up

How Easy? 3 Lines.

WordPress Another example from work. We’ve been doing a lot

more work in WordPress lately.

The Famous 5-Minute Installation Which is great, if you’re running

a blog. Once you start venturing into CMS territory, you’re adding a ton of stuff.

The Not-so-Famous Full Stack Install Web Server DB Server Plugins

Custom Theme Matching Matching So it ends up looking more like this.

Install WordPress, plugins, themes, and map appropriate folders. http://github.com/lewg/wordpress-vagrant 1

Site = 1 Setup Repo

“Everybody is using it.” Goal: Provide easy to use directions

to setup a dev environment for non- developers.

“You hacked the core?” Goal: Provide your setup to vendors

with a clear acceptance path.

In Practice Consistent Local Development for our most complicated WordPress

install.

What’s it do? Creates a new Ubuntu 12.04 VM Installs

and configures all server software Apache, PHP, MySQL, Apt, Subversion Downloads & Installs WordPress

What’s it do? Creates a new Ubuntu 12.04 VM Installs

and configures all server software Apache, PHP, MySQL, Apt, Subversion Downloads & Installs WordPress

What’s it do? Creates a new Ubuntu 12.04 VM Installs

and configures all server software Apache, PHP, MySQL, Apt, Subversion Downloads & Installs WordPress

What’s it do? Installs 30 wordpress.org plugins Installs 3 paid

plugins via mapped folder Maps in custom theme folder (own repo)

What’s it do? Installs 30 wordpress.org plugins Installs 3 paid

plugins via mapped folders Maps in custom theme folder (own repo)

What’s it do? Installs 30 wordpress.org plugins Installs 3 paid

plugins via mapped folder Maps in custom theme folder (own repo)

What does it REALLY do? Takes a complicated and time

consuming process, and reduces it to a few minutes.

Observations About Vagrant, Chef, and Life in General

= This is an endorsement.

Build your Production Boxes with Chef Test your Production Cookbooks

in Vagrant Vagrant to Demo New Features or Software Move to -dev with chef- solo or chef-server Once you start, at any point, you find yourself in this loop.

By Julius Schorzman (Own work) [CC-BY-SA-2.0 (http://creativecommons.org/licenses/by-sa/2.0)], via Wikimedia Commons

This has nothing to do with technology, but we’re nearing the end.

0 1 2 3 Number of Children Coffee Consuption Sleep

Just an observation.

And the end result.. your desk looks like a meth

lab!

Apt or Yum cache / Local or Datacenter Host your

own base boxes Put files into your private Vagrant repos Cache all the things

Vagrant http://vagrantup.com @mitchellh VeeWee https://github.com/jedi4ever/veewee Git http://git-scm.com/ Ruby http://www.ruby-lang.org/ Chef

http://opscode.com @opscode https://github.com/opscode https://github.com/opscode-cookbooks Librarian https://github.com/applicationsonline/librarian My Github Stuff https://github.com/lewg/