Consistent Local Development with Vagrant & Chef

Slide 1

Slide 1 text

Consistent Local Development with Vagrant & Chef Lew Goettner @lewg #PhillyDevOps July 18, 2012

Slide 2

Slide 2 text

about me Also, these are additional notes where I felt the slide needed a little explanation for Speaker Deck.

Slide 3

Slide 3 text

LEWIS JOSEPH GOETTNER THE @lewg I’m shooting at manual processes.

Slide 4

Slide 4 text

I work for the Wharton School.

Slide 5

Slide 5 text

By JSquish (Own work) [CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons This may come to mind...

Slide 6

Slide 6 text

By TexasDex at en.wikipedia [GFDL (http://www.gnu.org/copyleft/fdl.html) or CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0/)], from Wikimedia Commons Or perhaps this, if you think about the University of Pennsylvania.

Slide 7

Slide 7 text

Wharton Computing Our group of ~130 employees probably lands somewhere in here.

Slide 8

Slide 8 text

PHP WordPress ColdFusion Windows Ubuntu/Red Hat MSSQL MySQL Ruby My primary focus, and our mixed bag of technologies.

Slide 9

Slide 9 text

Supporting non-technical users Slower adoption of new technology Legacy systems There is always room for a better way! When you’re stuck in the middle..

Slide 10

Slide 10 text

storytime “The Generic Developer’s Path to a Perfect Development Setup”

Slide 11

Slide 11 text

“Here’s a shared dev space.” - Sysadmin the progression...

Slide 12

Slide 12 text

“I’ll just run it locally.” - Developer the progression... Except, I don’t really want on this crap running my machine all the time.

Slide 13

Slide 13 text

“I’ll run it in a VM!” - Developer the progression... But conﬁguring all this stuff isn’t all that fun.

Slide 14

Slide 14 text

“I’ll run it in a chef-configured VM!” - Developer the progression... Now my biggest gripe is the time it takes to install the OS.

Slide 15

Slide 15 text

“I’ll run it in a chef-configured VM, based on a post-install snapshot.” - Developer the progression... Now we’re talking!

Slide 16

Slide 16 text

“Why don’t we all start using this?” - Boss / Coworker / Intern the progression...

Slide 17

Slide 17 text

“Shit.” - Developer the progression...

Slide 18

Slide 18 text

vagrant

Slide 19

Slide 19 text

“Vagrant uses Oracle’s VirtualBox to build configurable, lightweight, and portable virtual machines dynamically.” - http://vagrantup.com

Slide 20

Slide 20 text

“I love the command line.” - Lew

Slide 21

Slide 21 text

% vagrant box add precise32 \ http://files.vagrantup.com/precise32.box % vagrant init precise32 % vagrant up 3 Lines = A Running VM

Slide 22

Slide 22 text

% vagrant box add precise32 \ http://files.vagrantup.com/precise32.box % vagrant init precise32 % vagrant up

Slide 23

Slide 23 text

Base OS Install Software for access & configuration VirtualBox Guest Additions Base Box

Slide 24

Slide 24 text

Built in tools for management: % vagrant box -h % vagrant package -h Base Box

Slide 25

Slide 25 text

VeeWee: tool for building your own https://github.com/jedi4ever/veewee Base Box

Slide 26

Slide 26 text

% vagrant box add precise32 \ http://files.vagrantup.com/precise32.box % vagrant init precise32 % vagrant up It’s a URL! So, create your own, and distribute via internet/intranet.

Slide 27

Slide 27 text

% vagrant box add precise32 \ http://files.vagrantup.com/precise32.box % vagrant init precise32 % vagrant up Drops a “Vagrantﬁle” into the current directory.

Slide 28

Slide 28 text

Vagrantfile = a Project Project = 1 to many VMs Vagrantfile

Slide 29

Slide 29 text

It’s just a text file Ruby DSL for configuring your VM(s) (no ruby skills required) Vagrantfile

Slide 30

Slide 30 text

% vagrant box add precise32 \ http://files.vagrantup.com/precise32.box % vagrant init precise32 % vagrant up

Slide 31

Slide 31 text

vagrant up http://www.ﬂickr.com/photos/lightningjeff/7505934038/

Slide 32

Slide 32 text

“I’m not impressed.” -Developer Well, there’s some powerful stuff there, but maybe you’ve already been doing your own version of this with VMWare, snapshots, etc.

Slide 33

Slide 33 text

No content

Slide 34

Slide 34 text

“Chef is an open-source systems integration framework built specifically for automating the cloud.” -http://www.opscode.com/chef/

Slide 35

Slide 35 text

“Chef means never hacking together shell scripts and/or dealing with config file hell again.” -Lew

Slide 36

Slide 36 text

Declarative: What, not how Idempotent: Only take action if required Convergent: Takes care of itself Configuration Management

Slide 37

Slide 37 text

You configure systems with Chef by writing self-documenting code. This code is lists of Resources that configure the system to do its job. Declarative Resources

Slide 38

Slide 38 text

package "bash" do action :install end

Slide 39

Slide 39 text

Chef Resources have Providers that take idempotent action to configure the resource, but only if it needs to change. Idempotent Actions

Slide 40

Slide 40 text

INFO: Processing package[apache2] action install (apache2::default line 20) DEBUG: package[apache2] checking package status for apache2 DEBUG: package[apache2] current version is 2.2.20-1ubuntu1.1 DEBUG: package[apache2] candidate version is 2.2.20-1ubuntu1.1 DEBUG: package[apache2] is already installed - nothing to do

Slide 41

Slide 41 text

Chef runs on the system, configuring the Node. In Chef, a single run should completely configure the system. Convergent Nodes

Slide 42

Slide 42 text

Chef Recipes are a pure Ruby domain specific language (DSL) Ruby Again!

Slide 43

Slide 43 text

Recipes are collected in Cookbooks along with associated components like config files or libraries. Continuing the Metaphor

Slide 44

Slide 44 text

A quick example of the “holy trinity” of configuration management. Example: Setup Apache

Slide 45

Slide 45 text

package "ntp" do action :install end template "/etc/ntp.conf" do source "ntp.conf.erb" owner "root" group "root" mode 0644 notifies :restart, resources(:service => node[:ntp][:service]) end service node[:ntp][:service] do action :start end

Slide 46

Slide 46 text

Slide 47

Slide 47 text

package "ntp" do action :install end Install the Package

Slide 48

Slide 48 text

template "/etc/ntp.conf" do source "ntp.conf.erb" owner "root" group "root" mode 0644 notifies :restart, resources(:service => node[:ntp] [:service]) end Configure the Package

Slide 49

Slide 49 text

ntp.conf.erb <% node[:ntp][:servers].each do |ntpserver| -%> server <%= ntpserver %> iburst restrict <%= ntpserver %> nomodify notrap noquery <% end -%> A snippet from the ntp.conf.erb template

Slide 50

Slide 50 text

service node[:ntp][:service] do action :start end Start the Service

Slide 51

Slide 51 text

Roles are collections of Recipes (and roles) and Attributes that you assign to Nodes. Stacking them Up

Slide 52

Slide 52 text

name "ntp_client" description "Keep the clock in sync on these boxes" run_list( "recipe[ntp::default]" ) default_attributes( "ntp" => { "servers" => ["timeserver1.upenn.edu", "timeserver2.upenn.edu"] } ) ntp_client.rb

Slide 53

Slide 53 text

Server: “Test Box” Role: “ntp_client” Role: “ntp_client” Recipe: “ntp::default” Attributes: “ntp.servers” Recipe: “ntp::default” Resource: “package” Installs Software Resource: “template” Builds Conﬁg (using Attributes) Resource: “services” Starts Service

Slide 54

Slide 54 text

Server Role Recipe Resource Resource Resource Role Recipe Resource Resource Resource Resource Recipe Resource Recipe Recipe Resource Resource Resource Resource Resource Resource Resource Getting powerful, right?

Slide 55

Slide 55 text

http://www.flickr.com/photos/natalielucier/3620143737/ There’s so much more to Chef then I could possibly cover in 10 minutes!

Slide 56

Slide 56 text

% chef-client From 0 to conﬁgured.. And it’s likely that your nodes would be setup to run that for you!

Slide 57

Slide 57 text

+ Vagrant has support for provisioning with Chef (and Puppet) out of the box.

Slide 58

Slide 58 text

config.vm.provision :chef_solo do |chef| chef.cookbooks_path = "~/my-cookbooks" chef.roles_path = "~/my-roles" chef.add_role "my_app_server" end Vagrantfile

Slide 59

Slide 59 text

“VIM is the bee’s knees!” - Developer A The Holy Wars

Slide 60

Slide 60 text

“Eclipse & Plugin X, Y, & Z are better then unicorn riding sharks!” - Developer B The Holy Wars Ok, maybe developers don’t actually talk like this..

Slide 61

Slide 61 text

“Let’s fight to the death!!” - Both Developers The Holy Wars Probably not likely..

Slide 62

Slide 62 text

“Let’s talk this thing to death in IRC!!” - Both Developers The Holy Wars Very likely.

Slide 63

Slide 63 text

Vagrantfile config.vm.share_folder "web", "/var/www", "~/my-site" config.vm.forward_port 80, 8080 Mapping folders into the running VM solves the problem. Use what you’d like. Additionally, you never actually have to log into the running VM.

Slide 64

Slide 64 text

“One more thing..” -Bill Gates

Slide 65

Slide 65 text

“One more thing..” -Steve Jobs Just making sure you’re still with me.

Slide 66

Slide 66 text

% vagrant destroy Despite all I’ve mentioned to this point, this may actually be my favorite vagrant command.

Slide 67

Slide 67 text

“vagrant destroy” is the hulk smash of virtual computing!

Slide 68

Slide 68 text

vagrant destroy Blow away the VM. But what’s left? Everything required to recreate it at will! Save space, fix problems, tinker at will.

Slide 69

Slide 69 text

Recap Powered by meme

Slide 70

Slide 70 text

No content

Slide 71

Slide 71 text

Developer - Before Local Dev? Don’t want it on all the time. 1 VM? Mixing Libraries, Dependencies, etc. Multiple VMs: Painful to manage, large footprint

Slide 72

Slide 72 text

Developer - After A dedicated, disposable, consistent development environment... per project!

Slide 73

Slide 73 text

No content

Slide 74

Slide 74 text

Organization - Before Fixed/shared dev? Developers like different tools Onboarding? Long time from hire to productivity Mismatched Envs: Leads to problems in production

Slide 75

Slide 75 text

Organization - After Providing consistent local development leads to smoother deployments, drastically shortens on-boarding time, and puts no regulations on developer tools.

Slide 76

Slide 76 text

No content

Slide 77

Slide 77 text

cfstorytime I think that’s a ColdFusion joke.

Slide 78

Slide 78 text

How things fall apart.. “I’m ready to get started!” - New Hire

Slide 79

Slide 79 text

How things fall apart.. 1: Install the correct JDK. Note: Actual Directions!!

Slide 80

Slide 80 text

How things fall apart.. 2: Run the CF Installer Note: Actual Directions!!

Slide 81

Slide 81 text

How things fall apart.. 3: Run the 9.0.1 Installer Note: Actual Directions!!

Slide 82

Slide 82 text

How things fall apart.. 4: Download the two zip files that are part of "Cumulative Hot fix 2" Note: Actual Directions!!

Slide 83

Slide 83 text

How things fall apart.. 4a: Unzip one file, move a jar to a specific location, copy *.jar from the /lib folder to one place, copy *.properties from the /lib folder to another place Note: Actual Directions!!

Slide 84

Slide 84 text

How things fall apart.. 4b: Unzip the other file replacing some of your webroot contents Note: Actual Directions!!

Slide 85

Slide 85 text

How things fall apart.. 5: Download the latest hotfix, unzip, and move a .jar to a specific location Note: Actual Directions!!

Slide 86

Slide 86 text

How things fall apart.. “Ok, and that’s it?” - New Hire

Slide 87

Slide 87 text

How things fall apart.. Not pictured: 5-10 More Steps There’s SSL stuff, data sources, mapping shared libraries, etc.

Slide 88

Slide 88 text

How things fall apart.. “WTF?” - New Hire

Slide 89

Slide 89 text

Homer Liwag [CC-BY-SA-3.0-2.5-2.0-1.0 (www.creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons Ok, no actual magic, but when I was tasked with making some changes to one of our legacy systems, I wanted to do it without having to go through the previously described process.

Slide 90

Slide 90 text

Chef + Vagrant = ❤ Chef+Vagrant setup leads to Collaborative effort/forking leads to ColdFusion9 cookbook + internal repos Ended up working with coworkers from other groups to get this setup.

Slide 91

Slide 91 text

@Training: OS Consistency? 1/2 Windows 1/2 Mac It was time for a training, so I surveyed the room to get a feel for the state of people’s dev setups. I don’t think these results would be uncommon in many large organizations.

Slide 92

Slide 92 text

@Training: Software Consistency? Up to date Some hotfixes Base install

Slide 93

Slide 93 text

@Training: Frontend Consistency? IIS Apache Standalone

Slide 94

Slide 94 text

Tiny Victories! We’ve had some new employees use this process, and additionally, have been working toward getting some of the larger projects into this type of setup.

Slide 95

Slide 95 text

librarian

Slide 96

Slide 96 text

Bundler : Gems :: Librarian : _____ Remember These? Remember these from the SATs?

Slide 97

Slide 97 text

Bundler : Gems :: Librarian : Cookbooks Remember These? If you’re familiar with the ruby bundler, you’ll understand librarian immediately.

Slide 98

Slide 98 text

Cheffile site 'http://community.opscode.com/api/v1' cookbook 'apt' cookbook 'mysql' Can pull from the community site.

Slide 99

Slide 99 text

Cheffile # This branch looks nice! cookbook 'wordpress', :git => 'https://github.com/lewg/wordpress', :ref => 'all-the-things' Or from a speciﬁc branch of a git repo.

Slide 100

Slide 100 text

Distributing And a word of caution

Slide 101

Slide 101 text

“VCS or GTFU” - Mark Jaquith I saw Mark give a presentation a while back with a slide like this, and agreed with the sentiment, so I put it in a previous version of this presentation.

Slide 102

Slide 102 text

“DVCS or GTFU” - Lew But I wanted to enhance it slightly. Unfortunately when I mentioned it to him...

Slide 103

Slide 103 text

Whoops.

Slide 104

Slide 104 text

“VCS or GTFO” - Mark Jaquith Sorry about that! The correct quote. But I was curious, so I took it to the internet.

Slide 105

Slide 105 text

Grow the F*ck Up Used when adults are acting like kids. - urbandictionary.com GTFU Turns out it means something!

Slide 106

Slide 106 text

“DVCS or GTFU” - Lew So, it’s a little harsher then I would have gone with, but I feel like I’m stuck with it at this point. Also...

Slide 107

Slide 107 text

“Git or get out?” - Lew meh.. My other ideas were pretty terrible.

Slide 108

Slide 108 text

“Modern version control systems make these setups insanely easy to distribute and modify.” - Lew What I’m trying to say.. And you should be using them if you can.

Slide 109

Slide 109 text

% git clone http://my-setup % librarian-chef install % vagrant up How Easy? 3 Lines.

Slide 110

Slide 110 text

WordPress Another example from work. We’ve been doing a lot more work in WordPress lately.

Slide 111

Slide 111 text

The Famous 5-Minute Installation Which is great, if you’re running a blog. Once you start venturing into CMS territory, you’re adding a ton of stuff.

Slide 112

Slide 112 text

The Not-so-Famous Full Stack Install Web Server DB Server Plugins Custom Theme Matching Matching So it ends up looking more like this.

Slide 113

Slide 113 text

Install WordPress, plugins, themes, and map appropriate folders. http://github.com/lewg/wordpress-vagrant 1 Site = 1 Setup Repo

Slide 114

Slide 114 text

“Everybody is using it.” Goal: Provide easy to use directions to setup a dev environment for non- developers.

Slide 115

Slide 115 text

“You hacked the core?” Goal: Provide your setup to vendors with a clear acceptance path.

Slide 116

Slide 116 text

In Practice Consistent Local Development for our most complicated WordPress install.

Slide 117

Slide 117 text

What’s it do? Creates a new Ubuntu 12.04 VM Installs and configures all server software Apache, PHP, MySQL, Apt, Subversion Downloads & Installs WordPress

Slide 118

Slide 118 text

What’s it do? Creates a new Ubuntu 12.04 VM Installs and configures all server software Apache, PHP, MySQL, Apt, Subversion Downloads & Installs WordPress

Slide 119

Slide 119 text

What’s it do? Creates a new Ubuntu 12.04 VM Installs and configures all server software Apache, PHP, MySQL, Apt, Subversion Downloads & Installs WordPress

Slide 120

Slide 120 text

What’s it do? Installs 30 wordpress.org plugins Installs 3 paid plugins via mapped folder Maps in custom theme folder (own repo)

Slide 121

Slide 121 text

What’s it do? Installs 30 wordpress.org plugins Installs 3 paid plugins via mapped folders Maps in custom theme folder (own repo)

Slide 122

Slide 122 text

What’s it do? Installs 30 wordpress.org plugins Installs 3 paid plugins via mapped folder Maps in custom theme folder (own repo)

Slide 123

Slide 123 text

What does it REALLY do? Takes a complicated and time consuming process, and reduces it to a few minutes.

Slide 124

Slide 124 text

Observations About Vagrant, Chef, and Life in General

Slide 125

Slide 125 text

= This is an endorsement.

Slide 126

Slide 126 text

Build your Production Boxes with Chef Test your Production Cookbooks in Vagrant Vagrant to Demo New Features or Software Move to -dev with chef- solo or chef-server Once you start, at any point, you ﬁnd yourself in this loop.

Slide 127

Slide 127 text

By Julius Schorzman (Own work) [CC-BY-SA-2.0 (http://creativecommons.org/licenses/by-sa/2.0)], via Wikimedia Commons This has nothing to do with technology, but we’re nearing the end.

Slide 128

Slide 128 text

0 1 2 3 Number of Children Coffee Consuption Sleep Just an observation.

Slide 129

Slide 129 text

And the end result.. your desk looks like a meth lab!

Slide 130

Slide 130 text

Apt or Yum cache / Local or Datacenter Host your own base boxes Put files into your private Vagrant repos Cache all the things

Slide 131

Slide 131 text

Vagrant http://vagrantup.com @mitchellh VeeWee https://github.com/jedi4ever/veewee Git http://git-scm.com/ Ruby http://www.ruby-lang.org/ Chef http://opscode.com @opscode https://github.com/opscode https://github.com/opscode-cookbooks Librarian https://github.com/applicationsonline/librarian My Github Stuff https://github.com/lewg/