Consistent Local Development with Vagrant & Chef

Transcript

1. Consistent Local Development with Vagrant & Chef Lew Goettner @lewg

   #PhillyDevOps July 18, 2012

2. about me Also, these are additional notes where I felt

   the slide needed a little explanation for Speaker Deck.

3. LEWIS JOSEPH GOETTNER THE @lewg I’m shooting at manual processes.

4. I work for the Wharton School.

5. By JSquish (Own work) [CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons This

   may come to mind...

6. By TexasDex at en.wikipedia [GFDL (http://www.gnu.org/copyleft/fdl.html) or CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0/)], from

   Wikimedia Commons Or perhaps this, if you think about the University of Pennsylvania.

7. Wharton Computing Our group of ~130 employees probably lands somewhere

   in here.

8. PHP WordPress ColdFusion Windows Ubuntu/Red Hat MSSQL MySQL Ruby My

   primary focus, and our mixed bag of technologies.

9. Supporting non-technical users Slower adoption of new technology Legacy systems

   There is always room for a better way! When you’re stuck in the middle..

10. storytime “The Generic Developer’s Path to a Perfect Development Setup”

11. “Here’s a shared dev space.” - Sysadmin the progression...

12. “I’ll just run it locally.” - Developer the progression... Except,

    I don’t really want on this crap running my machine all the time.

13. “I’ll run it in a VM!” - Developer the progression...

    But configuring all this stuff isn’t all that fun.

14. “I’ll run it in a chef-configured VM!” - Developer the

    progression... Now my biggest gripe is the time it takes to install the OS.

15. “I’ll run it in a chef-configured VM, based on a

    post-install snapshot.” - Developer the progression... Now we’re talking!

16. “Why don’t we all start using this?” - Boss /

    Coworker / Intern the progression...

17. “Shit.” - Developer the progression...

18. vagrant

19. “Vagrant uses Oracle’s VirtualBox to build configurable, lightweight, and portable

    virtual machines dynamically.” - http://vagrantup.com

20. “I love the command line.” - Lew

21. % vagrant box add precise32 \ http://files.vagrantup.com/precise32.box % vagrant init

    precise32 % vagrant up 3 Lines = A Running VM

22. % vagrant box add precise32 \ http://files.vagrantup.com/precise32.box % vagrant init

    precise32 % vagrant up

23. Base OS Install Software for access & configuration VirtualBox Guest

    Additions Base Box

24. Built in tools for management: % vagrant box -h %

    vagrant package -h Base Box

25. VeeWee: tool for building your own https://github.com/jedi4ever/veewee Base Box

26. % vagrant box add precise32 \ http://files.vagrantup.com/precise32.box % vagrant init

    precise32 % vagrant up It’s a URL! So, create your own, and distribute via internet/intranet.

27. % vagrant box add precise32 \ http://files.vagrantup.com/precise32.box % vagrant init

    precise32 % vagrant up Drops a “Vagrantfile” into the current directory.

28. Vagrantfile = a Project Project = 1 to many VMs

    Vagrantfile

29. It’s just a text file Ruby DSL for configuring your

    VM(s) (no ruby skills required) Vagrantfile

30. % vagrant box add precise32 \ http://files.vagrantup.com/precise32.box % vagrant init

    precise32 % vagrant up

31. vagrant up http://www.flickr.com/photos/lightningjeff/7505934038/

32. “I’m not impressed.” -Developer Well, there’s some powerful stuff there,

    but maybe you’ve already been doing your own version of this with VMWare, snapshots, etc.
33. None

34. “Chef is an open-source systems integration framework built specifically for

    automating the cloud.” -http://www.opscode.com/chef/

35. “Chef means never hacking together shell scripts and/or dealing with

    config file hell again.” -Lew

36. Declarative: What, not how Idempotent: Only take action if required

    Convergent: Takes care of itself Configuration Management

37. You configure systems with Chef by writing self-documenting code. This

    code is lists of Resources that configure the system to do its job. Declarative Resources

38. package "bash" do action :install end

39. Chef Resources have Providers that take idempotent action to configure

    the resource, but only if it needs to change. Idempotent Actions

40. INFO: Processing package[apache2] action install (apache2::default line 20) DEBUG: package[apache2]

    checking package status for apache2 DEBUG: package[apache2] current version is 2.2.20-1ubuntu1.1 DEBUG: package[apache2] candidate version is 2.2.20-1ubuntu1.1 DEBUG: package[apache2] is already installed - nothing to do

41. Chef runs on the system, configuring the Node. In Chef,

    a single run should completely configure the system. Convergent Nodes

42. Chef Recipes are a pure Ruby domain specific language (DSL)

    Ruby Again!

43. Recipes are collected in Cookbooks along with associated components like

    config files or libraries. Continuing the Metaphor

44. A quick example of the “holy trinity” of configuration management.

    Example: Setup Apache

45. package "ntp" do action :install end template "/etc/ntp.conf" do source

    "ntp.conf.erb" owner "root" group "root" mode 0644 notifies :restart, resources(:service => node[:ntp][:service]) end service node[:ntp][:service] do action :start end

46. package "ntp" do action :install end template "/etc/ntp.conf" do source

    "ntp.conf.erb" owner "root" group "root" mode 0644 notifies :restart, resources(:service => node[:ntp][:service]) end service node[:ntp][:service] do action :start end Resource to Install Resource to Configure Resource to Start

47. package "ntp" do action :install end Install the Package

48. template "/etc/ntp.conf" do source "ntp.conf.erb" owner "root" group "root" mode

    0644 notifies :restart, resources(:service => node[:ntp] [:service]) end Configure the Package

49. ntp.conf.erb <% node[:ntp][:servers].each do |ntpserver| -%> server <%= ntpserver %>

    iburst restrict <%= ntpserver %> nomodify notrap noquery <% end -%> A snippet from the ntp.conf.erb template

50. service node[:ntp][:service] do action :start end Start the Service

51. Roles are collections of Recipes (and roles) and Attributes that

    you assign to Nodes. Stacking them Up

52. name "ntp_client" description "Keep the clock in sync on these

    boxes" run_list( "recipe[ntp::default]" ) default_attributes( "ntp" => { "servers" => ["timeserver1.upenn.edu", "timeserver2.upenn.edu"] } ) ntp_client.rb

53. Server: “Test Box” Role: “ntp_client” Role: “ntp_client” Recipe: “ntp::default” Attributes:

    “ntp.servers” Recipe: “ntp::default” Resource: “package” Installs Software Resource: “template” Builds Config (using Attributes) Resource: “services” Starts Service

54. Server Role Recipe Resource Resource Resource Role Recipe Resource Resource

    Resource Resource Recipe Resource Recipe Recipe Resource Resource Resource Resource Resource Resource Resource Getting powerful, right?

55. http://www.flickr.com/photos/natalielucier/3620143737/ There’s so much more to Chef then I could

    possibly cover in 10 minutes!

56. % chef-client From 0 to configured.. And it’s likely that

    your nodes would be setup to run that for you!

57. + Vagrant has support for provisioning with Chef (and Puppet)

    out of the box.

58. config.vm.provision :chef_solo do |chef| chef.cookbooks_path = "~/my-cookbooks" chef.roles_path = "~/my-roles"

    chef.add_role "my_app_server" end Vagrantfile

59. “VIM is the bee’s knees!” - Developer A The Holy

    Wars

60. “Eclipse & Plugin X, Y, & Z are better then

    unicorn riding sharks!” - Developer B The Holy Wars Ok, maybe developers don’t actually talk like this..

61. “Let’s fight to the death!!” - Both Developers The Holy

    Wars Probably not likely..

62. “Let’s talk this thing to death in IRC!!” - Both

    Developers The Holy Wars Very likely.

63. Vagrantfile config.vm.share_folder "web", "/var/www", "~/my-site" config.vm.forward_port 80, 8080 Mapping folders

    into the running VM solves the problem. Use what you’d like. Additionally, you never actually have to log into the running VM.

64. “One more thing..” -Bill Gates

65. “One more thing..” -Steve Jobs Just making sure you’re still

    with me.

66. % vagrant destroy Despite all I’ve mentioned to this point,

    this may actually be my favorite vagrant command.

67. “vagrant destroy” is the hulk smash of virtual computing!

68. vagrant destroy Blow away the VM. But what’s left? Everything

    required to recreate it at will! Save space, fix problems, tinker at will.

69. Recap Powered by meme

70. None

71. Developer - Before Local Dev? Don’t want it on all

    the time. 1 VM? Mixing Libraries, Dependencies, etc. Multiple VMs: Painful to manage, large footprint

72. Developer - After A dedicated, disposable, consistent development environment... per

    project!
73. None

74. Organization - Before Fixed/shared dev? Developers like different tools Onboarding?

    Long time from hire to productivity Mismatched Envs: Leads to problems in production

75. Organization - After Providing consistent local development leads to smoother

    deployments, drastically shortens on-boarding time, and puts no regulations on developer tools.
76. None

77. cfstorytime I think that’s a ColdFusion joke.

78. How things fall apart.. “I’m ready to get started!” -

    New Hire

79. How things fall apart.. 1: Install the correct JDK. Note:

    Actual Directions!!

80. How things fall apart.. 2: Run the CF Installer Note:

    Actual Directions!!

81. How things fall apart.. 3: Run the 9.0.1 Installer Note:

    Actual Directions!!

82. How things fall apart.. 4: Download the two zip files

    that are part of "Cumulative Hot fix 2" Note: Actual Directions!!

83. How things fall apart.. 4a: Unzip one file, move a

    jar to a specific location, copy *.jar from the /lib folder to one place, copy *.properties from the /lib folder to another place Note: Actual Directions!!

84. How things fall apart.. 4b: Unzip the other file replacing

    some of your webroot contents Note: Actual Directions!!

85. How things fall apart.. 5: Download the latest hotfix, unzip,

    and move a .jar to a specific location Note: Actual Directions!!

86. How things fall apart.. “Ok, and that’s it?” - New

    Hire

87. How things fall apart.. Not pictured: 5-10 More Steps There’s

    SSL stuff, data sources, mapping shared libraries, etc.

88. How things fall apart.. “WTF?” - New Hire

89. Homer Liwag [CC-BY-SA-3.0-2.5-2.0-1.0 (www.creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons Ok, no actual

    magic, but when I was tasked with making some changes to one of our legacy systems, I wanted to do it without having to go through the previously described process.

90. Chef + Vagrant = ❤ Chef+Vagrant setup leads to Collaborative

    effort/forking leads to ColdFusion9 cookbook + internal repos Ended up working with coworkers from other groups to get this setup.

91. @Training: OS Consistency? 1/2 Windows 1/2 Mac It was time

    for a training, so I surveyed the room to get a feel for the state of people’s dev setups. I don’t think these results would be uncommon in many large organizations.

92. @Training: Software Consistency? Up to date Some hotfixes Base install

93. @Training: Frontend Consistency? IIS Apache Standalone

94. Tiny Victories! We’ve had some new employees use this process,

    and additionally, have been working toward getting some of the larger projects into this type of setup.

95. librarian

96. Bundler : Gems :: Librarian : _____ Remember These? Remember

    these from the SATs?

97. Bundler : Gems :: Librarian : Cookbooks Remember These? If

    you’re familiar with the ruby bundler, you’ll understand librarian immediately.

98. Cheffile site 'http://community.opscode.com/api/v1' cookbook 'apt' cookbook 'mysql' Can pull from

    the community site.

99. Cheffile # This branch looks nice! cookbook 'wordpress', :git =>

    'https://github.com/lewg/wordpress', :ref => 'all-the-things' Or from a specific branch of a git repo.

100. Distributing And a word of caution

101. “VCS or GTFU” - Mark Jaquith I saw Mark give

     a presentation a while back with a slide like this, and agreed with the sentiment, so I put it in a previous version of this presentation.

102. “DVCS or GTFU” - Lew But I wanted to enhance

     it slightly. Unfortunately when I mentioned it to him...

103. Whoops.

104. “VCS or GTFO” - Mark Jaquith Sorry about that! The

     correct quote. But I was curious, so I took it to the internet.

105. Grow the F*ck Up Used when adults are acting like

     kids. - urbandictionary.com GTFU Turns out it means something!

106. “DVCS or GTFU” - Lew So, it’s a little harsher

     then I would have gone with, but I feel like I’m stuck with it at this point. Also...

107. “Git or get out?” - Lew meh.. My other ideas

     were pretty terrible.

108. “Modern version control systems make these setups insanely easy to

     distribute and modify.” - Lew What I’m trying to say.. And you should be using them if you can.

109. % git clone http://my-setup % librarian-chef install % vagrant up

     How Easy? 3 Lines.

110. WordPress Another example from work. We’ve been doing a lot

     more work in WordPress lately.

111. The Famous 5-Minute Installation Which is great, if you’re running

     a blog. Once you start venturing into CMS territory, you’re adding a ton of stuff.

112. The Not-so-Famous Full Stack Install Web Server DB Server Plugins

     Custom Theme Matching Matching So it ends up looking more like this.

113. Install WordPress, plugins, themes, and map appropriate folders. http://github.com/lewg/wordpress-vagrant 1

     Site = 1 Setup Repo

114. “Everybody is using it.” Goal: Provide easy to use directions

     to setup a dev environment for non- developers.

115. “You hacked the core?” Goal: Provide your setup to vendors

     with a clear acceptance path.

116. In Practice Consistent Local Development for our most complicated WordPress

     install.

117. What’s it do? Creates a new Ubuntu 12.04 VM Installs

     and configures all server software Apache, PHP, MySQL, Apt, Subversion Downloads & Installs WordPress

118. What’s it do? Creates a new Ubuntu 12.04 VM Installs

     and configures all server software Apache, PHP, MySQL, Apt, Subversion Downloads & Installs WordPress

119. What’s it do? Creates a new Ubuntu 12.04 VM Installs

     and configures all server software Apache, PHP, MySQL, Apt, Subversion Downloads & Installs WordPress

120. What’s it do? Installs 30 wordpress.org plugins Installs 3 paid

     plugins via mapped folder Maps in custom theme folder (own repo)

121. What’s it do? Installs 30 wordpress.org plugins Installs 3 paid

     plugins via mapped folders Maps in custom theme folder (own repo)

122. What’s it do? Installs 30 wordpress.org plugins Installs 3 paid

     plugins via mapped folder Maps in custom theme folder (own repo)

123. What does it REALLY do? Takes a complicated and time

     consuming process, and reduces it to a few minutes.

124. Observations About Vagrant, Chef, and Life in General

125. = This is an endorsement.

126. Build your Production Boxes with Chef Test your Production Cookbooks

     in Vagrant Vagrant to Demo New Features or Software Move to -dev with chef- solo or chef-server Once you start, at any point, you find yourself in this loop.

127. By Julius Schorzman (Own work) [CC-BY-SA-2.0 (http://creativecommons.org/licenses/by-sa/2.0)], via Wikimedia Commons

     This has nothing to do with technology, but we’re nearing the end.

128. 0 1 2 3 Number of Children Coffee Consuption Sleep

     Just an observation.

129. And the end result.. your desk looks like a meth

     lab!

130. Apt or Yum cache / Local or Datacenter Host your

     own base boxes Put files into your private Vagrant repos Cache all the things

131. Vagrant http://vagrantup.com @mitchellh VeeWee https://github.com/jedi4ever/veewee Git http://git-scm.com/ Ruby http://www.ruby-lang.org/ Chef

     http://opscode.com @opscode https://github.com/opscode https://github.com/opscode-cookbooks Librarian https://github.com/applicationsonline/librarian My Github Stuff https://github.com/lewg/