Virtualize your stack with Vagrant and Puppet - Columbus Code Camp 2013 Edition

Transcript

1. Virtualize Your Stack With Vagrant and Puppet Jacob Mather Software

   Engineer at Mashery

2. Virtualize your environment with Vagrant and Puppet - Columbus Code

   Camp - October 12th, 2013 Who am I? What I do: • Software Engineer at Mashery • Co-organizer of the San Francisco PHP User Group • Community Evangelist Where I can be found: • Blog: http://jmather.com • Twitter: @thejmather

3. Virtualize your environment with Vagrant and Puppet - Columbus Code

   Camp - October 12th, 2013 Why I am giving this talk

4. Let’s talk about Vagrant

5. Virtualize your environment with Vagrant and Puppet - Columbus Code

   Camp - October 12th, 2013 What is Vagrant? A tool that allows you to create and configure lightweight, reproducible, and portable development environments.

6. Virtualize your environment with Vagrant and Puppet - Columbus Code

   Camp - October 12th, 2013 Vagrant is built... • in Ruby • to be simple

7. Virtualize your environment with Vagrant and Puppet - Columbus Code

   Camp - October 12th, 2013 Vagrant terms • Box, or base box, is the ‘initial image’ of a virtual machine. • Host, or your computer, the machine running the VMs • Guest, or vm, referring to an individual VM instance • Provider, an adapter to a VMS (like VirtualBox or VMWare) • Provisioner, provides post-boot configuration of guests

8. Virtualize your environment with Vagrant and Puppet - Columbus Code

   Camp - October 12th, 2013 How to use Vagrant • Install Vagrant - http://vagrantup.com • Check out your repository • Run: vagrant up

9. Virtualize your environment with Vagrant and Puppet - Columbus Code

   Camp - October 12th, 2013 Basic Vagrant Commands • vagrant up - Turn guest on • vagrant halt - Turn guest off • vagrant status - Show guest status • vagrant destroy - Delete guest • vagrant suspend - Suspend guest • vagrant resume - Resume guest • vagrant ssh - SSH into guest

10. Let’s talk about code

11. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Basic Vagrantfile Vagrant.configure("2") do |config| config.vm.box = "base-box-name" config.vm.box_url = ”http://some/url.box" end

12. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Basic Vagrantfile (add file system) Vagrant.configure("2") do |config| config.vm.box = "base-box-name" config.vm.box_url = ”http://some/url.box" config.vm.synced_folder "/data", "/vagrant_data" end

13. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Basic Vagrantfile (add port forwarding) Vagrant.configure("2") do |config| config.vm.box = "base-box-name" config.vm.box_url = ”http://some/url.box" config.vm.synced_folder "/data", "/vagrant_data" config.vm.network :forwarded_port, guest:80, host: 8080 end

14. Let’s talk about clouds

15. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Replicating production matters If you are writing code that will be deployed to a multiple-host environment, why aren’t you writing your code in a multiple- host environment?

16. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Basic Vagrantfile Vagrant.configure("2") do |config| config.vm.hostname = “box” config.vm.box = "base-box-name" config.vm.box_url = ”http://some/url.box" end

17. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Basic Vagrantfile (Multi-Guest ready) Vagrant.configure("2") do |config| config.vm.define “box” do |node| node.vm.hostname = “box” node.vm.box = "base-box-name" node.vm.box_url = ”http://some/url.box" end end

18. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Basic Vagrantfile (Multi-Guest) Vagrant.configure("2") do |config| config.vm.define “box” do |node| node.vm.hostname = “box” node.vm.box = "base-box-name" node.vm.box_url = ”http://some/url.box" end config.vm.define “other-box” do |node| node.vm.hostname = “other-box” node.vm.box = "base-box-name" node.vm.box_url = ”http://some/url.box" end end

19. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Basic Vagrantfile (Multi-Guest, useful, part 1) nodes = { :'web-server' => { :hostname => 'server.example.com', :ipaddress => '192.168.56.60', }, :'db-server' => { :hostname => 'db.example.com', :ipaddress => '192.168.56.61', } }

20. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Basic Vagrantfile (Multi-Guest, useful, part 2) nodes = { … node configuration … } Vagrant.configure("2") do |config| nodes.each_pair do |name,options| config.vm.define name do |node| node.vm.box = "base-box-name" node.vm.box_url = "http://some/url.box" node.vm.hostname = options[:hostname] node.vm.network :private_network, ip: options[:ipaddress] end end end

21. Let’s talk about Providers

22. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Vagrant Providers Providers Vagrant

23. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Vagrant Providers VirtualBox VMWare (WS or Fusion) Parallels Desktop Providers Vagrant

24. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Vagrant Providers VirtualBox VMWare (WS or Fusion) Parallels Desktop Amazon AWS Digital Ocean Providers Vagrant

25. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Vagrant Providers (desktop) • VirtualBox is FREE! (sort of) • Some people have trouble with crashes • Some people have trouble with suspend/resume • Built in FS bridging is pretty slow • VMWare (Workstation or Fusion!) is PAID! • You have to buy both VMWare ($49) and provider ($79) • Supposed to be more stable and faster • Parallels Desktop is FREE! • Open source projects FOR THE WIN!

26. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Vagrant Providers (cloud) • Amazon AWS • Digital Ocean • RackSpace Cloud

27. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Vagrant Provider Limits Spinning up guests is cool, but what do I do with a box once it’s up? How do I control it? How do I make it useful? Computing hardware without software is nearly useless.

28. Let’s talk about Provisioners

29. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Vagrant Provisioners Provisioners Vagrant

30. Shell Scripts Puppet Chef Ansible Virtualize your environment with Vagrant

    and Puppet - Columbus Code Camp - October 12th, 2013 Vagrant Provisioners Provisioners Vagrant

31. Let’s talk about Puppet

32. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 What is Puppet? Puppet is IT automation software that helps system administrators manage infrastructure throughout its lifecycle, from provisioning and configuration to patch management and compliance.

33. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 What?

34. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 What is Puppet in simpler words? Puppet gives you an object oriented way to manage the configuration of your servers, either on your laptop, in the cloud, or in your favorite data center.

35. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Puppet is built... • to be simple • with it’s own declarative language

36. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Puppet terms • Manifests - This is “application” code • Modules - This is “library” code • Templates - Exactly as it sounds • Facter - Environment data • Hiera - Configuration data

37. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 How to use Puppet? Select a base box for Vagrant from Puppet Labs http://puppet-vagrant-boxes.puppetlabs.com/ I tend to use CentOS 6.4 for VirtualBox

38. Let’s talk about code

39. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Basic Vagrantfile (add Puppet) Vagrant.configure("2") do |config| config.vm.box = "base-box-name" config.vm.box_url = "http://some/url.box" config.vm.provision :puppet do |puppet| puppet.manifests_path = "puppet/manifests/" puppet.manifest_file = "guest.pp" puppet.module_path = "puppet/modules/" end end

40. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Basic Puppet Manifest (guest.pp) class guest { notify { “Look! My puppet code is working!”: } }

41. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Basic Puppet Manifest (add Apache) class guest { notify { “Look! My puppet code is working!”: } package { “httpd”: ensure => present, } service { “httpd”: require => Package[“httpd”], ensure => running, enabled => true, } }

42. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Basic Puppet Manifest (refactored Apache) # manifests/guest.pp class guest { notify { “Code working!”: } require server::httpd } # modules/server/httpd.pp class server::httpd { package { “httpd”: ensure => present, } service { “httpd”: require => Package[“httpd”], ensure => running, enabled => true, } }

43. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Puppet Manifest (controlling flow) class guest { notify { “Say me first”: } notify { “Say me second”: require => Notify[“Say me first”], } notify { “Say me third”: require => Notify[“Say me second”], } }

44. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Puppet Manifest (adding parameters) # modules/server/writetestfile.pp class server::writetestfile ($content) { file { “/tmp/test”: content => $content, } } # modules/server/writetestfiledefault.pp class server::writetestfiledefault ($content = “default”) { file { “/tmp/test”: content => $content } }

45. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Puppet Manifest (passing parameters) # manifests/guest.pp -- using defaults class guest { include server::writetestfiledefault } # manifests/guest.pp -- passing custom value class guest { class { “server::writetestfiledefault”: content => “test content”, } }

46. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Puppet Manifest (including classes) class guest { # the class below must be loaded BEFORE guest require server::writetestfiledefault # the class below must be loaded WITH guest include server::writetestfiledefault # the class below must be loaded AFTER guest class { “server::writetestfiledefault”: content => “test content”, } }

47. Let’s talk about how to do things conditionally

48. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 External Input in Puppet Remember there are three primary sources of where data that branches your logic will come from: • User Space Variables • $things = “you set yourself” • Facts • Specific details about your current environment, such as the name of the host, ip address, operating system, etc... • Hiera • Configuration you pass in to enable building for multiple configurations from the same shared codebase

49. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Puppet Manifest (variables) class guest { # local variable $host = “mybox” if ($host == “mybox”) { notify { “Box!”: } } # Box! if ($host == “mybox”) { notify { “${host}!”: } } # mybox! }

50. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Puppet Manifest (facts) class guest { # fact - environment configuration if ($::hostname == “box”) { notify { “Box!”: } } # output: Box! if ($hostname == “box”) { notify { “Box!”: } } # output: Box! # assign $hostname locally... $hostname = “some string” if ($::hostname == “box”) { notify { “Box!”: } } # output: Box! if ($hostname == “box”) { notify { “Box!”: } } # no output }

51. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Puppet Manifest (facts, debugging) class guest { # what facts are there? Let’s find out! # this should be one line, but it is just too long to fit $template = "<%= scope.to_hash.reject { |k,v| !( k.is_a? (String) && v.is_a?(String) ) }.to_yaml %>" # create a file... file { "/tmp/facts.yaml": content => inline_template($template), } }

52. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Puppet Manifest (if conditionals) class guest { if (“a” == “b”) { notify { “Inconceivable!”: } } elsif (“hello” =~ “help”) { notify { “Regex!”: } } else { notify { “Echo something...”: } } }

53. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Puppet Manifest (case conditionals) class guest { # $::operatingsystem is a fact, provided directly by Facter case $::operatingsystem { centos, redhat: { $apache = "httpd" } debian, ubuntu: { $apache = "apache2" } default: { fail("Unrecognized operating system") } } case $::operatingsystem { /linux/: { fail(“We don’t support anything with Linux in it”) } } }

54. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Puppet Manifest (selector conditionals) class guest { # $::operatingsystem is a fact, provided directly by Facter $package = $::operatingsystem ? { centos => “httpd”, redhat => “httpd”, /(i?)(ubuntu|debian)/ => “apache2”, default => undef } }

55. Let’s talk about real multiple guest configuration

56. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Configure our guest nodes... nodes = { :'web-server' => { :hostname => 'server.example.com', :ipaddress => '192.168.56.60', }, :'db-server' => { :hostname => 'db.example.com', :ipaddress => '192.168.56.61', } }

57. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Tell Vagrant about them nodes = { … node configuration … } Vagrant.configure("2") do |config| nodes.each_pair do |name,options| config.vm.define name do |node| node.vm.box = "base-box-name" node.vm.box_url = "http://some/url.box" node.vm.hostname = options[:hostname] node.vm.network :private_network, ip: options[:ipaddress] end end end

58. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Tell Vagrant about Puppet nodes = { … node configuration … } Vagrant.configure("2") do |config| nodes.each_pair do |name,options| config.vm.define name do |node| node.vm... # omit general config node.vm.provision :puppet do |puppet| puppet.manifests_path = "puppet/manifests/" puppet.manifest_file = "guests.pp" puppet.module_path = "puppet/modules/" end end end end

59. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Tell Puppet about the guest nodes class guests { if ($::hostname == “web-server”) { # only run for web server } if ($::hostname == “db-server”) { # only run for db server } }

60. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Tell Puppet about the guest nodes (redux) class guests { if ($::hostname =~ /^web-/) { # only run for web servers } if ($::hostname =~ /^db-/) { # only run for db servers } }

61. Let’s talk about how everything ties together

62. Shell Scripts Puppet Chef Ansible VirtualBox VMWare (WS or Fusion)

    Parallels Desktop Amazon AWS Digital Ocean Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Vagrant Architecture Providers Provisioners Vagrant

63. Let’s talk about how to convince your boss

64. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Development environments come in all shapes and sizes • Some develop in production – Yes, really. • Some develop on a non production server – Not a copy of production, not by a long shot • Some develop locally on a “non production server” – Home grown solutions that “work” until they don’t • Some develop on a development server – Actually tries to be like production – Not always like production – May have additional tuning to help with developmenty things • Some develop in a true development environment – Locally or remote, coming soon to a computer near you

65. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Why developing in production is really bad Assuming, of course, that you are testing your code before you make it live. Production should always move from one stable version of code to another stable version of code. When you edit code in production, you inherently introduce instability, and can no longer be assured that your code is running as tested.

66. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Why developing on a non production setup is also bad Debugging environmental issues is slow and painful. Developers should be solving problems with software, not debugging configuration issues. When you build your code in an environment that isn’t related to production in any tangible way, you will often discover just how different your development environment is from production.

67. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Where your development environment lives Remote • Easy for operations to set up • Easy to enforce a rigid structure • Limits developer’s choice of tool chain selection • Can get expensive and complicated to maintain • Developers must be online to do any development Local • Can require beefier hardware • Can take extra work to set up • Developers have the freedom to work how they work best • Developers can assist with infrastructural initiatives

68. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Bad development environments will sap the resources out of your team. In addition to the time spent developing and testing new code and fixing bugs, you will spend time: When development environments attack Debugging broken environments or settings Debugging broken behavior between dev and prod Debugging broken behavior between two dev systems Working in broken ways because “it works”

69. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Failure to specifically build an environment for development will directly impact developer productivity and product quality.

70. Let’s talk about what you should remember

71. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Takeaways • Bad (or unstable) environments (be it dev, qa, staging, or prod) hurt developers. • Puppet lets you automate all of the configuration of your systems (dev, qa, staging, and prod), ensuring consistency between them. • Vagrant is easy to use, and will let you use your production puppet configs to build development environments. • You will manage server configuration, whether you decide to or not. If you choose not to do it formally it will come back to bite you when you forget something later.

72. Let’s talk about where to go from here

73. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 If you’re still not sold... h"p://PuPHPet.com/ / PuPHPet/provides/a/ simple/config/tool/for/ those/who/don’t/want/ to/=nker/with/puppet/ and/need/a/LAMP/stack/ up/and/running/NOW./ V2 was released this morning!!!

74. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 And also look at... • Packer • http://packer.io • Take your Vagrant and Puppet usage to the next level by using Packer to pre-provision images for easy first spins • Build prebuilt systems for multiple VMS and cloud platforms • Docker • http://docker.io • A different idea of how to build multiple machine VM environments

75. Virtualize your environment with Vagrant and Puppet - Columbus Code

    Camp - October 12th, 2013 Thank you! More information about this talk can be found at: http://jmather.com/talks/2013-10-12