Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Advanced Vagrant Usage with Puppet

Advanced Vagrant Usage with Puppet

This talk was given at PuppetConf 2012. It covers how I use Vagrant with Puppet and how I automate a lot of what I do with Puppet.

Mitchell Hashimoto

September 27, 2012
Tweet

More Decks by Mitchell Hashimoto

Other Decks in Programming

Transcript

  1. Advanced Vagrant
    Usage with Puppet

    View full-size slide

  2. I’m Mitchell Hashimoto
    Also known as @mitchellh

    View full-size slide

  3. I made Vagrant.
    Hopefully you use it. I think you’ll like it.
    http://vagrantup.com

    View full-size slide

  4. I’m an automation freak.
    This talk will show this to be true.

    View full-size slide

  5. Vagrant Usage
    (Ops Focused)

    View full-size slide

  6. Benefits we want
    from Vagrant...

    View full-size slide

  7. “The Cloud”
    but on your machine.

    View full-size slide

  8. Self service.
    Instant provisioning.
    Cost efficient.
    Elastic.
    Pay per use.
    Paul Strong’s Cloud

    View full-size slide

  9. - Manifest development,
    both simple and not so simple
    Benefits We Want

    View full-size slide

  10. - Manifest development,
    both simple and not so simple
    - Repeatability
    Benefits We Want

    View full-size slide

  11. - Manifest development,
    both simple and not so simple
    - Repeatability
    - Fast feedback
    Benefits We Want

    View full-size slide

  12. - Manifest development,
    both simple and not so simple
    - Repeatability
    - Fast feedback
    - Confidence
    Benefits We Want

    View full-size slide

  13. Confession: I’ve been
    doing Puppet full time for
    awhile now.

    View full-size slide

  14. Current state of
    Vagrant + Puppet...

    View full-size slide

  15. Basic manifest
    development and testing.

    View full-size slide

  16. We can do better.
    We can do much better.

    View full-size slide

  17. We can do better with
    what is available right now.

    View full-size slide

  18. Teaser: We will do magic with
    what is coming in the future.
    <3 <3 <3

    View full-size slide

  19. My state of
    Vagrant + Puppet...

    View full-size slide

  20. Fully automated
    Puppet Master setup.

    View full-size slide

  21. Testing exported
    resources, hiera, and nodes.

    View full-size slide

  22. Common deploy process
    across Vagrant and EC2.

    View full-size slide

  23. Repeatable workflow of
    dev to staging to prod.

    View full-size slide

  24. Golden master box
    creation for development.

    View full-size slide

  25. Time to share
    what I’ve learned.

    View full-size slide

  26. Advanced
    Vagrant Usage

    View full-size slide

  27. Advanced
    Vagrant Usage

    View full-size slide

  28. Advanced
    Automation for Puppet Work

    View full-size slide

  29. Fully Automated
    Puppet Master Setup

    View full-size slide

  30. I asked: “How do people
    bring up or recover a
    Puppet master?”

    View full-size slide

  31. “Most people roll their
    Puppet Master by hand.”
    - Anonymous PuppetLabs Employee

    View full-size slide

  32. Puppet Master is crucial to
    testing realistic scenarios.

    View full-size slide

  33. Automated Puppet Master
    - Pushing broken Puppet
    crashes the server.

    View full-size slide

  34. Automated Puppet Master
    - Pushing broken Puppet
    crashes the server.
    - Local development against a
    Puppet Master has benefits.

    View full-size slide

  35. Automated Puppet Master
    - Pushing broken Puppet
    crashes the server.
    - Local development against a
    Puppet Master has benefits.
    - Automation all the way down.

    View full-size slide

  36. Multi-level bootstrap.

    View full-size slide

  37. 1. Bash script to minimally
    install Puppet master and
    agent.

    View full-size slide

  38. 2. puppet apply to
    minimally setup Puppet
    Master infrastructure

    View full-size slide

  39. 3. puppet agent to
    completely setup and
    harden the master.

    View full-size slide

  40. Result: Production-quality
    Puppet Master whenever
    you need it.

    View full-size slide

  41. Vagrant::Config.run do |config|
    # ...
    config.vm.hostname = "puppet"
    config.vm.provision :shell, :path =>
    "bootstrap.sh"
    end

    View full-size slide

  42. Testing
    Exported Resources,
    Hiera, and nodes.

    View full-size slide

  43. I asked: “How do people
    test more than the most
    basic Puppet module?”

    View full-size slide

  44. “I suspect the answer is
    that they just don't test
    their modules adequately.”
    - Anonymous PuppetLabs Employee

    View full-size slide

  45. Solution: Automated
    Puppet Master + Multi-VM

    View full-size slide

  46. Automated Puppet Master
    is production ready:
    PuppetDB, Hiera, etc.

    View full-size slide

  47. Multi-VM enables Vagrant
    to manage a cluster of
    machines that can
    communicate.
    http://vagrantup.com/v1/docs/multivm.html

    View full-size slide

  48. Testing Exported Resources

    View full-size slide

  49. Create two nodes.
    Export one.
    Collect other.
    Ruby/Shell script.
    Testing Exported Resources

    View full-size slide

  50. node 'test_exporter' {
    @@nginx::site { "test":
    content => "\n",
    tag => "origin",
    }
    }
    node 'test_collector' {
    include role::origin
    }

    View full-size slide

  51. Vagrant::Config.run do |config|
    config.vm.define :export do |n|
    n.vm.hostname = "test_exporter"
    n.vm.provision :puppet_server,
    :options => "--verbose --debug"
    end
    config.vm.define :collect do |n|
    n.vm.hostname = "test_collecter"
    n.vm.provision :puppet_server,
    :options => "--verbose --debug"
    end
    end

    View full-size slide

  52. #!/bin/bash
    test -f /etc/nginx/sites-available/test

    View full-size slide

  53. Testing Hiera

    View full-size slide

  54. Create full Hiera hierarchy.
    Launch node.
    Test hierarchy.

    View full-size slide

  55. ---
    test_region: "us-east-1"
    ---
    test_role: "hiera"
    ---
    test_name: "test-hiera”

    View full-size slide

  56. Vagrant::Config.run do |config|
    config.vm.hostname = "test_hiera"
    config.vm.provision :puppet_server,
    :options => "--verbose --debug"
    end
    end

    View full-size slide

  57. node 'test_hiera' {
    $region = hiera("test_region")
    $role = hiera("test_role")
    $name = hiera("test_name")
    file { "/tmp/results":
    content => "$region $role $name",
    mode => "0644",
    }
    }

    View full-size slide

  58. Testing Nodes

    View full-size slide

  59. Create node.
    Provision.
    Test behavior.

    View full-size slide

  60. Vagrant::Config.run do |config|
    config.vm.define :master do |master|
    master.vm.hostname = "puppet"
    master.vm.provision :shell, :path =>
    "bootstrap.sh"
    end
    config.vm.define :node do |node|
    node.vm.hostname = "postgresql"
    node.vm.provision :puppet_server,
    :options => "--verbose --debug"
    end
    end

    View full-size slide

  61. Pain points: Node
    destroy/up requires cert
    clean on master plus a
    PuppetDB deactivate.

    View full-size slide

  62. Common Deploy
    Process Across
    Vagrant and EC2

    View full-size slide

  63. Getting your Puppet code
    to your masters. Solved?

    View full-size slide

  64. Goal: Make it the same for
    Vagrant, production, and
    anything in between.

    View full-size slide

  65. My solution: Bash script
    to git pull, rsync, and
    restart the master.

    View full-size slide

  66. fab deploy:vagrant
    fab deploy:production
    fab deploy:dev-mitchellh
    http://fabfile.org

    View full-size slide

  67. @task
    def deploy(environment):
    # ...
    run("sudo /opt/puppet-updater/update")

    View full-size slide

  68. - Git pull
    - Find env-* branches for
    environments.
    - RSync
    - Restart Puppet Master
    http://bit.ly/Qyg3RW
    Updater Script

    View full-size slide

  69. Note: I don’t use Puppet
    environments for dev
    because I like to keep
    production master just for
    production.

    View full-size slide

  70. (Plus, the automated
    Puppet Master setup is
    just so easy!)

    View full-size slide

  71. Repeatable Workflow
    From Dev to Staging
    to Production

    View full-size slide

  72. Dev is in VirtualBox.
    Staging is in EC2.
    Production is in EC2.

    View full-size slide

  73. Goal: Same workflow.

    View full-size slide

  74. rake launch:vagrant,postgresql
    rake destroy:production,riak-001
    rake provision:staging,haproxy-002

    View full-size slide

  75. Wrapper around vagrant
    and AWS library.

    View full-size slide

  76. Hides some cruft: destroy
    will cert clean and
    deactivate from
    PuppetDB, for example.

    View full-size slide

  77. Golden Master Box
    Creation for
    Development

    View full-size slide

  78. vagrant up a complete
    dev environment can be
    slow.

    View full-size slide

  79. Take advantage of
    Puppet’s idempotence
    and vagrant package

    View full-size slide

  80. Two-pass Puppet run
    for development.

    View full-size slide

  81. Pass 1 (pre-package):
    Installation and
    configuration.

    View full-size slide

  82. Pass 2 (vagrant up):
    Service starting and
    maybe configuration.

    View full-size slide

  83. vagrant package takes
    current Vagrant VM and
    produces a distributable
    box.

    View full-size slide

  84. Build discipline around
    updating the base box.

    View full-size slide

  85. Example: Work on any
    box you want, update to
    latest base box prior to
    committing.

    View full-size slide

  86. Bonus points:
    Put this in a CI.

    View full-size slide

  87. Automate
    All the things

    View full-size slide

  88. THANKS!
    @mitchellh

    View full-size slide