Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Joy of Cooking: Deploying Drupal with Chef

The Joy of Cooking: Deploying Drupal with Chef

An introduction to Chef and information on deploying Drupal with Chef. This presentation was originally given at DrupalCon 2013.

Nathen Harvey

May 23, 2013
Tweet

More Decks by Nathen Harvey

Other Decks in Technology

Transcript

  1. Nathen
    •Technical Community Manager at Opscode
    •Co-host of the Food Fight Show Podcast
    •Meetup Organizer

    View full-size slide

  2. Chef is an automation platform for developers & systems engineers to continuously
    define, build, and manage infrastructure.
    CHEF USES:
    Recipes and Cookbooks
    that describe Infrastructure as Code.
    Chef enables people to easily build &
    manage complex & dynamic applications
    at massive scale
    • New model for describing infrastructure that promotes
    reuse
    • Programmatically provision and configure
    • Reconstruct business from code repository,
    data backup, and bare metal resources


    Chef

    View full-size slide

  3. http://www.flickr.com/photos/steffenz/337700069/
    http://www.flickr.com/photos/kky/704056791/
    Applications

    View full-size slide

  4. Infrastructure
    http://www.flickr.com/photos/sbh/462754460/

    View full-size slide

  5. Collection of Resources
    http://www.flickr.com/photos/philliecasablanca/3354734116/
    • Networking
    • Files
    • Directories
    • Symlinks
    • Mounts
    • Routes
    • Users
    • Groups
    • Tasks
    • Packages
    • Software
    • Services
    • Configuration
    • Other Stuff

    View full-size slide

  6. Acting in Concert
    http://www.flickr.com/photos/glowjangles/4081048126/

    View full-size slide

  7. To Provide a Service
    http://www.flickr.com/photos/28309157@N08/3743455858/

    View full-size slide

  8. And it Evolves
    http://www.flickr.com/photos/16339684@N00/2681435235/

    View full-size slide

  9. Application Server
    See Node

    View full-size slide

  10. Application Server
    Application Database
    See Nodes

    View full-size slide

  11. Application Server
    Application Databases
    See Nodes Grow

    View full-size slide

  12. Application Servers
    Application Databases
    See Nodes Grow

    View full-size slide

  13. Application Servers
    Application Databases
    Load Balancer
    See Nodes Grow

    View full-size slide

  14. Application Servers
    Application Databases
    Load Balancers
    See Nodes Grow

    View full-size slide

  15. Application Servers
    Application Database Cache
    Load Balancers
    Application Databases
    See Nodes Grow

    View full-size slide

  16. Application Servers
    Application Database Cache
    Load Balancers
    Application Databases
    Tied Together with Configuration

    View full-size slide

  17. Application Servers
    Application Database Cache
    Load Balancers
    Floating IP?
    Application Databases
    Infrastructure is a Snowflake

    View full-size slide

  18. Load Balancers
    Application Servers
    NoSQL
    Database Slaves
    ApplicationCache
    Database Cache
    Database
    Evolving Complexity

    View full-size slide

  19. Configuration Management
    http://www.flickr.com/photos/philliecasablanca/3354734116/

    View full-size slide

  20. New Requirements
    •A new developer joined the team, get her set-up with the
    application
    •Add New Relic monitoring to the application
    •Add a new module to the development site
    •Do not store the unencrypted admin password in the git
    repository

    View full-size slide

  21. Configuration Desperation
    http://www.flickr.com/photos/francoforeshock/5716969942/

    View full-size slide

  22. • But you already
    guessed that, didn’t
    you?
    Chef Solves This Problem

    View full-size slide

  23. http://www.flickr.com/photos/louisb/4555295187/
    • Programmatically provision
    and configure
    • Treat like any other code base
    • Reconstruct business from
    code repository, data backup,
    and bare metal resources.
    Chef is Infrastructure as Code

    View full-size slide

  24. http://www.flickr.com/photos/ssoosay/5126146763/
    • Chef generates configurations
    directly on nodes from their
    run list
    • Reduce management
    complexity through
    abstraction
    • Store the configuration of your
    programs in version control
    Programs

    View full-size slide

  25. •Define Policy
    •Say what, not how
    •Pull not Push
    http://www.flickr.com/photos/bixentro/2591838509/
    Declarative Interface to Resources

    View full-size slide

  26. That Looks Like This
    package "apache2"
    template "/etc/apache2/apache2.conf" do
    soucre "apache2.conf.erb"
    owner "root"
    group "root"
    mode "0644"
    variables(:allow_override => "All")
    notifies :reload, "service[apache2]"
    end
    service "apache2" do
    action [:enable,:start]
    supports :reload => true
    end

    View full-size slide

  27. • Recipes are collections of
    Resources
    • Cookbooks contain recipes,
    templates, files, custom
    resources, etc
    • Code re-use and modularity
    • Hundreds already on
    Community.opscode.com
    http://www.flickr.com/photos/shutterhacks/4474421855/
    Recipes and Cookbooks

    View full-size slide

  28. http://www.flickr.com/photos/kathycsus/2686772625
    • IP addresses
    • Hostnames
    • FQDNs
    • Search for nodes with
    Roles
    • Find configuration
    data
    Search

    View full-size slide

  29. pool_members = search("node","role:awesome_site)
    template "/etc/haproxy/haproxy.cfg" do
    source "haproxy-app_lb.cfg.erb"
    owner "root"
    group "root"
    mode "0644"
    variables :pool_members => pool_members.uniq
    notifies :restart, "service[haproxy]"
    end
    Pass Results To Templates

    View full-size slide

  30. # Set up application listeners here.
    listen application 0.0.0.0:80
    balance roundrobin
    <% @pool_members.each do |member| -%>
    server <%= member[:hostname] %> <%= member[:ipaddress] %>: weight 1 maxconn 1 check
    <% end -%>
    <% if node["haproxy"]["enable_admin"] -%>
    listen admin 0.0.0.0:22002
    mode http
    stats uri /
    <% end -%>
    Pass Results To Templates

    View full-size slide

  31. Drupal App
    Memcache
    MySQL Slaves
    MySQL Master
    Nagios
    Graphite
    So when this...

    View full-size slide

  32. Drupal App
    Memcache
    MySQL Slaves
    MySQL Master
    Nagios
    Graphite
    ...becomes this...

    View full-size slide

  33. Drupal App
    Memcache
    MySQL Slaves
    MySQL Master
    Nagios
    Graphite
    ...this can happen automatically

    View full-size slide

  34. Nagios
    Graphite
    Drupal App
    Memcache
    MySQL Slaves
    • Load balancer config
    • Nagios host ping
    • Nagios host ssh
    • Nagios host HTTP
    • Nagios host app health
    • Graphite CPU
    • Graphite Memory
    • Graphite Disk
    • Graphite SNMP
    • Memcache firewall
    • MySQL firewall
    • 11+ resource changes for 1 node addition
    Count the Resources

    View full-size slide

  35. Deploying Drupal with
    Chef

    View full-size slide

  36. Chef Components

    View full-size slide

  37. Our Workflow
    •Build Drupal locally
    •Deploy Drupal on EC2
    •Iterate

    View full-size slide

  38. Local Development
    •Git repository
    •Virtual Machine
    •Chef
    •knife
    •Chef Server

    View full-size slide

  39. Git Repository
    $ mkdir -p ~/drupalcon/chef-repo
    $ cd ~/drupalcon/chef-repo
    $ git init

    View full-size slide

  40. Building Drupal Locally
    •Deploy in a local virtual
    machine managed by
    Vagrant, provisioned with
    Chef

    View full-size slide

  41. Vagrantfile
    Vagrant.configure("2") do |config|
    config.vm.hostname = "drupalcon-dev"
    config.vm.box = "opscode-ubuntu-12.04"
    config.vm.network :private_network, ip: "33.33.33.10"
    config.vm.network :forwarded_port, guest: 80, host: 8080
    end

    View full-size slide

  42. Register VM with Chef
    Vagrant.configure("2") do |config|
    ...
    config.vm.provision :chef_client do |chef|
    chef.chef_server_url = "https://api.opscode.com/organizations/nhdrupalcon"
    chef.validation_key_path = ".chef/nhdrupalcon-validator.pem"
    chef.validation_client_name = "nhdrupalcon-validator"
    end
    end
    $ vagrant provision

    View full-size slide

  43. Chef Components

    View full-size slide

  44. Cookbooks
    •Download from community.opscode.com
    •Write our own
    •Upload to the Chef Server

    View full-size slide

  45. Community Site
    •950+ Cookbooks
    •Codify tribal knowledge
    •Reference

    View full-size slide

  46. Write our own Cookbook
    $ knife cookbook create awesome_site
    ** Creating cookbook awesome_site
    ** Creating README for cookbook: awesome_site
    ** Creating CHANGELOG for cookbook: awesome_site
    ** Creating metadata for cookbook: awesome_site

    View full-size slide

  47. Write our own Cookbook
    include_recipe "drupal"
    web_app "drupal" do
    template "drupal.conf.erb"
    docroot node['drupal']['dir']
    server_name server_fqdn
    server_aliases node['fqdn']
    end

    View full-size slide

  48. Data
    •Separate data from policy
    •Policy: Site has a document root
    •Data: /var/www/drupal
    •Policy: Apache has a Server Alias
    •Data: Fully-qualified domain name of the server
    web_app "drupal" do
    template "drupal.conf.erb"
    docroot node['drupal']['dir']
    server_name server_fqdn
    server_aliases node['fqdn']
    end

    View full-size slide

  49. Where does data come from?
    •Attributes
    •Data Bags
    •Encrypted Data Bags
    •Search

    View full-size slide

  50. Attributes
    •Specific details about a node
    •Defined by:
    •The state of the node
    •Cookbooks
    •Roles
    •Environments

    View full-size slide

  51. Ohai
    "languages": {
    "ruby": {
    },
    "perl": {
    "version": "5.14.2",
    "archname": "x86_64-linux-gnu-thread-multi"
    },
    "python": {
    "version": "2.7.3",
    "builddate": "Aug 1 2012, 05:14:39"
    },
    "php": {
    "version": "5.3.10-1ubuntu3.6",
    "builddate": "(cli) (built: Mar"
    }
    },
    "kernel": {
    "name": "Linux",
    "release": "3.2.0-32-virtual",
    "version": "#51-Ubuntu SMP Wed Sep 26 21:53:42 UTC 2012",
    "machine": "x86_64",
    "modules": {
    "isofs": {
    "size": "40257",
    "refcount": "0"
    },
    "acpiphp": {
    "size": "24231",
    "refcount": "0"
    }
    },
    "os": "GNU/Linux"
    },
    "os": "linux",
    "os_version": "3.2.0-32-virtual",
    "ohai_time": 1369328621.3456137,
    "network": {
    "interfaces": {
    "lo": {
    "mtu": "16436",
    "flags": [
    "LOOPBACK",
    "UP",
    "LOWER_UP"
    ],
    "encapsulation": "Loopback",
    "addresses": {
    "127.0.0.1": {
    "family": "inet",
    "prefixlen": "8",
    "netmask": "255.0.0.0",
    "scope": "Node"
    },
    "::1": {
    "family": "inet6",
    "prefixlen": "128",
    "scope": "Node"
    }
    },
    "state": "unknown"
    },
    "eth0": {
    "type": "eth",
    "number": "0",
    "mtu": "1500",

    View full-size slide

  52. Attributes
    •Cookbooks, Roles, and Environments can also set
    attribute data
    •Extremely flexible mechanism for configuration
    •Cookbook:
    •Environment:
    default['drupal']['modules'] = ['views', 'webform']
    default_attributes({ "drupal" => {"modules" => ["views","webform","token"]}})

    View full-size slide

  53. Data Bags
    •Global Data
    •Stored in JSON
    •Accessible from the Chef Server
    •Can be encrypted
    {
    "id": "nharvey",
    "groups": ["sysadmin"],
    "uid": 2001,
    "shell": "/bin/bash",
    "comment": "Nathen Harvey ",
    "nagios": {
    "email": "[email protected]"
    }
    }

    View full-size slide

  54. Search
    pool_members = search("node","awesome_site)
    template "/etc/haproxy/haproxy.cfg" do
    source "haproxy-app_lb.cfg.erb"
    owner "root"
    group "root"
    mode "0644"
    variables :pool_members => pool_members.uniq
    notifies :restart, "service[haproxy]"
    end

    View full-size slide

  55. Upload our Cookbooks
    •We’re using
    •Cookbooks from the community site
    •A cookbook we wrote
    •Upload the cookbooks using knife
    $ knife cookbook upload -a

    View full-size slide

  56. Deploy Drupal

    View full-size slide

  57. Run List
    •Ordered list of roles or recipes to be run on a node
    •The run list represents the policy to be applied
    •chef-client
    •Executes on the node on a regular basis
    •Receives the run list from the Chef Server
    •Brings the node in-line with the policy

    View full-size slide

  58. Run List
    Server
    Server
    Server
    Server
    chef-server
    API
    chef-client
    “role[webserver]”
    node
    ntp
    client.rb
    openssh
    server.rb
    apache
    default.rb
    drupal
    default.rb
    chef-client
    “role[database]”
    node
    ntp
    client.rb
    openssh
    server.rb
    mysql
    server.rb

    View full-size slide

  59. Run chef-client
    config.vm.provision :chef_client do |chef|
    chef.chef_server_url = "https://api.opscode.com/organizations/nhdrupalcon"
    chef.validation_key_path = ".chef/nhdrupalcon-validator.pem"
    chef.validation_client_name = "nhdrupalcon-validator"
    chef.add_recipe "awesome_site"
    end
    $ vagrant provision

    View full-size slide

  60. Next Steps
    •Share the Vagrantfile with your teammates
    •Develop to a releasable state
    •Deploy to production

    View full-size slide

  61. Production Deploy
    •We’ll use Amazon’s EC2 for our initial deploy
    •...but we could deploy to any server
    •Using the same cookbooks as our development
    environment, deploy to EC2
    $ knife ec2 server create -r "recipe[awesome_site]" -f m1.medium -I ami-641c8e0d -N ec2_drupal

    View full-size slide

  62. Current State
    •Local virtual machine for development running Drupal
    •“Production” instance on EC2

    View full-size slide

  63. Next Steps
    •Move Database to a different node
    •Add a Load Balancer and additional application servers
    •Add monitoring
    •Add log aggregation
    •etc.

    View full-size slide

  64. More about Chef

    View full-size slide

  65. Open Source
    •Apache 2 Software License
    •Continually growing number of contributors!
    •1400+ individuals, 200+ companies
    •Development repositories:
    •github.com/opscode
    •github.com/opscode-cookbooks
    •Regular code reviews via Google+ Hangout

    View full-size slide

  66. Get Started on Your Own
    •http://learnchef.com
    •http://docs.opscode.com
    •http://lists.opscode.com
    •#chef on Freenode IRC

    View full-size slide

  67. Wrap-up & Special Offer

    View full-size slide

  68. Infrastructure as Code
    •Repeatability
    •Automation
    •Agility
    •Scalability
    •Reassurance
    •Disaster Recovery
    •Happiness

    View full-size slide

  69. Thanks!
    •Promet Source
    •Will Milton
    •github.com/promet/drupal_cookbook
    •curl -L bit.ly/10McZSZ | bash; vagrant up
    •Marius Ducea
    •community.opscode.com/cookbooks/drupal

    View full-size slide

  70. Chef Training
    •Chef Introductory Workshop
    •Tomorrow in Corvallis
    •FREE to all DrupalCon Attendees
    •http://bit.ly/learnchef-drupalcon

    View full-size slide

  71. Thank You!
    •What Questions Do You Have?
    •@nathenharvey
    •Free Training Tomorrow (register by 9PM)
    •bit.ly/learnchef-drupalcon
    •Feedback on this session: portland2013.drupal.org/node/2963

    View full-size slide