$30 off During Our Annual Pro Sale. View Details »

Chef: Automation for the Cloud

Nathen Harvey
September 05, 2013

Chef: Automation for the Cloud

Introduction to Chef for automating your cloud. This was originally given at a Rackspace inteRACTIVE event.

Nathen Harvey

September 05, 2013
Tweet

More Decks by Nathen Harvey

Other Decks in Technology

Transcript

  1. Chef: Automation for the Cloud Nathen Harvey nharvey@opscode.com @nathenharvey

  2. Nathen Harvey • Technical Community Manager at Opscode • Co-host

    of the Food Fight Show Podcast • Meetup Organizer • @nathenharvey
  3. We have a problem...

  4. http://www.flickr.com/photos/michaelheiss/3090102907/ Complexity

  5. Items of Manipulation (Resources) • Nodes • Networking • Files

    • Directories • Symlinks • Mounts • Routes • Users • Groups • Packages • Services • Filesystems
  6. Application A tale of growth...

  7. Application Application Database Add a database

  8. Application App Databases Make database redundant

  9. App Servers App Databases Application server redundancy

  10. App LB App Servers App Databases Add a load balancer

  11. App LBs App Servers App Databases Webscale!

  12. App LBs App Servers App DB Cache App DBs Now

    we need a caching layer
  13. App LBs App Servers App DB Cache App DBs Infrastructure

    has a Topology
  14. Round Robin DNS App Servers App DB Cache App DBs

    Floating IP? Your Infrastructure is a Snowflake
  15. App LBs App Servers < Shiny! DB slaves Cache DB

    Cache DBs Complexity Increases Quickly Are we monitoring??
  16. The Chef Framework • Reasonability • Flexibility • Library &

    Primitives • TIMTOWTDI • Sane defaults http://www.flickr.com/photos/wonderlane/3609342683/sizes/l/in/photostream/
  17. The Chef Tool(s) • ohai • chef-client • chef-shell •

    knife • The Ruby language Omnibus - Full Stack Native Packages
  18. The Chef API • HTTPS, RESTful API w/ JSON, RSA

    key auth • Infrastructure data store such as node data • Search Service • Derivative Services? http://www.flickr.com/photos/core-materials/4419853626/sizes/o/in/photostream/
  19. The Chef Community • Apache License, Version 2.0 • Hundreds

    of Individual and Corporate contributors. • Hundreds of cookbooks available from the community • http://community.opscode.com
  20. How does it work? http://i3.kym-cdn.com/photos/images/original/000/046/123/magnets.jpg

  21. Chef is Infrastructure as Code • Programmatically provision and configure

    • Treat like any other code base • Reconstruct business from code repository, data backup, and bare metal resources. http://www.flickr.com/photos/louisb/4555295187/
  22. Programs • Chef generates configurations directly on nodes from their

    run list • Reduce management complexity through abstraction • Store the configuration of your programs in version control http://www.flickr.com/photos/ssoosay/5126146763/
  23. Declarative Interface to Resources • Define Policy • Say what,

    not how • Pull not Push http://www.flickr.com/photos/bixentro/2591838509/
  24. That Looks Like This package "apache2" template "/etc/apache2/apache2.conf" do source

    "apache2.conf.erb" owner "root" group "root" mode "0644" variables(:allow_override => "All") notifies :reload, "service[apache2]" end service "apache2" do action [:enable,:start] supports :reload => true end
  25. http://www.flickr.com/photos/peterrosbjerg/3913766224/ Chef Nodes • Chef runs on nodes • Chef

    nodes do the heavy lifting • Authority about themselves • Stored on the server when using Chef Server • Indexed for search
  26. Search • Search for nodes with Roles • Find Topology

    Data • IP addresses • Hostnames • FQDNs http://www.flickr.com/photos/kathycsus/2686772625
  27. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite So

    when this...
  28. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite ...becomes

    this
  29. Memcache Postgres Slaves Postgres Master Nagios Graphite Jboss App Memcache

    Postgres Slaves Postgres Master Nagios Graphite ...this can happen automatically
  30. Nagios Graphite Nagios Graphite Memcache Postgres Slaves • Load balancer

    config • Nagios host ping • Nagios host ssh • Nagios host HTTP • Nagios host app health • Graphite CPU • Graphite Memory • Graphite Disk • Graphite SNMP • Memcache firewall • Postgres firewall • Postgres authZ config • 12+ resource changes for 1 node addition Count the Resources Jboss App
  31. Landscape of Chef-managed Infrastructure

  32. knife

  33. knife - with the Chef Server • knife node •

    create / edit / delete • list • knife cookbook ... • knife role ... • knife environment ...
  34. • SSH to the machine given existing credentials • Install

    the Chef Client • Register with the Chef Server • Run the initial Run List • Now managed with Chef! $ knife bootstrap SERVER -r ‘role[webserver]’ knife bootstrap
  35. $ knife rackspace knife rackspace ** RACKSPACE COMMANDS ** knife

    rackspace flavor list (options) knife rackspace image list (options) knife rackspace network create (options) knife rackspace network delete NETWORK_ID [NETWORK_ID] (options) knife rackspace network list (options) knife rackspace server create (options) knife rackspace server delete SERVER_ID [SERVER_ID] (options) knife rackspace server list (options)
  36. $ knife rackspace flavor list Rackspace Flavors ID Name VCPUs

    RAM Disk 2 512MB Standard Instance 1 512 20 GB 3 1GB Standard Instance 1 1024 40 GB 4 2GB Standard Instance 2 2048 80 GB 5 4GB Standard Instance 2 4096 160 GB 6 8GB Standard Instance 4 8192 320 GB 7 15GB Standard Instance 6 15360 620 GB 8 30GB Standard Instance 8 30720 1200 GB
  37. $ knife rackspace image list Rackspace Images ID Name 01b1691b-c7b9-4bc0-a655-530f51dbfdb4

    Arch 2013.8 25a5f2e8-f522-4fe0-b0e0-dbaa62405c25 CentOS 6.4 b7511a0b-dd00-4246-9095-4ade53a72af4 Fedora 19 (Schrodinger's Cat) 9620c99d-9eeb-4142-b88f-7ec004901ffb OpenSUSE 12.3 e0c3a14d-b1dc-451b-ade4-d419501aa121 Red Hat Enterprise Linux 6.4 c6f9c411-e708-4952-91e5-62ded5ea4d3e Ubuntu 12.10 (Quantal Quetzal) 1bbc5e56-ca2c-40a5-94b8-aa44822c3947 Ubuntu 13.04 (Raring Ringtail) 8e5b96b6-25f7-4c10-ad31-c10afa02ed43 Windows Server 2008 R2 SP1 ...
  38. Web Server Create a simple Web Server

  39. $ knife rackspace server create -N web01 -f 2 -I

    1bbc5e56-ca2c-40a5-94b8-aa44822c3947 -r "role[my_app]" Create a Server Instance ID: 21329a45-1fd3-4dad-a48b-7e8aadedd9e0 Name: web01 Flavor: 512MB Standard Instance Image: Ubuntu 13.04 (Raring Ringtail) Metadata: [] RackConnect Wait: no ServiceLevel Wait: no ...
  40. Web Servers Add another Web Server

  41. $ knife rackspace server create -N web02 -f 2 -I

    1bbc5e56-ca2c-40a5-94b8-aa44822c3947 -r "role[my_app]" Create a Server Instance ID: 19fcf019-3126-4ebd-878b-1fe7b90dbecc Name: web02 Flavor: 512MB Standard Instance Image: Ubuntu 13.04 (Raring Ringtail) Metadata: [] RackConnect Wait: no ServiceLevel Wait: no ...
  42. App LB Web Servers Add a Load Balancer

  43. $ knife rackspace server create -N haproxy -f 2 -I

    1bbc5e56-ca2c-40a5-94b8-aa44822c3947 -r "role[haproxy]" Create a Load Balancer Instance ID: 54daecaf-dcb8-4c8b-8b3f-d6ede4ae5689 Name: haproxy Flavor: 512MB Standard Instance Image: Ubuntu 13.04 (Raring Ringtail) Metadata: [] RackConnect Wait: no ServiceLevel Wait: no ...
  44. App LB Web Servers Stitch together the configuration

  45. $ knife search "run_list:role\[demo_app\]" Search with Knife 2 items found

    Node Name: web01 FQDN: web01 IP: 192.237.214.213 Run List: role[demo_app] Node Name: web02 Environment: _default FQDN: web02 IP: 192.237.219.98 Run List: role[demo_app]
  46. pool_members = search("node", "role:#{node['haproxy']['app_server_role']}" <% @pool_members.each do |member| %> server

    <%= member[:hostname] %> <%= member[:ipaddress] %>:<%= node['haproxy']['member_port'] %> weight 1 <% end %> Search in Cookbooks • Search for all web servers • Add each to the Load Balancer configuration
  47. App LB Web Servers Add another web server

  48. $ knife rackspace server list List Servers Instance ID Name

    Public IP Private IP Flavor Image State 46383a09... web02 192.237.219.98 10.183.21.146 2 c6f9c41... active 795d968b... web01 192.237.214.213 10.183.21.152 2 c6f9c41... active d159c852... haproxy 192.237.222.93 10.183.25.200 2 c6f9c41... active
  49. $ knife search "*:*" -a platform -a platform_version More fun

    with knife 3 items found web01: platform: ubuntu platform_version: 12.10 web02: platform: ubuntu platform_version: 12.10 haproxy: platform: ubuntu platform_version: 12.10
  50. $ knife openstack knife openstack knife openstack flavor list (options)

    knife openstack group list (options) knife openstack image list (options) knife openstack server create (options) knife openstack server delete SERVER [SERVER] (options) knife openstack server list (options)
  51. $ knife openstack group list knife openstack group list Name

    Protocol From To CIDR Description default tcp 22 22 0.0.0.0/0 default default icmp -1 -1 0.0.0.0/0 default haproxy tcp 22002 22002 0.0.0.0/0 22022
  52. $ knife openstack image list knife openstack image list ID

    Name Snapshot 03860dc3-f4b5-4ecf-bb13-804d6618cf15 canonical-ubuntu-10.04-amd64 no 663656ce-2fe4-4164-b842-214f221cff55 canonical-ubuntu-12.04-amd64 no ad8a6e48-ea86-4afc-8aee-f427c02eb3ce canonical-ubuntu-13.04-amd64 no 6efbafc0-fcb1-4623-9f7a-17125bba413a centos-6.2 no e0184596-577f-4eb0-9887-d70117c6b77b debian-6.0.4-amd64 no
  53. $ knife openstack server list knife openstack server list Instance

    ID Name Public IP Private IP Flavor Image Keypair State 08f2d9f7-...-63aed8f096cc os-45... 50.56.12.229 2 737969f8-...-f3cf63bd25c5 rs-demo a-...- 43c6bbf5-...-392d955ce5b1 os-99... 50.56.12.232 2 737969f8-...-f3cf63bd25c5 rs-demo a-...- c1b9e3df-...-ed998b516608 os-55... 50.56.12.230 2 737969f8-...-f3cf63bd25c5 rs-demo a-...- f3edc5da-...-d957e09809e3 os-07... 50.56.12.231 2 737969f8-...-f3cf63bd25c5 rs-demo active
  54. $ knife openstack server create -a -f 2 -I 737...25c5

    -S rs- demo -x ubuntu -r "role[base]" knife openstack server create Instance Name: os-45539345723309377 Instance ID: 08f2d9f7-eeb0-45e7-8562-63aed8f096cc Waiting for server......... Flavor: 2 Image: 737969f8-6091-4896-ba9c-f3cf63bd25c5 SSH Identity File: /Users/mray/.ssh/rs-demo.pem SSH Keypair: rs-demo Public IP Address: 10.241.0.12 Floating IP Address: 50.56.12.229 Waiting for sshd.....done Bootstrapping Chef on 50.56.12.229 Instance Name: os-45539345723309377
  55. • Instant infrastructure • Unlimited capacity • Autoscaling • No

    commitment • Immediate replacement Why the Cloud?
  56. • Reliability • Performance • Security • Price Why not

    the Cloud?
  57. Data Gravity

  58. Know our escape plan for every infrastructure provider

  59. Chef for Infrastructure Portability • knife rackspace • knife openstack

    • knife azure • knife cloudstack • knife ec2 • knife google • knife hp • knife vsphere • ...and many others
  60. • AWS • Rackspace • HP • Google • Azure

    • many others ™ Desktop, Virtualization & Cloud • Vagrant • VMware • CloudStack • Eucalyptus • OpenStack • bare metal
  61. • AWS • Rackspace • HP • Google • Azure

    • many others Desktop, Virtualization & Cloud • Vagrant • VMware • CloudStack • Eucalyptus • OpenStack • bare metal ™
  62. What does this all mean? •Every infrastructure is a unique

    snowflake •Understand the costs associated with the features of your platform(s) of choice. •Chef enables Infrastructure Portability •Use the same infrastructure code for wherever you deploy •"Data Gravity" is a concern ™
  63. There’s a lot more... • Attributes • Environments • Roles

    • Chef Handlers • LWRPs • Workflow Helpers • Testing Frameworks • and even more awesome
  64. Continuous Delivery

  65. Thank You! • What questions do you have? • http://bit.ly/interacktive-chef

    - Slides • http://bit.ly/nhinteracktive - Code • nharvey@opscode.com • @nathenharvey