$30 off During Our Annual Pro Sale. View Details »

Achieving Infrastructure Portability with Chef

Nathen Harvey
February 22, 2013

Achieving Infrastructure Portability with Chef

Chef presentation from Build A Cloud Day at SCaLE 11x

Nathen Harvey

February 22, 2013
Tweet

More Decks by Nathen Harvey

Other Decks in Technology

Transcript

  1. Achieving Infrastructure Portability with Chef ScALE 11x Nathen Harvey nharvey@opscode.com

    IRC/Twitter/GitHub: nathenharvey www.opscode.com
  2. Data Centers...

  3. To the Cloud!

  4. • Instant infrastructure • Unlimited capacity • Autoscaling • No

    commitment • Immediate replacement Why the Cloud?
  5. • Each cloud defines themselves against Amazon • Entry into

    the market is easier • Not a lot of price competition • Feature parity is growing Cloud Differentiation
  6. The Dark Side of the Cloud

  7. • Reliability • Performance • Security • Price Why not

    the Cloud?
  8. Data Gravity

  9. Know our escape plan for every infrastructure provider

  10. Why Chef?

  11. See Node Application Server

  12. See Nodes Application Server Application Database

  13. See Nodes Grow Application Server Application Databases

  14. Application Servers Application Databases See Nodes Grow

  15. Application Servers Application Databases Load Balancer See Nodes Grow

  16. See Nodes Grow Application Servers Application Databases Load Balancers

  17. See Nodes Grow Application Servers Application Database Cache Load Balancers

    Application Databases
  18. Tied together with Config Application Servers Application Database Cache Load

    Balancers Application Databases
  19. Infrastructure is a Snowflake Application Servers Application Database Cache Load

    Balancers Floating IP? Application Databases
  20. Evolving Complexity Load Balancers Application Servers NoSQL Database Slaves ApplicationCache

    Database Cache Database
  21. Complexity Grows Quickly DC1 DC3 DC2

  22. http://www.flickr.com/photos/16339684@N00/2681435235/ And it Continues to Evolve That's great and all,

    but tell me about Chef!
  23. Chef is Infrastructure as Code http://www.flickr.com/photos/louisb/4555295187/ • Programmatically provision and

    configure • Treat like any other code base • Reconstruct business from code repository, data backup, and bare metal resources.
  24. • Chef-Client generates configurations directly on nodes from their run

    list • Reduce management complexity through abstraction • Store the configuration of your programs in version control http://www.flickr.com/photos/ssoosay/5126146763/ Nodes
  25. Collections of Resources • Networking • Files • Directories •

    Symlinks • Mounts • Routes • Users • Groups • Tasks • Packages • Software • Services • Configurations • Other Stuff http://www.flickr.com/photos/stevekeys/3123167585/
  26. Declarative Interface to Resources • Define policy • Say what,

    not how • Pull not Push http://www.flickr.com/photos/bixentro/2591838509/
  27. Ruby! extra_packages = case node['platform'] when "ubuntu","debian" %w{ ruby1.8 ruby1.8-dev

    rdoc1.8 ri1.8 libopenssl-ruby } end extra_packages.each do |pkg| package pkg do action :install end end
  28. Recipes and Cookbooks • Recipes are collections of Resources •

    Cookbooks contain recipes, templates, files, custom resources, etc • Code re-use and modularity • Hundreds already on Community.opscode.com http://www.flickr.com/photos/shutterhacks/4474421855/
  29. http://www.flickr.com/photos/kathycsus/2686772625 • IP addresses • Hostnames • FQDNs • Search

    for nodes with Roles • Find configuration data Search
  30. pool_members = search("node","role:webserver”) template "/etc/haproxy/haproxy.cfg" do source "haproxy-app_lb.cfg.erb" owner "root"

    group "root" mode 0644 variables :pool_members => pool_members.uniq notifies :restart, "service[haproxy]" end Pass Results to Templates
  31. # Set up application listeners here. listen application 0.0.0.0:80 balance

    roundrobin <% @pool_members.each do |member| -%> server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check <% end -%> <% if node["haproxy"]["enable_admin"] -%> listen admin 0.0.0.0:22002 mode http stats uri / <% end -%> Pass Results to Templates
  32. Jboss App Memcache Postgres Slaves Postgres Master So when this

    Nagios Graphite
  33. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite Becomes

    this
  34. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite Updates

    can be automatic
  35. Nagios Graphite Count the resources Jboss App Memcache Postgres Slaves

    • Load balancer config • Nagios host ping • Nagios host ssh • Nagios host HTTP • Nagios host app health • Graphite CPU • Graphite Memory • Graphite Disk • Graphite SNMP • Memcache firewall • Postgres firewall • Postgres authZ config • 12+ resource changes for 1 node addition
  36. Build anything • Simple internal applications • Complex external applications

    • Workstations • Hadoop clusters • IaaS infrastructure • PaaS infrastructure • SaaS applications • Storage systems • You name it http://www.flickr.com/photos/hyku/245010680/
  37. And manage it simply http://www.flickr.com/photos/helico/404640681/ • Automatically reconfigure everything •

    Linux, Windows, Unixes, BSDs • Load balancers • Metrics collection systems • Monitoring systems • Cloud migrations become trivial
  38. Code Sample Landscape of Chef-managed Infrastructure

  39. knife

  40. knife with the Chef Server • knife node • create/delete/edit

    • list • knife cookbook ... • knife role ... • knife environment ...
  41. knife bootstrap knife bootstrap SERVER -r 'role[webserver]' -i ~/.ssh/id_rsa •

    SSH to the machine given existing credentials • Install the Chef Client • Register with the Chef Server • Run the initial Run List • Now managed with Chef!
  42. knife ec2 $ knife ec2 Available ec2 subcommands: (for details,

    knife SUB-COMMAND -- help) ** EC2 COMMANDS ** knife ec2 flavor list (options) knife ec2 instance data (options) knife ec2 server create (options) knife ec2 server delete SERVER [SERVER] (options) knife ec2 server list (options) $ knife ec2 server create -S keypair -i ~/.ssh/id_rsa -x ubuntu -I ami-4721882e -f m1.small -r 'role[webserver]'
  43. knife openstack $ knife openstack Available openstack subcommands: (for details,

    knife SUB- COMMAND --help) ** OPENSTACK COMMANDS ** knife openstack flavor list (options) knife openstack image list (options) knife openstack server create (options) knife openstack server delete SERVER [SERVER] (options) knife openstack server list (options) $ knife openstack server create -S keypair -i ~/.ssh/id_rsa -x ubuntu -I 1231 -f standard.small -r 'role[webserver]'
  44. Chef for Infrastructure Portability • knife ec2 • knife rackspace

    • knife hp • knife google • knife azure • knife cloudstack • knife openstack • knife vsphere • ... and many others
  45. The Chef Community • Apache License, Version 2.0 • 850+

    Individual contributors • 150+ Corporate contributors • HP, Dell, Rackspace, VMware, Joyent, Calxeda, Heroku, SUSE and many more • 550+ cookbooks • http://community.opscode.com
  46. • Vagrant • VMware • CloudStack • Eucalyptus • OpenStack

    • bare metal • AWS • Rackspace • HP • Google • Azure • many others Desktop, Virtualization, Private & Public Clouds
  47. • Vagrant • VMware • CloudStack • Eucalyptus • OpenStack

    • bare metal Desktop, Virtualization, Private & Public Clouds • AWS • Rackspace • HP • Google • Azure • many others
  48. Tale of the Tape • Artur Bergman, CEO at Fastly

    • It's All About Speed • http://youtu.be/qRnTejOMbZU • Jason Stowe, CEO at Cycle Computing • CycleCloud + Chef = 50,000-core Utility Supercomputer for Science • http://youtu.be/cEaQB6e7G0Q
  49. Chef and Abstractions • Resources and Providers • Cookbooks (may)

    normalize deployment • Knife treats APIs the same • Chef strives to not be opinionated
  50. Environments • Lock down versions of cookbooks • stable releases

    vs. development • Enforce attributes for deployment • Ports, addresses, etc. • Different run lists based on environment • Debugging enabled?
  51. Environments • Use the same infrastructure code for wherever you

    deploy • Development, QA, Pre-Production, Prod • Role-based Access Controls to restrict the promotion of deployment code
  52. TL;DR • Every infrastructure is a unique snowflake • Understand

    the costs associated with the features of your platform(s) of choice. • Chef enables Infrastructure Portability • "Data Gravity" is the primary concern
  53. Further Resources http://bit.ly/cichi201302

  54. Further Resources • http://opscode.com/ • http://community.opscode.com/ • http://docs.opscode.com • http://wiki.opscode.com/

    • http://lists.opscode.com • http://youtube.com/user/Opscode
  55. Food Fight Show • http://foodfightshow.org • The Podcast Where DevOps

    Chef Do Battle • Regular updates about new Cookbooks, Knife-plugins, and more • Best Practices for working with Chef
  56. Get Involved Locally

  57. More Local User Groups • http://wiki.opscode.com/display/chef/ Community+Events

  58. More Training in LA • SOCAL-CHEF saves you $200 •

    http://opscode.eventbrite.com/
  59. #ChefConf 2013 Tex OPSCODE-SCALE - Save 10%

  60. Thanks! Nathen Harvey nharvey@opscode.com IRC/Twitter/GitHub: nathenharvey www.opscode.com