Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Prepping the Kitchen

someara
October 02, 2011

Prepping the Kitchen

Chef Concepts and Fundamental - Presented at Surge 2011

someara

October 02, 2011
Tweet

More Decks by someara

Other Decks in Technology

Transcript

  1. Infrastructure "It is common to think in terms of individual

    machines rather than view an entire infrastructure as a combined whole" “A good infrastructure, whether departmental, divisional, or enterprise-wide, is a single loosely- coupled virtual machine, with hundreds or thousands of hard drives and CPU's.” -- Bootstrapping an Infrastructure USENIX LISA ’98 http://www.infrastructures.org/papers/bootstrap/bootstrap.html
  2. .... as code! • Programmatically provision and configure • Treat

    like any other code base • Reconstruct operations from code repository, data backup, and bare metal resources. http://www.flickr.com/photos/louisb/4555295187/
  3. • Infrastructure changes over time • Entropy • Changing business

    requirements Considerations http://www.flickr.com/photos/seatbelt67/502255276/
  4. Manual Configuration • Labor intensive • Error prone • Hard

    to reproduce • Unsustainable http://www.flickr.com/photos/pureimaginations/4805330106/
  5. Scripting • Typically very brittle • Throw away, one off

    scripts • grep sed awk perl • curl | bash http://www.flickr.com/photos/40389360@N00/2428706650/
  6. File Distribution • NFS mounts • rdist • scp-on-a-for-loop •

    rsync on cron http://www.flickr.com/photos/walkadog/4317655660
  7. This used to be awesome for i in `cat servers.txt`

    ; do scp ntp.conf root@$i:/etc/ ntpd.conf ; done for i in `cat servers.txt` ; do ssh root@$i /etc/init.d/ntpd restart ; done for i in `cat servers.txt` ; do ssh root@$i chkconfig ntpd on ; done • ^ does not scale http://www.flickr.com/photos/alexerde/3479006495
  8. Declarative Syntax http://www.flickr.com/photos/bixentro/2591838509/ • Define policy • Say what, not

    how • Abstract interface to resources • Enables some interesting behavior
  9. Declarative Syntax • You’ll hear this a lot • Property

    of declarative interface • Eliminates brittleness of scripting • Identity function: f(x)=x package "ntp" do action :install end Idempotence service "ntpd" do action [:enable,:start] end cookbook_file "/etc/ntp.conf" do source "ntp.conf" owner "root" group "root" mode 0644 action :create notifies :restart, “service[ntpd]” end
  10. Declarative Syntax package "ntp" do action :install end Idempotence service

    "ntpd" do action [:enable,:start] end cookbook_file "/etc/ntp.conf" do source "ntp.conf" owner "root" group "root" mode 0644 action :create notifies :restart, “service[ntpd]” end while true do end • You’ll hear this a lot • Property of declarative interface • Eliminates brittleness of scripting • Identity function: f(x)=x • Safe to repeat
  11. Declarative Syntax • Agents “converge” a system to desired state

    • Repetition inches closer to desired state • It eventually gets there • SCIENCE! Convergence http://www.flickr.com/photos/tolomea/4852616645/
  12. Declarative Syntax Convergence package "ntp" do action :install ignore_failure true

    end service "ntpd" do action [:enable,:start] ignore_failure true end cookbook_file "/etc/ntp.conf" do source "ntp.conf" owner "root" group "root" mode 0644 action :create notifies :restart, “service[ntpd]” ignore_failure true end • Agents “converge” a system to desired state • Repetition inches closer to desired state • It eventually gets there • SCIENCE!
  13. Declarative Syntax Convergence • Fights entropy, unauthorized changes, and gingivitis

    • Update function inputs to deal with changing requirements # echo “boom” > /etc/ntp.conf ; \ chef-client $ grep server /etc/ntp.conf | head -n 1 us.pool.ntp.org $ ps -e | grep ntp 1799 ? 00:00:00 ntpd # /etc/init.d/ntpd stop ; chef-client ps -e | grep ntp 1822 ? 00:00:00 ntpd
  14. • Often made by hand (still!?) • Stop that. •

    Generate them based on database content • Infrastructures evolve Config Generation http://www.flickr.com/photos/jabella/4753170413/
  15. App LBs App Servers App DB Cache App DBs Stitched

    together with configs Floating IP?
  16. App LBs App Servers NoSQL DB slaves Cache DB Cache

    DBs Complexity increases quickly
  17. Generate configs • Centralized generation • Version control! • Distribute

    with packages, Chef, git, whatever. http://www.flickr.com/photos/ssoosay/5126146763/
  18. Generate configs • Local generation directly on nodes • Reduces

    management complexity • No need to distribute • Version control the programs instead http://www.flickr.com/photos/ssoosay/5126146763/
  19. • Declarative interface to resources • Database of nodes and

    their roles • Grab remote configs • Generate configs locally All That Stuff
  20. • Data Driven Infrastructure • Use APIs to obtain data

    • chef-server, SQL, anything. • Feed resources parameters • IPs, FQDNs, memory sizes, • Templates, package, firewall rules and more!
  21. Architecture • Code Repository • Chef Server • Chef Clients

    • Data Bags • Recipes and Cookbooks • Roles and Run Lists http://www.flickr.com/photos/boedker/3871267007
  22. Server Server Server Server chef-server Cookbook Knife Role Data Bag

    Knife Knife RESTful API Cookbook Cookbook Chef Server • Upload from laptop with knife
  23. Chef Clients Server Server Server Server chef-server RESTful API •

    Clients are API users • Read • Write • Search chef-client chef-client chef-client chef-client chef-client Knife Knife
  24. Chef Clients Server Server Server Server chef-server RESTful API chef-client

    • Clients are API users • Public keys on server • Private keys local to machines chef-client chef-client chef-client chef-client Knife Knife someara.pem jtimberman.pem node5.fqdn.pem someara.pub jtimberman.pub node5.fqdn.pub
  25. Run Lists Server Server Server Server chef-server API chef-client Ohai!

    Give me recipe[ntp::client] node ntp client.rb
  26. Run Lists Server Server Server Server chef-server API chef-client Ohai!

    Give me “ntp::client”, “openssh::server” node ntp client.rb openssh server.rb
  27. Run Lists Server Server Server Server chef-server API chef-client Ohai!

    Give me “recipe[ntp::client]”, “recipe[openssh::server]”, “recipe[apache]”, “recipe[php]” node ntp client.rb openssh server.rb apache default.rb php default.rb
  28. Roles Role Recipe Recipe Recipe Role Role Recipe Recipe Recipe

    Role Recipe Server Server Server Server chef-server API Knife
  29. Server Server Server Server chef-server API chef-client Ohai! Give me

    “role[base]”, “role[webserver]” node ntp client.rb openssh server.rb apache default.rb php default.rb Roles
  30. Server Server Server Server chef-server API chef-client “role[webserver]” node ntp

    client.rb openssh server.rb apache default.rb php default.rb Roles chef-client “role[database]” node ntp client.rb openssh server.rb mysql server.rb
  31. Bootstrapping nodes • Get chef-client installed • Write run list

    to a file • “Press go” http://www.flickr.com/photos/liftarn/1447521121/
  32. • knife ec2 server create -r ‘role [webserver]’ • knife

    bootstrap 10.9.8.7 -r ‘role[webserver]’ • Cobbler Bootstrapping nodes http://www.flickr.com/photos/hakonjarl/4010080214/
  33. Bootstrapping nodes • Ohai generates a JSON attributes list •

    Run list and attributes are combined into a Node object • Can be viewed and searched through API { "kernel": { "machine": "x86_64", "name": "Darwin", "os": "Darwin", "version": "Darwin Kernel Version 10.4.0: Fri Apr 23 18:28:53 PDT 2010; root:xnu-1504.7.4~1/RELEASE_I386", "release": "10.4.0" }, "platform_version": "10.6.4", "platform": "mac_os_x", "platform_build": "10F569", "domain": "local", "os": "darwin", "current_user": "mray", "ohai_time": 1278602661.60043, "os_version": "10.4.0", "uptime": "18 days 17 hours 49 minutes 18 seconds", "ipaddress": "10.13.37.116", "hostname": "morbo", "fqdn": "morbomorbo.local", "uptime_seconds": 1619358 }
  34. Cookbooks and Recipes • Cookbooks contain recipes • And everything

    they need to work • Templates, files, custom resources, etc http://www.flickr.com/photos/shutterhacks/4474421855/
  35. $ tree -a cookbooks/haproxy/ ᵓᴷᴷ README.md ᵓᴷᴷ attributes ᴹ ᵋᴷᴷ

    default.rb ᵓᴷᴷ metadata.rb ᵓᴷᴷ recipes ᴹ ᵓᴷᴷ app_lb.rb ᴹ ᵋᴷᴷ default.rb ᵋᴷᴷ templates ᵋᴷᴷ default ᵓᴷᴷ haproxy-app_lb.cfg.erb ᵓᴷᴷ haproxy-default.erb ᵋᴷᴷ haproxy.cfg.erb • Cookbooks contain recipes • And everything they need to work • Templates, files, custom resources, etc Cookbooks
  36. package "haproxy" do action :install end template "/etc/default/haproxy" do source

    "haproxy-default.erb" owner "root" group "root" mode 0644 notifies :restart, "service[haproxy]" end service "haproxy" do action [:enable, :start] end • Recipes contain lists of resources Recipes
  37. package "apache2" do version "2.2.11-2ubuntu2.6" action :install end template "/etc/apache2/apache2.conf"

    do source "apache2.conf.erb" owner "root" group "root" mode 0644 action :create end Resources
  38. • Have a type package "apache2" do version "2.2.11-2ubuntu2.6" action

    :install end template "/etc/apache2/apache2.conf" do source "apache2.conf.erb" owner "root" group "root" mode 0644 action :create end Resources
  39. • Have a type • Have a name package "apache2"

    do version "2.2.11-2ubuntu2.6" action :install end template "/etc/apache2/apache2.conf" do source "apache2.conf.erb" owner "root" group "root" mode 0644 action :create end Resources
  40. • Have a type • Have a name • Have

    parameters package "apache2" do version "2.2.11-2ubuntu2.6" action :install end template "/etc/apache2/apache2.conf" do source "apache2.conf.erb" owner "root" group "root" mode 0644 action :create end Resources
  41. • Have a type • Have a name • Have

    parameters • Take action to put the resource in the declared state package "apache2" do version "2.2.11-2ubuntu2.6" action :install end template "/etc/apache2/apache2.conf" do source "apache2.conf.erb" owner "root" group "root" mode 0644 action :create end Resources
  42. Searching • All object in Chef server are indexed by

    Solr http://www.flickr.com/photos/fotos_medem/3399096196/
  43. Searching • All object in Chef server are indexed by

    Solr • Can search through the API http://www.flickr.com/photos/fotos_medem/3399096196/
  44. Searching • All object in Chef server are indexed by

    Solr • Can search through the API • From knife and in recipes http://www.flickr.com/photos/fotos_medem/3399096196/
  45. Searching • All object in Chef server are indexed by

    Solr • Can search through the API • From knife and in recipes • Returns an array of JSON Node objects http://www.flickr.com/photos/fotos_medem/3399096196/
  46. pool_members = search("node","role:webserver”) template "/etc/haproxy/haproxy.cfg" do source "haproxy-app_lb.cfg.erb" owner "root"

    group "root" mode 0644 variables :pool_members => pool_members.uniq notifies :restart, "service[haproxy]" end Pass results into Templates
  47. pool_members = search("node","role:webserver”) template "/etc/haproxy/haproxy.cfg" do source "haproxy-app_lb.cfg.erb" owner "root"

    group "root" mode 0644 variables :pool_members => pool_members.uniq notifies :restart, "service[haproxy]" end Pass results into Templates
  48. # Set up application listeners here. listen application 0.0.0.0:80 balance

    roundrobin <% @pool_members.each do |member| -%> server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check <% end -%> <% if node["haproxy"]["enable_admin"] -%> listen admin 0.0.0.0:22002 mode http stats uri / <% end -%> Pass results into Templates
  49. Change • Various ways • Add or remove a node

    to the infrastructure • Run chef-client
  50. $ grep servers /etc/haproxy/haproxy.cfg servers node2.mylan 10.9.8.10 servers node3.mylan 10.9.8.11

    $ knife ec2 server create -r ‘webserver’ $ knife ec2 server create -r ‘webserver’ $ knife ssh ‘role:webserver’ chef-client $ grep servers /etc/haproxy/haproxy.cfg servers node2.mylan 10.9.8.10 servers node3.mylan 10.9.8.11 servers node4.mylan 10.9.8.12 servers node5.mylan 10.9.8.13 Run chef-client
  51. Change Inputs • Edit recipes • Edit run lists •

    chef-client http://www.flickr.com/photos/dhutchman/128541987