Achieving Infrastructure Portability with Chef

Chef presentation from Build A Cloud Day at SCaLE 11x

Transcript

1. Achieving Infrastructure Portability with Chef ScALE 11x Nathen Harvey nharvey@opscode.com IRC/Twitter/GitHub: nathenharvey www.opscode.com
2. Data Centers...
3. To the Cloud!
4. • Instant infrastructure • Unlimited capacity • Autoscaling • No commitment • Immediate replacement Why the Cloud?
5. • Each cloud defines themselves against Amazon • Entry into the market is easier • Not a lot of price competition • Feature parity is growing Cloud Differentiation
6. The Dark Side of the Cloud
7. • Reliability • Performance • Security • Price Why not the Cloud?
8. Data Gravity
9. Know our escape plan for every infrastructure provider
10. Why Chef?
11. See Node Application Server
12. See Nodes Application Server Application Database
13. See Nodes Grow Application Server Application Databases
14. Application Servers Application Databases See Nodes Grow
15. Application Servers Application Databases Load Balancer See Nodes Grow
16. See Nodes Grow Application Servers Application Databases Load Balancers
17. See Nodes Grow Application Servers Application Database Cache Load Balancers Application Databases
18. Tied together with Config Application Servers Application Database Cache Load Balancers Application Databases
19. Infrastructure is a Snowflake Application Servers Application Database Cache Load Balancers Floating IP? Application Databases
20. Evolving Complexity Load Balancers Application Servers NoSQL Database Slaves ApplicationCache Database Cache Database
21. Complexity Grows Quickly DC1 DC3 DC2
22. http://www.flickr.com/photos/16339684@N00/2681435235/ And it Continues to Evolve That's great and all, but tell me about Chef!
23. Chef is Infrastructure as Code http://www.flickr.com/photos/louisb/4555295187/ • Programmatically provision and configure • Treat like any other code base • Reconstruct business from code repository, data backup, and bare metal resources.
24. • Chef-Client generates configurations directly on nodes from their run list • Reduce management complexity through abstraction • Store the configuration of your programs in version control http://www.flickr.com/photos/ssoosay/5126146763/ Nodes
25. Collections of Resources • Networking • Files • Directories • Symlinks • Mounts • Routes • Users • Groups • Tasks • Packages • Software • Services • Configurations • Other Stuff http://www.flickr.com/photos/stevekeys/3123167585/
26. Declarative Interface to Resources • Define policy • Say what, not how • Pull not Push http://www.flickr.com/photos/bixentro/2591838509/
27. Ruby! extra_packages = case node['platform'] when "ubuntu","debian" %w{ ruby1.8 ruby1.8-dev rdoc1.8 ri1.8 libopenssl-ruby } end extra_packages.each do |pkg| package pkg do action :install end end
28. Recipes and Cookbooks • Recipes are collections of Resources • Cookbooks contain recipes, templates, files, custom resources, etc • Code re-use and modularity • Hundreds already on Community.opscode.com http://www.flickr.com/photos/shutterhacks/4474421855/
29. http://www.flickr.com/photos/kathycsus/2686772625 • IP addresses • Hostnames • FQDNs • Search for nodes with Roles • Find configuration data Search
30. pool_members = search("node","role:webserver”) template "/etc/haproxy/haproxy.cfg" do source "haproxy-app_lb.cfg.erb" owner "root" group "root" mode 0644 variables :pool_members => pool_members.uniq notifies :restart, "service[haproxy]" end Pass Results to Templates
31. # Set up application listeners here. listen application 0.0.0.0:80 balance roundrobin <% @pool_members.each do |member| -%> server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check <% end -%> <% if node["haproxy"]["enable_admin"] -%> listen admin 0.0.0.0:22002 mode http stats uri / <% end -%> Pass Results to Templates
32. Jboss App Memcache Postgres Slaves Postgres Master So when this Nagios Graphite
33. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite Becomes this
34. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite Updates can be automatic
35. Nagios Graphite Count the resources Jboss App Memcache Postgres Slaves • Load balancer config • Nagios host ping • Nagios host ssh • Nagios host HTTP • Nagios host app health • Graphite CPU • Graphite Memory • Graphite Disk • Graphite SNMP • Memcache firewall • Postgres firewall • Postgres authZ config • 12+ resource changes for 1 node addition
36. Build anything • Simple internal applications • Complex external applications • Workstations • Hadoop clusters • IaaS infrastructure • PaaS infrastructure • SaaS applications • Storage systems • You name it http://www.flickr.com/photos/hyku/245010680/
37. And manage it simply http://www.flickr.com/photos/helico/404640681/ • Automatically reconfigure everything • Linux, Windows, Unixes, BSDs • Load balancers • Metrics collection systems • Monitoring systems • Cloud migrations become trivial
38. Code Sample Landscape of Chef-managed Infrastructure
39. knife
40. knife with the Chef Server • knife node • create/delete/edit • list • knife cookbook ... • knife role ... • knife environment ...
41. knife bootstrap knife bootstrap SERVER -r 'role[webserver]' -i ~/.ssh/id_rsa • SSH to the machine given existing credentials • Install the Chef Client • Register with the Chef Server • Run the initial Run List • Now managed with Chef!
42. knife ec2 $ knife ec2 Available ec2 subcommands: (for details, knife SUB-COMMAND -- help) ** EC2 COMMANDS ** knife ec2 flavor list (options) knife ec2 instance data (options) knife ec2 server create (options) knife ec2 server delete SERVER [SERVER] (options) knife ec2 server list (options) $ knife ec2 server create -S keypair -i ~/.ssh/id_rsa -x ubuntu -I ami-4721882e -f m1.small -r 'role[webserver]'
43. knife openstack $ knife openstack Available openstack subcommands: (for details, knife SUB- COMMAND --help) ** OPENSTACK COMMANDS ** knife openstack flavor list (options) knife openstack image list (options) knife openstack server create (options) knife openstack server delete SERVER [SERVER] (options) knife openstack server list (options) $ knife openstack server create -S keypair -i ~/.ssh/id_rsa -x ubuntu -I 1231 -f standard.small -r 'role[webserver]'
44. Chef for Infrastructure Portability • knife ec2 • knife rackspace • knife hp • knife google • knife azure • knife cloudstack • knife openstack • knife vsphere • ... and many others
45. The Chef Community • Apache License, Version 2.0 • 850+ Individual contributors • 150+ Corporate contributors • HP, Dell, Rackspace, VMware, Joyent, Calxeda, Heroku, SUSE and many more • 550+ cookbooks • http://community.opscode.com
46. • Vagrant • VMware • CloudStack • Eucalyptus • OpenStack • bare metal • AWS • Rackspace • HP • Google • Azure • many others Desktop, Virtualization, Private & Public Clouds
47. • Vagrant • VMware • CloudStack • Eucalyptus • OpenStack • bare metal Desktop, Virtualization, Private & Public Clouds • AWS • Rackspace • HP • Google • Azure • many others
48. Tale of the Tape • Artur Bergman, CEO at Fastly • It's All About Speed • http://youtu.be/qRnTejOMbZU • Jason Stowe, CEO at Cycle Computing • CycleCloud + Chef = 50,000-core Utility Supercomputer for Science • http://youtu.be/cEaQB6e7G0Q
49. Chef and Abstractions • Resources and Providers • Cookbooks (may) normalize deployment • Knife treats APIs the same • Chef strives to not be opinionated
50. Environments • Lock down versions of cookbooks • stable releases vs. development • Enforce attributes for deployment • Ports, addresses, etc. • Different run lists based on environment • Debugging enabled?
51. Environments • Use the same infrastructure code for wherever you deploy • Development, QA, Pre-Production, Prod • Role-based Access Controls to restrict the promotion of deployment code
52. TL;DR • Every infrastructure is a unique snowflake • Understand the costs associated with the features of your platform(s) of choice. • Chef enables Infrastructure Portability • "Data Gravity" is the primary concern
53. Further Resources http://bit.ly/cichi201302
54. Further Resources • http://opscode.com/ • http://community.opscode.com/ • http://docs.opscode.com • http://wiki.opscode.com/ • http://lists.opscode.com • http://youtube.com/user/Opscode
55. Food Fight Show • http://foodfightshow.org • The Podcast Where DevOps Chef Do Battle • Regular updates about new Cookbooks, Knife-plugins, and more • Best Practices for working with Chef
56. Get Involved Locally
57. More Local User Groups • http://wiki.opscode.com/display/chef/ Community+Events
58. More Training in LA • SOCAL-CHEF saves you $200 • http://opscode.eventbrite.com/
59. #ChefConf 2013 Tex OPSCODE-SCALE - Save 10%
60. Thanks! Nathen Harvey nharvey@opscode.com IRC/Twitter/GitHub: nathenharvey www.opscode.com