Chef and OpenStack

Presentation from my talk at OpenStack Denver 5/20/2013

Transcript

1. Chef and OpenStack Joshua Timberman joshua@opscode.com jtimberman Monday, May 20, 13
2. % whoami • System Administrator • I work for Opscode • I write Chef cookbooks totally legit mustache! Monday, May 20, 13
3. http://www.flickr.com/photos/rhysasplundh/4807749216/ Monday, May 20, 13
4. Setting Expectations... • Both Chef and OpenStack are large frameworks • Opscode does 3 day training classes on Chef • Deploying OpenStack can take weeks, if not months • This talk is a technical overview/preview Monday, May 20, 13
5. We have a problem http://www.flickr.com/photos/peterkaminski/2174679908/ Monday, May 20, 13
6. http://www.flickr.com/photos/michaelheiss/3090102907/ Complexity Monday, May 20, 13
7. Items of Manipulation (Resources) • Nodes • Networking • Files • Directories • Symlinks • Mounts • Routes • Users • Groups • Packages • Services • Filesystems Monday, May 20, 13
8. Application A tale of growth... Monday, May 20, 13
9. Application Application Database Add a database Monday, May 20, 13
10. Application App Databases Make database redundant Monday, May 20, 13
11. App Servers App Databases Application server redundancy Monday, May 20, 13
12. App LB App Servers App Databases Add a load balancer Monday, May 20, 13
13. App LBs App Servers App Databases Webscale! Monday, May 20, 13
14. App LBs App Servers App DB Cache App DBs Now we need a caching layer Monday, May 20, 13
15. App LBs App Servers App DB Cache App DBs Infrastructure Has a Topology Monday, May 20, 13
16. Round Robin DNS App Servers App DB Cache App DBs Floating IP? Your's Is a Snowflake Monday, May 20, 13
17. App LBs App Servers < Shiny! DB slaves Cache DB Cache DBs Complexity Increases Quickly Are we monitoring?? Monday, May 20, 13
18. And that's just one app/site... Ruby App Python App Java App Monday, May 20, 13
19. It Increases Globally USA EUR APAC Monday, May 20, 13
20. OpenStack is similar... Monday, May 20, 13
21. Monday, May 20, 13
22. Why Chef? • Rackspace Cloud • AT&T • Dreamhost • HP Cloud • IBM SmartCloud • Cloudscaling • Others... Monday, May 20, 13
23. The Chef Framework • Reasonability • Flexibility • Library & Primitives • TIMTOWTDI • Sane defaults http://www.flickr.com/photos/wonderlane/3609342683/sizes/l/in/photostream/ Monday, May 20, 13
24. The Chef Tool(s) • ohai • chef-client • chef-shell • knife • The Ruby language Omnibus - Full Stack Native Packages Monday, May 20, 13
25. The Chef Flavors • Chef Solo • Chef Server • Opscode Hosted Chef • Opscode Private Chef http://www.flickr.com/photos/seanbuchandpt/6020113519/ Monday, May 20, 13
26. The Chef API • HTTPS, RESTful API w/ JSON, RSA key auth • Infrastructure data store such as node data • Search Service • Derivative Services? http://www.flickr.com/photos/core-materials/4419853626/sizes/o/in/photostream/ Monday, May 20, 13
27. The Chef Community • Apache License, Version 2.0 • Hundreds of Individual and Corporate contributors. • Hundreds of cookbooks available from the community • http://community.opscode.com Monday, May 20, 13
28. How does it work? http://i3.kym-cdn.com/photos/images/original/000/046/123/magnets.jpg Monday, May 20, 13
29. Chef Enables Infrastructure as Code • Resources • Recipes • Cookbooks and Roles • Source Code http://www.flickr.com/photos/louisb/4555295187/ Monday, May 20, 13
30. Resources and Recipes Monday, May 20, 13
31. package "openstack-nova-common" do action :install end directory "/etc/nova" do owner "nova" group "nova" mode 00700 end template "/etc/nova/nova.conf" do source "nova.conf.erb" owner "nova" group "nova" mode 00644 Declarative abstraction to system resources Monday, May 20, 13
32. package "openstack-nova-common" do action :install end template "/etc/nova/nova.conf" do source "nova.conf.erb" owner "nova" group "nova" mode 00644 notifies :restart, "service[openstack-nova-api]" end service "openstack-nova-api" do supports :status => true, :restart => true action [:enable, :start] end Monday, May 20, 13
33. Resources describe what Not how. Monday, May 20, 13
34. Resources take action through Providers Monday, May 20, 13
35. Providers perform the how Monday, May 20, 13
36. def install_package(name, version) package_name = "#{name}=#{version}" package_name = name if @is_virtual_package run_command_with_systems_locale( :command => "apt-get -q -y #{expand_options(@new_resource.options)} install #{package_name}", :environment => { "DEBIAN_FRONTEND" => "noninteractive" } ) end Provider Example Monday, May 20, 13
37. Package Resource package "git" { yum install git apt-get install git pacman sync git pkg_add -r git Providers are determined by node's platform Monday, May 20, 13
38. Cookbooks • Cookbooks are collections of Recipes • Cookbooks contain related components • Files, Templates, Libraries • A cookbook is responsible for configuring a single thing, e.g. • nova • postgresql • A recipe is responsible for a component, e.g. • api • server • client Monday, May 20, 13
39. Example cookbook openstack-compute ├── README.md ├── attributes └── default.rb ├── files └── default └── nova-compute.conf ├── metadata.rb ├── recipes ├── api-ec2.rb ├── default.rb └── nova-common.rb └── templates └── default └── nova.conf.erb >openstack-compute.tar.gz Monday, May 20, 13
40. Roles • Roles describe nodes • Roles indicate functionality • Roles have a first class API primitive Monday, May 20, 13
41. Example Role name "os-compute-api" description "OpenStack API for Compute" run_list( "role[os-base]", "recipe[openstack-compute]" ) Monday, May 20, 13
42. http://www.flickr.com/photos/peterrosbjerg/3913766224/ Chef Nodes • Chef nodes do the heavy lifting • Authority about themselves • Stored on the server when using Chef Server • Indexed for search Monday, May 20, 13
43. Search • Search for nodes with Roles • Find Topology Data • IP addresses • Hostnames • FQDNs http://www.flickr.com/photos/kathycsus/2686772625 Monday, May 20, 13
44. Manage OpenStack with Chef Monday, May 20, 13
45. OpenStack is complex • Nova (compute) • Swift (object store) • Glance (image) • Quantum (network) • Keystone (identity) • Cinder (block storage) • Horizon (dashboard) Monday, May 20, 13
46. Deploying OpenStack • Start with the Chef Repository • Upload Chef components to a Chef Server • "Bootstrap" nodes with the appropriate role • role[os-ops-database] (mysql/postgresql) • role[os-ops-messaging] (rabbitmq) • role[os-identity] (Keystone) • role[os-image] (Glance) • role[os-network] (Quantum) • role[os-compute-scheduler] (Nova) • role[os-compute-api] (Nova) • role[os-block-storage] (Cinder) • role[os-dashboard] (Horizon) Monday, May 20, 13
47. Chef for OpenStack: Project • Community for the automated deployment and management of OpenStack • Reduce fragmentation and encourage collaboration • Deploying OpenStack is not "secret sauce" • Project not a product • Apache 2 license Monday, May 20, 13
48. Various Projects and Code • Chef Repository for Deploying OpenStack • Documentation for Chef for OpenStack • Cookbooks for each component • Knife plugin Monday, May 20, 13
49. Get Involved • groups.google.com/group/opscode-chef-openstack • #openstack-chef on irc.freenode.net • github.com/mattray/chef-openstack-repo • @chefopenstack, @mattray Monday, May 20, 13
50. Manage Infrastructure in OpenStack with Chef Monday, May 20, 13
51. knife openstack % knife openstack knife openstack flavor list knife openstack group list knife openstack image list knife openstack server create knife openstack server delete knife openstack server list Monday, May 20, 13
52. http://www.flickr.com/photos/myklroventine/3474391066/ Knife is your CLI to Chef Monday, May 20, 13
53. knife openstack server create % grep openstack ~/.chef/knife.rb knife[:openstack_ssh_key_id] = ENV['OPENSTACK_SSH_KEY'] knife[:openstack_username] = ENV['OPENSTACK_USERNAME'] knife[:openstack_password] = ENV['OPENSTACK_PASSWORD'] knife[:openstack_tenant] = ENV['OPENSTACK_TENANT'] knife[:openstack_auth_url] = ENV['OPENSTACK_AUTH_URL'] Monday, May 20, 13
54. knife openstack server create knife openstack server create \ -x root \ -f 7 \ -S jtimberman \ -I e0184596-577f-4eb0-9887-d70117c6b77b \ -r 'role[frontend-app]' Monday, May 20, 13
55. knife openstack server create knife openstack server create \ -x root \ -f 7 \ -S jtimberman \ -I e0184596-577f-4eb0-9887-d70117c6b77b \ -r 'role[frontend-app]' SSH as this user Use this SSH keypair Monday, May 20, 13
56. knife openstack server create knife openstack server create \ -x root \ -f 7 \ -S jtimberman \ -I e0184596-577f-4eb0-9887-d70117c6b77b \ -r 'role[frontend-app]' OpenStack "flavor" identifier Monday, May 20, 13
57. knife openstack server create knife openstack server create \ -x root \ -f 7 \ -S jtimberman \ -I e0184596-577f-4eb0-9887-d70117c6b77b \ -r 'role[frontend-app]' OpenStack "image" (e.g., Debian 6.0) Monday, May 20, 13
58. knife openstack server create knife openstack server create \ -x root \ -f 7 \ -S jtimberman \ -I e0184596-577f-4eb0-9887-d70117c6b77b \ -r 'role[frontend-app]' Chef "run list" for the node Monday, May 20, 13
59. knife openstack server create Instance Name: os-7971334684686417 Instance ID: e609a9bd-8e64-4184-94b6-4ef1ca7b9909 Waiting for server.................. Flavor: 7 Image: e0184596-577f-4eb0-9887-d70117c6b77b SSH Keypair: jtimberman Public IP Address: 172.31.6.54 Waiting for sshd..done Bootstrapping Chef on 172.31.6.54 172.31.6.54 Setting up chef (11.4.4-2.debian.6.0.5) ... 172.31.6.54 Thank you for installing Chef! 172.31.6.54 Starting Chef Client, version 11.4.4 172.31.6.54 Creating a new client identity for os-7971334684686417 using the validator key. 172.31.6.54 resolving cookbooks for run list: ["apt", "apache2"] Monday, May 20, 13
60. Chef Client Runs, Victory! 172.31.6.54 Chef Client finished, 18 resources updated Instance Name: os-7971334684686417 Instance ID: e609a9bd-8e64-4184-94b6-4ef1ca7b9909 Flavor: 7 Image: e0184596-577f-4eb0-9887-d70117c6b77b SSH Keypair: jtimberman Public IP Address: 172.31.6.54 Environment: _default Run List: role[frontend-app] Monday, May 20, 13
61. Monday, May 20, 13
62. knife node show % knife node show os-7971334684686417 Node Name: os-7971334684686417 Environment: _default FQDN: debianx8664.opscode.us IP: 172.31.6.54 Run List: role[frontend-app] Roles: frontend-app Recipes: apt, apache2 Platform: debian 6.0.4 Tags: % knife node show os-7971334684686417 -a cloud.provider os-7971334684686417: cloud.provider: openstack Talks to Chef Server Monday, May 20, 13
63. knife-openstack compatibility • Uses the OpenStack API • Diablo, Essex, Folsom, Grizzly • Cloudscaling • Crowbar • DreamHost • Nebula • Piston • Rackspace Private Cloud Monday, May 20, 13
64. Chef for Infrastructure Portability • knife openstack • Crowbar, DreamHost, Nebula, Piston • knife hp • knife rackspace • knife cloudstack • knife ec2 • knife google • Consolidation coming soon Monday, May 20, 13
65. Summary • Big players are betting on OpenStack • Many orgs are using Chef to deploy OpenStack • There's a growing community of users for Chef + OpenStack http://www.flickr.com/photos/villes/358790270/ Monday, May 20, 13
66. Questions? Joshua Timberman -@jtimberman joshua@opscode.com Resources: opscode.com/chef learnchef.com community.opscode.com groups.google.com/group/opscode-chef-openstack Monday, May 20, 13