Chef and OpenStack - Speaker Deck

Slide 1

Slide 1 text

Chef and OpenStack Joshua Timberman [email protected] jtimberman Monday, May 20, 13

Slide 2

Slide 2 text

% whoami • System Administrator • I work for Opscode • I write Chef cookbooks totally legit mustache! Monday, May 20, 13

Slide 3

Slide 3 text

http://www.ﬂickr.com/photos/rhysasplundh/4807749216/ Monday, May 20, 13

Slide 4

Slide 4 text

Setting Expectations... • Both Chef and OpenStack are large frameworks • Opscode does 3 day training classes on Chef • Deploying OpenStack can take weeks, if not months • This talk is a technical overview/preview Monday, May 20, 13

Slide 5

Slide 5 text

We have a problem http://www.ﬂickr.com/photos/peterkaminski/2174679908/ Monday, May 20, 13

Slide 6

Slide 6 text

http://www.ﬂickr.com/photos/michaelheiss/3090102907/ Complexity Monday, May 20, 13

Slide 7

Slide 7 text

Items of Manipulation (Resources) • Nodes • Networking • Files • Directories • Symlinks • Mounts • Routes • Users • Groups • Packages • Services • Filesystems Monday, May 20, 13

Slide 8

Slide 8 text

Application A tale of growth... Monday, May 20, 13

Slide 9

Slide 9 text

Application Application Database Add a database Monday, May 20, 13

Slide 10

Slide 10 text

Application App Databases Make database redundant Monday, May 20, 13

Slide 11

Slide 11 text

App Servers App Databases Application server redundancy Monday, May 20, 13

Slide 12

Slide 12 text

App LB App Servers App Databases Add a load balancer Monday, May 20, 13

Slide 13

Slide 13 text

App LBs App Servers App Databases Webscale! Monday, May 20, 13

Slide 14

Slide 14 text

App LBs App Servers App DB Cache App DBs Now we need a caching layer Monday, May 20, 13

Slide 15

Slide 15 text

App LBs App Servers App DB Cache App DBs Infrastructure Has a Topology Monday, May 20, 13

Slide 16

Slide 16 text

Round Robin DNS App Servers App DB Cache App DBs Floating IP? Your's Is a Snowflake Monday, May 20, 13

Slide 17

Slide 17 text

App LBs App Servers < Shiny! DB slaves Cache DB Cache DBs Complexity Increases Quickly Are we monitoring?? Monday, May 20, 13

Slide 18

Slide 18 text

And that's just one app/site... Ruby App Python App Java App Monday, May 20, 13

Slide 19

Slide 19 text

It Increases Globally USA EUR APAC Monday, May 20, 13

Slide 20

Slide 20 text

OpenStack is similar... Monday, May 20, 13

Slide 21

Slide 21 text

Monday, May 20, 13

Slide 22

Slide 22 text

Why Chef? • Rackspace Cloud • AT&T • Dreamhost • HP Cloud • IBM SmartCloud • Cloudscaling • Others... Monday, May 20, 13

Slide 23

Slide 23 text

The Chef Framework • Reasonability • Flexibility • Library & Primitives • TIMTOWTDI • Sane defaults http://www.ﬂickr.com/photos/wonderlane/3609342683/sizes/l/in/photostream/ Monday, May 20, 13

Slide 24

Slide 24 text

The Chef Tool(s) • ohai • chef-client • chef-shell • knife • The Ruby language Omnibus - Full Stack Native Packages Monday, May 20, 13

Slide 25

Slide 25 text

The Chef Flavors • Chef Solo • Chef Server • Opscode Hosted Chef • Opscode Private Chef http://www.ﬂickr.com/photos/seanbuchandpt/6020113519/ Monday, May 20, 13

Slide 26

Slide 26 text

The Chef API • HTTPS, RESTful API w/ JSON, RSA key auth • Infrastructure data store such as node data • Search Service • Derivative Services? http://www.ﬂickr.com/photos/core-materials/4419853626/sizes/o/in/photostream/ Monday, May 20, 13

Slide 27

Slide 27 text

The Chef Community • Apache License, Version 2.0 • Hundreds of Individual and Corporate contributors. • Hundreds of cookbooks available from the community • http://community.opscode.com Monday, May 20, 13

Slide 28

Slide 28 text

How does it work? http://i3.kym-cdn.com/photos/images/original/000/046/123/magnets.jpg Monday, May 20, 13

Slide 29

Slide 29 text

Chef Enables Infrastructure as Code • Resources • Recipes • Cookbooks and Roles • Source Code http://www.ﬂickr.com/photos/louisb/4555295187/ Monday, May 20, 13

Slide 30

Slide 30 text

Resources and Recipes Monday, May 20, 13

Slide 31

Slide 31 text

package "openstack-nova-common" do action :install end directory "/etc/nova" do owner "nova" group "nova" mode 00700 end template "/etc/nova/nova.conf" do source "nova.conf.erb" owner "nova" group "nova" mode 00644 Declarative abstraction to system resources Monday, May 20, 13

Slide 32

Slide 32 text

package "openstack-nova-common" do action :install end template "/etc/nova/nova.conf" do source "nova.conf.erb" owner "nova" group "nova" mode 00644 notifies :restart, "service[openstack-nova-api]" end service "openstack-nova-api" do supports :status => true, :restart => true action [:enable, :start] end Monday, May 20, 13

Slide 33

Slide 33 text

Resources describe what Not how. Monday, May 20, 13

Slide 34

Slide 34 text

Resources take action through Providers Monday, May 20, 13

Slide 35

Slide 35 text

Providers perform the how Monday, May 20, 13

Slide 36

Slide 36 text

def install_package(name, version) package_name = "#{name}=#{version}" package_name = name if @is_virtual_package run_command_with_systems_locale( :command => "apt-get -q -y #{expand_options(@new_resource.options)} install #{package_name}", :environment => { "DEBIAN_FRONTEND" => "noninteractive" } ) end Provider Example Monday, May 20, 13

Slide 37

Slide 37 text

Package Resource package "git" { yum install git apt-get install git pacman sync git pkg_add -r git Providers are determined by node's platform Monday, May 20, 13

Slide 38

Slide 38 text

Cookbooks • Cookbooks are collections of Recipes • Cookbooks contain related components • Files, Templates, Libraries • A cookbook is responsible for conﬁguring a single thing, e.g. • nova • postgresql • A recipe is responsible for a component, e.g. • api • server • client Monday, May 20, 13

Slide 39

Slide 39 text

Example cookbook openstack-compute ├── README.md ├── attributes └── default.rb ├── files └── default └── nova-compute.conf ├── metadata.rb ├── recipes ├── api-ec2.rb ├── default.rb └── nova-common.rb └── templates └── default └── nova.conf.erb >openstack-compute.tar.gz Monday, May 20, 13

Slide 40

Slide 40 text

Roles • Roles describe nodes • Roles indicate functionality • Roles have a ﬁrst class API primitive Monday, May 20, 13

Slide 41

Slide 41 text

Example Role name "os-compute-api" description "OpenStack API for Compute" run_list( "role[os-base]", "recipe[openstack-compute]" ) Monday, May 20, 13

Slide 42

Slide 42 text

http://www.ﬂickr.com/photos/peterrosbjerg/3913766224/ Chef Nodes • Chef nodes do the heavy lifting • Authority about themselves • Stored on the server when using Chef Server • Indexed for search Monday, May 20, 13

Slide 43

Slide 43 text

Search • Search for nodes with Roles • Find Topology Data • IP addresses • Hostnames • FQDNs http://www.ﬂickr.com/photos/kathycsus/2686772625 Monday, May 20, 13

Slide 44

Slide 44 text

Manage OpenStack with Chef Monday, May 20, 13

Slide 45

Slide 45 text

OpenStack is complex • Nova (compute) • Swift (object store) • Glance (image) • Quantum (network) • Keystone (identity) • Cinder (block storage) • Horizon (dashboard) Monday, May 20, 13

Slide 46

Slide 46 text

Deploying OpenStack • Start with the Chef Repository • Upload Chef components to a Chef Server • "Bootstrap" nodes with the appropriate role • role[os-ops-database] (mysql/postgresql) • role[os-ops-messaging] (rabbitmq) • role[os-identity] (Keystone) • role[os-image] (Glance) • role[os-network] (Quantum) • role[os-compute-scheduler] (Nova) • role[os-compute-api] (Nova) • role[os-block-storage] (Cinder) • role[os-dashboard] (Horizon) Monday, May 20, 13

Slide 47

Slide 47 text

Chef for OpenStack: Project • Community for the automated deployment and management of OpenStack • Reduce fragmentation and encourage collaboration • Deploying OpenStack is not "secret sauce" • Project not a product • Apache 2 license Monday, May 20, 13

Slide 48

Slide 48 text

Various Projects and Code • Chef Repository for Deploying OpenStack • Documentation for Chef for OpenStack • Cookbooks for each component • Knife plugin Monday, May 20, 13

Slide 49

Slide 49 text

Get Involved • groups.google.com/group/opscode-chef-openstack • #openstack-chef on irc.freenode.net • github.com/mattray/chef-openstack-repo • @chefopenstack, @mattray Monday, May 20, 13

Slide 50

Slide 50 text

Manage Infrastructure in OpenStack with Chef Monday, May 20, 13

Slide 51

Slide 51 text

knife openstack % knife openstack knife openstack flavor list knife openstack group list knife openstack image list knife openstack server create knife openstack server delete knife openstack server list Monday, May 20, 13

Slide 52

Slide 52 text

http://www.ﬂickr.com/photos/myklroventine/3474391066/ Knife is your CLI to Chef Monday, May 20, 13

Slide 53

Slide 53 text

knife openstack server create % grep openstack ~/.chef/knife.rb knife[:openstack_ssh_key_id] = ENV['OPENSTACK_SSH_KEY'] knife[:openstack_username] = ENV['OPENSTACK_USERNAME'] knife[:openstack_password] = ENV['OPENSTACK_PASSWORD'] knife[:openstack_tenant] = ENV['OPENSTACK_TENANT'] knife[:openstack_auth_url] = ENV['OPENSTACK_AUTH_URL'] Monday, May 20, 13

Slide 54

Slide 54 text

knife openstack server create knife openstack server create \ -x root \ -f 7 \ -S jtimberman \ -I e0184596-577f-4eb0-9887-d70117c6b77b \ -r 'role[frontend-app]' Monday, May 20, 13

Slide 55

Slide 55 text

knife openstack server create knife openstack server create \ -x root \ -f 7 \ -S jtimberman \ -I e0184596-577f-4eb0-9887-d70117c6b77b \ -r 'role[frontend-app]' SSH as this user Use this SSH keypair Monday, May 20, 13

Slide 56

Slide 56 text

knife openstack server create knife openstack server create \ -x root \ -f 7 \ -S jtimberman \ -I e0184596-577f-4eb0-9887-d70117c6b77b \ -r 'role[frontend-app]' OpenStack "ﬂavor" identiﬁer Monday, May 20, 13

Slide 57

Slide 57 text

knife openstack server create knife openstack server create \ -x root \ -f 7 \ -S jtimberman \ -I e0184596-577f-4eb0-9887-d70117c6b77b \ -r 'role[frontend-app]' OpenStack "image" (e.g., Debian 6.0) Monday, May 20, 13

Slide 58

Slide 58 text

knife openstack server create knife openstack server create \ -x root \ -f 7 \ -S jtimberman \ -I e0184596-577f-4eb0-9887-d70117c6b77b \ -r 'role[frontend-app]' Chef "run list" for the node Monday, May 20, 13

Slide 59

Slide 59 text

knife openstack server create Instance Name: os-7971334684686417 Instance ID: e609a9bd-8e64-4184-94b6-4ef1ca7b9909 Waiting for server.................. Flavor: 7 Image: e0184596-577f-4eb0-9887-d70117c6b77b SSH Keypair: jtimberman Public IP Address: 172.31.6.54 Waiting for sshd..done Bootstrapping Chef on 172.31.6.54 172.31.6.54 Setting up chef (11.4.4-2.debian.6.0.5) ... 172.31.6.54 Thank you for installing Chef! 172.31.6.54 Starting Chef Client, version 11.4.4 172.31.6.54 Creating a new client identity for os-7971334684686417 using the validator key. 172.31.6.54 resolving cookbooks for run list: ["apt", "apache2"] Monday, May 20, 13

Slide 60

Slide 60 text

Chef Client Runs, Victory! 172.31.6.54 Chef Client finished, 18 resources updated Instance Name: os-7971334684686417 Instance ID: e609a9bd-8e64-4184-94b6-4ef1ca7b9909 Flavor: 7 Image: e0184596-577f-4eb0-9887-d70117c6b77b SSH Keypair: jtimberman Public IP Address: 172.31.6.54 Environment: _default Run List: role[frontend-app] Monday, May 20, 13

Slide 61

Slide 61 text

Monday, May 20, 13

Slide 62

Slide 62 text

knife node show % knife node show os-7971334684686417 Node Name: os-7971334684686417 Environment: _default FQDN: debianx8664.opscode.us IP: 172.31.6.54 Run List: role[frontend-app] Roles: frontend-app Recipes: apt, apache2 Platform: debian 6.0.4 Tags: % knife node show os-7971334684686417 -a cloud.provider os-7971334684686417: cloud.provider: openstack Talks to Chef Server Monday, May 20, 13

Slide 63

Slide 63 text

knife-openstack compatibility • Uses the OpenStack API • Diablo, Essex, Folsom, Grizzly • Cloudscaling • Crowbar • DreamHost • Nebula • Piston • Rackspace Private Cloud Monday, May 20, 13

Slide 64

Slide 64 text

Chef for Infrastructure Portability • knife openstack • Crowbar, DreamHost, Nebula, Piston • knife hp • knife rackspace • knife cloudstack • knife ec2 • knife google • Consolidation coming soon Monday, May 20, 13

Slide 65

Slide 65 text

Summary • Big players are betting on OpenStack • Many orgs are using Chef to deploy OpenStack • There's a growing community of users for Chef + OpenStack http://www.ﬂickr.com/photos/villes/358790270/ Monday, May 20, 13

Slide 66

Slide 66 text

Questions? Joshua Timberman -@jtimberman [email protected] Resources: opscode.com/chef learnchef.com community.opscode.com groups.google.com/group/opscode-chef-openstack Monday, May 20, 13