Share
Embed
Ohio LinuxFest 2012: Servers So Easy A Caveman Can Do It
Automate your infrastructure needs using Ruby, Vagrant, EC2/Rackspace, Chef and Capistrano. New employee needs to setup their MBP to run your Rails app? Want to get them a cloud server on day one to develop against? Want to deploy a git branch for people to use? Need to add 3 new production servers to handle a traffic spike in a hurry? With ruby, some command line tools and a few Chef recipes, you (or anyone n your team) can take the tedium out of these maintenance drains and get back to working on the product.
Chris Laco
September 21, 2012
Transcript
58 slides
-
Servers So Easy A Caveman Can Do It Christopher H.
Laco » [email protected] » @claco 1 1 -
2 Legal Disclaimer Ohio Revised Code 314159 Section (c), Subsection
(a), Paragraph (t) states in part: “...any person or persons presenting computer related content to a group [audience] is required to show at a minimum of at least one (1) domestic feline [cat] picture during previously stated presentation.” 2 -
Your humble speaker Reformed Music Major Turned Nerd H.A.H.S. Hosting
At Home Syndrom S.I.B.D. Servers In Basement Disease Full Stack Nerd Hardware/Software/Network/”DevOps” 5 5 -
“Back in my day...” 11 “And we liked it...” “WE
LOVED IT!” 11 -
Aggravations Your First Day. Computer? Configured? Configuration required the “server
guy” MBP setup is different than servers Testing locally is different than testing upstream “Works on my machine” is dangerous Troubleshooting production is risky/difficult Upgrades introduce risk 12 12 -
Aggravations continued... Changing deployment logic is troublesome Disaster recovery is
costly/long Adding capacity takes time Nothing is repeatable Managing multiple servers is tedious 13 13 -
Goals Get app instance on Day #1 Any Engineer/QA can
spin up machines Same configuration everywhere Test outside of the MBP bubble Reproduce production problems out of band Test OS/Software upgrades easily Tune production deployment without production 14 14 -
Goals continued... Recover from server failures quickly Add more servers
when traffic increases Make setup repeatable Manage servers in bulk 15 15 -
Step 3: Deploy Step 2: Configure Step 1: Provision 16
Three Steps To A New Server 16 -
Where Do We Put The Server? 19 Amazon AWS /
EC2 Rackspace RackCloud SoftLayer, SliceHost, Linode, TerraHost, OpenStack, Eucalyptus Local VirtualBox Install Existing Servers Managed Host / Service Provider 19 -
What Needs Provisioned? 20 Operating System + root access Install
just enough to run configure / deploy steps later Install Ruby / Ohai / Chef / RubyShadow / Bundler Remove Future Roadblocks in Configuration / Deployment SSH Config: Disable Require TTY, Env Keep PATH Disable SeLinux (KickStart Bug! / Chef Recipes / Apache) Configure $PATH: environment, bashrc, profile, etc LD PATH: ldconfig (bundler deployment cache issues) 20 -
How Do We Do It? 21 VirtualBox 4.2.0 (Sandy Bridge
+ 10.8.2 Issues!) veewee - Creates “Boxes” - https://github.com/jedi4ever/veewee vagrant - Manages Boxes / Instances - http://vagrantup.com/ EC2 / RackCloud - https://github.com/opscode/ knife / knife-ec2 - Manages AWS Instances knife / knife-rackspace - Manages RackCloud Instances Managed / Existing Servers ssh / sudo / su 21 -
VirtualBox Provisioning 22 Install VirtualBox 4.2.0 Install Ruby Gems $
gem install veewee vagrant Define/Customize a new machine image $ vagrant basebox define MyServer CentOS-5.6-x86_64-netboot Edit KickStart Config - ks.cfg Edit Post Install Script - postinstall.sh 22 -
VirtualBox Provisioning cont... 27 Install VirtualBox 4.2.0 Install Ruby Gems
$ gem install veewee vagrant Define/Customize a new machine image $ vagrant basebox define MyServer CentOS-5.6-x86_64-netboot Build the machine image $ vagrant basebox build MyServer $ vagrant basebox export MyServer 27 -
EC2/RackCloud Provisioning 31 Install Ruby Gems $ gem install chef
knife-ec2 knife-rackspace Configure API Keys in ~/.chef/knife.rb knife[:aws_access_key_id] = "Your AWS Access Key ID" knife[:aws_secret_access_key] = "Your AWS Access Key" Customize the server image $ vim ~/.chef/bootstrap/centos56.rb 31 -
EC2/RC Provisioning cont... 33 Install Ruby Gems $ gem install
chef knife-ec2 knife-rackspace Configure API Keys in ~/.chef/knife.rb knife[:aws_access_key_id] = "Your AWS Access Key ID" knife[:aws_secret_access_key] = "Your AWS Access Key" Customize the server image $ vim ~/.chef/bootstrap/centos56.rb Create the machine image $ knife ec2 create -I ami-0a59bb63 -d centos56 ... 33 -
Login To Your New Server 34 VirtualBox / Vagrant $
vagrant ssh [ssh vagrant@localhost -p 2222] EC2 $ ssh [email protected] -i ec2-group-key.pem Rackspace $ ssh [email protected] Questions? 34 -
What Is Chef? Configuration management for “Nodes” or servers It
is a “Cookbook” full of configuration “Recipes” plus “Data Bags” Install “build” user. Set password. Configure ssh key. Configure github access. Cookbooks, Recipes, Data Bags stored upstream on OpsCode server Client downloads recipes and runs them on each server Configure things differently by “Environment”: development, staging, production Configure “Roles” or groups of recipes: app, db, caching, services, etc Manage Cookbooks, Recipes, Roles and Nodes from command line 36 36 -
What Does A Data Bag Do? 37 { "id": "build",
"uid": 1000, "gid": 1000, "comment": "Build User", "shell": "/bin/bash", "password": "$1$31Pf4SgRy$edFhgUyhUBDE3%eUSD4rmk1", "ssh_keys": "ssh-rsa AAAABC2TbS43DAAABD4ER3DH4WT....default", "sudoers": "ALL=(ALL) ALL" } 37 -
What Does A Recipe Do? 38 home_dir = "/home/#{u['id']}" group
u['id'] do gid u['gid'] end user u['id'] do uid u['uid'] gid u['gid'] shell u['shell'] password u['password'] home home_dir end template "#{home_dir}/.ssh/authorized_keys" do source "authorized_keys.erb" owner u['id'] group u['gid'] || u['id'] mode "0600" variables :ssh_keys => u['ssh_keys'] end 38 -
What Does A Role Do? 39 name "app" description "App
role for all web servers." run_list "role[base]", # <-- Creates users from databag/recipe "recipe[apache2]", "recipe[apache2::mod_ssl]", "recipe[mysql::client]", "recipe[passenger_apache2]", "recipe[passenger_apache2::mod_rails]", "recipe[sphinx]", "recipe[company::ssl]", "recipe[company::mainsite]", "recipe[company::mobilesite]" 39 -
What Does An Environment Do? 40 name "development" default_attributes "company"
=> { "mainsite" => { "virtual_host" => "localhost" } } name "staging" default_attributes "company" => { "mainsite" => { "virtual_host" => "staging.company.com" } } name "production" default_attributes "company" => { "mainsite" => { "virtual_host" => "www.company.com" } } 40 -
Manage Everything Via Terminal 41 $ knife node list app1,
services2, staging-db, vagrant-userx-mainsite, .... $ knife cookbook list apache, xml, xslt, imagemagic, company::users, mysql, .... $ knife search node "chef_environment:production AND role:services" Node Name: services1 FQDN: services1.company.com IP: 172.16.2.3 $ knife ssh “name:app*” “sudo chef-client -r ‘recipe[users]’” app1.company.com Running Chef client...... app2.company.com Running Chef client...... 41 -
Upload Everything to OpsCode 42 $ knife environment from file
./environments.rb $ knife data bad from file Users ./build.json $ knife role from file ./base.rb ./web.rb $ knife cookbook upload --all 42 -
Running Chef 43 VirtualBox / Vagrant chef-client automatically runs after
vagrant up vagrant provision to manually reconfigure the server EC2 / Rackspace chef-client automatically runs after knife create bootstrap sudo chef-client to manually reconfigure the server Managed Servers sudo chef-client to manually configure the servers Automate from afar! knife ssh “name:mynode” “sudo chef-client” 43 -
What About My Mac! 44 Install Using Homebrew! - https://github.com/mathie/chef-homebrew
Install Using Dmg! - https://github.com/opscode/cookbooks/tree/master/dmg dmg_package "Google Chrome" do dmg_name "googlechrome" source "https://dl-ssl.google.com/.../GGRM/googlechrome.dmg" checksum "7daa2dc5c46d9bfb14f1d7ff4b33884325e5e63e6..." action :install end dmg_package "Virtualbox" do source "http://dlc.sun.com.edge...VirtualBox...-OSX.dmg" type "mpkg" end 44 -
Configure Capistrano 46 Create a multistage environment configuration (multistage plugin
or inline tasks) $ gem install capistrano $ cd MyApp $ Capify . $ vim config/deploy.rb 46 -
Override Defaults Per Destination 47 role :web, "localhost" role :db,
"localhost", :primary => true task :vagrant do set :port, 2222 end task :ec2 do # same as :rackspace role :web, ENV['address'] role :db, ENV['address'], :primary => true ssh_options[:keys] = “~/.ssh/your-default-key-pair.pem” end task :production do role :web, "app1.company.com", "app2.company.com” role :db, "app1.company.com", :primary => true ssh_options[:keys] = “~/.ssh/your-default-key-pair.pem” end 47 -
Deploy Your Application 48 $ cap <environment> <action> branch=value address=value
rails_env=environment $ cap ec2 deploy:initial address=xxx.xxx.xxx.xxx branch=mybranch $ cap rackspace deploy:initial address=xxx.xxx.xxx.xxx branch=mybranch $ cap vagrant deploy:initial [address=localhost] branch=mybranch $ cap production deploy:update [branch=master] $ cap deploy:update branch=mybranch (uses defaults) 48 -
Capistrano Deploy Tasks 49 deploy:initial deploy:setup, deploy:update, db:setup, sphinx:reindex, starling:restart,
workling:restart, deploy:restart Run on fresh instances: vagrant, ec2, rackspace. db:setup is disabled for production environment. deploy:web:enable / deploy:web:disable Also does cluster:put / cluster:pull deploy:tests:environment, deploy:test:connections. Anything! 49 -
Setup Project Contents 50 .chef - Preconfigured to talk to
OpsCode, EC2, Rackspace! chef - Company Cookbooks, Recipes, Roles, Environments, DataBags vagrant/definitions - Preconfigured CentOS 5.6 x64 Machine! vagrant/instances/mainsite - Preconfigured Vagrant Site Instance! cd vagrant/instances/mainsite; vagrant up; cap vagrant deploy:initial 50 -
Gitify Your Work. Clone And Go 51 $ git clone
[email protected]:Company/company_setup.git $ cd company_setup $ bundle install $ rake ec2/rackspace instance:create $ cd vagrant/instances/mainsite && vagrant up $ cd company_mainsite $ cap ec2/rackspace deploy:initial address=xxx.xxx.xxx.xxx branch=gerbilsauce 51 -
Goals Revisited New Guy/Gal gets instance on Day #1 Any
Engineer/QA can spin up machines Same configuration everywhere Add more servers when traffic increases Test OS/Software upgrades easily Tune production deployment without production Reproduce production problems out of band Recover from server failures quickly 53 53 -
Goals Revisited continued... Test outside of the MBP bubble Make
setup repeatable Manage servers in bulk 54 54 -
This Is “Easy”? 55 Sorry. I lied. Servers are difficult.
This is an infrastructure investment. Make knowledge transfer possible. Make the difficult possible so you can work on something else. 55