Why you should code your servers

The benefits of developing cookbooks for your app and for it's dependencies. Presenting chef, vagrant, berkshelf, opsworks, logstash and kibana

Transcript

1. Why you should code your servers @pedroaxl
2. Pedro Axelrud Entrepreneur and DevOps at mailee.me keeping your startup running devopsreactions.tumblr.com
3. let's talk about DevOps development + operations
4. development env 127.0.0.1
5. non standardized environment devopsreactions.tumblr.com
6. bundler $ gem install bundler $ cat Gemfile ! source ‘https://rubygems.org' gem 'nokogiri' gem 'rack', '~>1.1' gem 'rspec', :require => ‘spec' $ bundle install
7. berkshelf $ gem install berkshelf $ cat Berksfile ! site :opscode ! cookbook ‘postgresql’ cookbook ‘redisio’ cookbook ‘nginx’, ‘~> 0.101.5’ $ berks install
8. chef
9. cookbook structure $ ls . ! attributes default.rb files metadata.rb recipes default.rb templates
10. metadata.rb name 'maileed' maintainer 'Pedro Axelrud' maintainer_email 'pedroaxl@gmail.com' license 'All rights reserved' description 'Installs/Configures maileed' long_description IO.read(File.join(File.dirname(__FILE__), 'README.md')) version '0.1.0' ! depends 'git' depends 'python', '~> 1.4.6' depends 'redisio', '~> 1.7.1' depends 'dnsmasq', '~> 0.2.0'
11. recipe case node['platform_family'] when 'debian' include_recipe "apt" repo = 'http://ppa.launchpad.net/chris-lea/node.js/ubuntu' packages = %w{ nodejs } apt_repository 'node.js' do uri repo distribution node['lsb']['codename'] components ['main'] keyserver "hkp://keyserver.ubuntu.com:80" key "C7917B12" action :add end when 'rhel' include_recipe 'yum-epel' packages = %w{ nodejs nodejs-devel npm } when 'fedora' packages = %w{ nodejs nodejs-devel npm } when 'smartos' packages = %w{ nodejs } end ! packages.each do |node_pkg| package node_pkg end
12. recipe package 'postfix' ! file ::File.join(node['postfix']['base_dir'], 'main.cf') do content main_cf.erb user 'root' group 0 mode 0644 end ! directory ::File.join(node['postfix']['base_dir'], 'tables') do owner "root" group "root" mode 00755 action :create end ! service 'postfix' do action :start end
13. attributes default['postgresql']['config']['data_directory'] = "/var/lib/postgresql/ #{node['postgresql']['version']}/main" default['postgresql']['config']['hba_file'] = "/etc/postgresql/ #{node['postgresql']['version']}/main/pg_hba.conf" default['postgresql']['config']['ident_file'] = "/etc/postgresql/ #{node['postgresql']['version']}/main/pg_ident.conf" default['postgresql']['config']['external_pid_file'] = "/var/run/ postgresql/#{node['postgresql']['version']}-main.pid" default['postgresql']['config']['listen_addresses'] = 'localhost' default['postgresql']['config']['port'] = 5432 default['postgresql']['config']['max_connections'] = 100 default['postgresql']['config']['unix_socket_directory'] = '/var/run/ postgresql' if node['postgresql']['version'].to_f < 9.3 default['postgresql']['config']['unix_socket_directories'] = '/var/run/ postgresql' if node['postgresql']['version'].to_f >= 9.3 default['postgresql']['config']['shared_buffers'] = '24MB' default['postgresql']['config']['max_fsm_pages'] = 153600 if node['postgresql']['version'].to_f < 8.4 default['postgresql']['config']['log_line_prefix'] = '%t ' default['postgresql']['config']['datestyle'] = 'iso, mdy' default['postgresql']['config']['default_text_search_config'] = 'pg_catalog.english' default['postgresql']['config']['ssl'] = true
14. vagrant $ vagrant init ! A `Vagrantfile` has been placed in this directory. You are now ready to `vagrant up` your first virtual environment! Please read the comments in the Vagrantfile as well as documentation on `vagrantup.com` for more information on using Vagrant. !
15. vagrant $ vagrant up Bringing machine 'default' up with 'virtualbox' provider... [default] Clearing any previously set forwarded ports... [Berkshelf] Skipping Berkshelf with --no-provision [default] Clearing any previously set network interfaces... [default] Preparing network interfaces based on configuration... [default] Forwarding ports... [default] -- 22 => 2222 (adapter 1) [default] Booting VM... [default] Waiting for machine to boot. This may take a few minutes... [default] Machine booted and ready! [default] Setting hostname... [default] Configuring and enabling network interfaces... [default] Mounting shared folders... [default] -- /vagrant [default] -- /tmp/vagrant-chef-1/chef-solo-1/cookbooks
16. vagrant Vagrant.configure("2") do |config| config.vm.hostname = "mailee-berkshelf" config.vm.box = "ubuntu1310" config.vm.box_url = "https://dl.dropboxusercontent.com/s/ mqcaq2575qrq5a8/ubuntu1310.box? dl=1&token_hash=AAGGlsF0e6iBuq7YDMcs3jNbNfBdkElTqJv6qo340ihxQA" config.vm.network :private_network, ip: "33.33.33.10" config.berkshelf.enabled = true config.vm.provision :chef_solo do |chef| chef.json = { "postgresql" => { "password" => { "postgres" => "102030" } } chef.run_list = [ "recipe[postgresql::server]", "recipe[redisio::install]", "recipe[rbenv]", "recipe[mailee-app::default]" ] chef.cookbooks_path = "../mailee-cookbooks" end end
17. production env 0.0.0.0
18. amazon opsworks because heroku is too expensive
24. senior vs junior sysadmin during an outage devopsreactions.tumblr.com
25. setting up new host with puppet (or chef) devopsreactions.tumblr.com
26. deploying
27. watching the new team deploy to prod devopsreactions.tumblr.com
28. deploy require 'yaml' ! include_recipe "git" ! template '/etc/bluepill/maileed.pill' do source 'maileed.pill.erb' end ! ! git node['maileed']['deploy_dir'] do repository node['maileed']['repo'] revision "master" action :sync end ! config = { 'db' => node['maileed']['database'].to_hash, 'options' => node['maileed']['options'].to_hash, 'redis' => node['maileed']['redis'].to_hash } ! file "#{node['maileed']['deploy_dir']}/config.yml" do mode "0644" action :create content YAML::dump(config) end ! bluepill_service 'maileed' do action [:stop, :start] end
29. one more thing… what about log files?
31. when one uses 'ps | grep' instead of pgrep in scripts devopsreactions.tumblr.com
32. splunk
33. elasticsearch + logstash
34. deploying on the brand new powerful hardware devopsreactions.tumblr.com pedroaxl@gmail.com | pedroaxl.github.io