LinuxConf Japan 2014: Bringing Application Owners and Sysadmins together via Configuration Management Workflows

Too often application owners and systems administrators end up in an adversarial relationship due to mismatching workflows and expectations. In this talk Nigel Kersten will cover best practices for configuration management workflows that enable application owners and systems administrators to work together to ship reliable applications on top of reliable infrastructure. The principles covered in this talk will be applicable to any modern configuration management system, however concrete examples and demos will be done using Puppet Enterprise.

Transcript

1. Bringing App Owners and Sysadmins together via config management workflows Nigel Kersten, CIO/Operations, Puppet Labs.
2. WHAT’S THE PROBLEM? Sysadmins and App Owners don’t necessarily want the same thing
3. THE SYSADMIN VIEW OF APP OWNERS A bunch of cowboys
4. THE APP OWNER VIEW OF SYSADMINS Bureaucratic inhibitors of change
5. INTERESTS AREN’T (ALL) ALIGNED Sysadmins • Maintain stability • Wear the pager • Want a functional deployment App Owners • Create change • Sleep at night • Want a functional deployment
6. DEVELOPERS RUNNING ALL OPERATIONS IS NOT THE WAY OPERATIONS DOING ALL DEVELOPMENT IS NOT THE WAY
7. CREATING A THIRD “DEVOPS” TEAM IS DEFINITELY NOT THE WAY
8. HOW DO WE FIX THIS?
9. GENERAL PRINCIPLES • Single source of truth for everything • Self-Service • A common configuration language • Model your environments • Abstracted and simplified interfaces • Be generous with access
10. SINGLE SOURCE OF TRUTH • Application code • Configuration • Documentation • Issue tracking • Deployed application state
11. SELF-SERVICE IS KEY • Empower all teams involved in application delivery • Application Owners • Systems Administrators • Others: Quality Assurance, Release Engineering • Enable deployment of each other’s responsibilities
12. SELF-SERVICE WHICH PARTS? • Application deployment • Packaging from source control • OS/Middleware deployment • Test suite execution
13. CHOOSE A CONFIGURATION LANGUAGE • Unsure which to choose?
14. CHOOSE PUPPET
15. PUPPET OVERVIEW class ssh { package { 'openssh-server': ensure => present, } file { 'sshd_config': path => '/etc/ssh/sshd_config', source => ‘puppet:///modules/ssh/sshd_config', require => Package['openssh-server'], notify => Service['sshd'], } service { 'sshd': ensure => running, enable => true, require => File['sshd_config'], } }
16. MODEL YOUR ENVIRONMENTS • Production often has different attributes to development and staging • NTP, DNS, IPs • Not an excuse to avoid consistency
17. HIERA - DATA/MODEL SEPARATION • Keeps configuration data separate from the configuration model (Puppet code) • Enables reusability and model sharing • Puppet automatically looks for Hiera data for class parameters
18. HIERA - CONFIGURATION #/etc/puppetlabs/puppet/hiera.yaml --- :backends: - yaml :hierarchy: - "%{clientcert}" - "%{environment}" - global
19. HIERA - PER-ENVIRONMENT DATA #/etc/puppetlabs/puppet/hiera/dev.yaml --- ntp::servers: - 1.2.3.4 - 1.2.3.5 #/etc/puppetlabs/puppet/hiera/prod.yaml --- ntp::servers: - 4.3.2.1 - 4.3.2.2
20. ABSTRACTED AND SIMPLIFIED INTERFACES class ntp ( … $preferred_servers = $ntp::params::preferred_servers, $restrict = $ntp::params::restrict, $servers = $ntp::params::servers, $service_enable = $ntp::params::service_enable, $service_ensure = $ntp::params::service_ensure, $service_manage = $ntp::params::service_manage, $service_name = $ntp::params::service_name, $udlc = $ntp::params::udlc ) inherits ntp::params { … }
21. SIMPLIFIED INTERFACE - GUI
22. AUTOMATE THE LIFECYCLE • The path from pre-production to production should: • be obvious, accessible and automated • keep app code and configuration aligned
23. PUPPET ENVIRONMENTS FROM GIT BRANCHES • r10k - https://github.com/adrienthebo/r10k • auto-creates Puppet Environments from git branches # /etc/r10k.yaml # # The location to use for storing cached Git repos :cachedir: '/var/cache/r10k' # A list of git repositories to create :sources: # This will clone the git repository and instantiate an environment # per branch in /etc/puppet/environments :plops: remote: 'git@github.com:my-org/org-shared-modules' basedir: '/etc/puppet/environments'
24. ALIGN PACKAGES AND CONFIGURATION • Map package repositories to Puppet environments apt::source { 'corp_repo': location => 'http://corp.repo/', release => "${environment}", repos => 'main', }
25. SELF-SERVICE PACKAGING • Packages are awesome • Un-versioned Tarballs are not • Packaging is often unnecessarily complicated • fpm to the rescue.
26. FPM • https://github.com/jordansissel/fpm • Simplified package building • Easy to script and automate
27. FPM EXAMPLE - FROM THE DOCS # Normal build steps. % wget http://nodejs.org/dist/v0.6.0/node-v0.6.0.tar.gz % tar -zxf node-v0.6.0.tar.gz % cd node-v0.6.0 % ./configure --prefix=/usr % make # Install to a separate directory for capture. % mkdir /tmp/installdir % make install DESTDIR=/tmp/installdir # Create a nodejs deb with only bin and lib directories: # The 'VERSION' and 'ARCH' strings are automatically filled in # for you based on the other arguments given. % fpm -s dir -t deb -n nodejs -v 0.6.0 -C /tmp/installdir \ -p nodejs-VERSION_ARCH.deb \ -d "libssl0.9.8 (> 0)" \ -d "libstdc++6 (>= 4.4.3)" \ usr/bin usr/lib
28. TEST YOUR PUPPET CODE • Beaker - cloud enabled acceptance testing tool • https://github.com/puppetlabs/beaker/ • Uses RSpec
29. BEAKER - EXAMPLE describe 'mysql::server::account_security class' do describe 'running puppet code' do it 'should work with no errors' do pp = <<-EOS class { 'mysql::server': remove_default_accounts => true } EOS # Run it twice and test for idempotency apply_manifest(pp, :catch_failures => true) expect(apply_manifest(pp, :catch_failures => true).exit_code).to be_zero end end end
30. YOU CAN KEEP PRODUCTION RESPONSIBILITIES • Developers can still own creating a release • QA can still own certifying a release • Operations can still own operational stability and monitoring • Responsbilities != silos
31. THANK YOU Questions?