Upgrade to Pro — share decks privately, control downloads, hide ads and more …

LinuxConf Japan 2014: Bringing Application Owners and Sysadmins together via Configuration Management Workflows

LinuxConf Japan 2014: Bringing Application Owners and Sysadmins together via Configuration Management Workflows

Too often application owners and systems administrators end up in an adversarial relationship due to mismatching workflows and expectations. In this talk Nigel Kersten will cover best practices for configuration management workflows that enable application owners and systems administrators to work together to ship reliable applications on top of reliable infrastructure. The principles covered in this talk will be applicable to any modern configuration management system, however concrete examples and demos will be done using Puppet Enterprise.

De45f73e374e7f3d93c661f5b8022776?s=128

Nigel Kersten

May 20, 2014
Tweet

Transcript

  1. Bringing App Owners and Sysadmins together via config management workflows

    Nigel Kersten, CIO/Operations, Puppet Labs.
  2. WHAT’S THE PROBLEM? Sysadmins and App Owners don’t necessarily want

    the same thing
  3. THE SYSADMIN VIEW OF APP OWNERS A bunch of cowboys

  4. THE APP OWNER VIEW OF SYSADMINS Bureaucratic inhibitors of change

  5. INTERESTS AREN’T (ALL) ALIGNED Sysadmins • Maintain stability • Wear

    the pager • Want a functional deployment App Owners • Create change • Sleep at night • Want a functional deployment
  6. DEVELOPERS RUNNING ALL OPERATIONS IS NOT THE WAY OPERATIONS DOING

    ALL DEVELOPMENT IS NOT THE WAY
  7. CREATING A THIRD “DEVOPS” TEAM IS DEFINITELY NOT THE WAY

  8. HOW DO WE FIX THIS?

  9. GENERAL PRINCIPLES • Single source of truth for everything •

    Self-Service • A common configuration language • Model your environments • Abstracted and simplified interfaces • Be generous with access
  10. SINGLE SOURCE OF TRUTH • Application code • Configuration •

    Documentation • Issue tracking • Deployed application state
  11. SELF-SERVICE IS KEY • Empower all teams involved in application

    delivery • Application Owners • Systems Administrators • Others: Quality Assurance, Release Engineering • Enable deployment of each other’s responsibilities
  12. SELF-SERVICE WHICH PARTS? • Application deployment • Packaging from source

    control • OS/Middleware deployment • Test suite execution
  13. CHOOSE A CONFIGURATION LANGUAGE • Unsure which to choose?

  14. CHOOSE PUPPET

  15. PUPPET OVERVIEW class ssh { package { 'openssh-server': ensure =>

    present, } file { 'sshd_config': path => '/etc/ssh/sshd_config', source => ‘puppet:///modules/ssh/sshd_config', require => Package['openssh-server'], notify => Service['sshd'], } service { 'sshd': ensure => running, enable => true, require => File['sshd_config'], } }
  16. MODEL YOUR ENVIRONMENTS • Production often has different attributes to

    development and staging • NTP, DNS, IPs • Not an excuse to avoid consistency
  17. HIERA - DATA/MODEL SEPARATION • Keeps configuration data separate from

    the configuration model (Puppet code) • Enables reusability and model sharing • Puppet automatically looks for Hiera data for class parameters
  18. HIERA - CONFIGURATION #/etc/puppetlabs/puppet/hiera.yaml --- :backends: - yaml :hierarchy: -

    "%{clientcert}" - "%{environment}" - global
  19. HIERA - PER-ENVIRONMENT DATA #/etc/puppetlabs/puppet/hiera/dev.yaml --- ntp::servers: - 1.2.3.4 -

    1.2.3.5 #/etc/puppetlabs/puppet/hiera/prod.yaml --- ntp::servers: - 4.3.2.1 - 4.3.2.2
  20. ABSTRACTED AND SIMPLIFIED INTERFACES class ntp ( … $preferred_servers =

    $ntp::params::preferred_servers, $restrict = $ntp::params::restrict, $servers = $ntp::params::servers, $service_enable = $ntp::params::service_enable, $service_ensure = $ntp::params::service_ensure, $service_manage = $ntp::params::service_manage, $service_name = $ntp::params::service_name, $udlc = $ntp::params::udlc ) inherits ntp::params { … }
  21. SIMPLIFIED INTERFACE - GUI

  22. AUTOMATE THE LIFECYCLE • The path from pre-production to production

    should: • be obvious, accessible and automated • keep app code and configuration aligned
  23. PUPPET ENVIRONMENTS FROM GIT BRANCHES • r10k - https://github.com/adrienthebo/r10k •

    auto-creates Puppet Environments from git branches # /etc/r10k.yaml # # The location to use for storing cached Git repos :cachedir: '/var/cache/r10k' # A list of git repositories to create :sources: # This will clone the git repository and instantiate an environment # per branch in /etc/puppet/environments :plops: remote: 'git@github.com:my-org/org-shared-modules' basedir: '/etc/puppet/environments'
  24. ALIGN PACKAGES AND CONFIGURATION • Map package repositories to Puppet

    environments apt::source { 'corp_repo': location => 'http://corp.repo/', release => "${environment}", repos => 'main', }
  25. SELF-SERVICE PACKAGING • Packages are awesome • Un-versioned Tarballs are

    not • Packaging is often unnecessarily complicated • fpm to the rescue.
  26. FPM • https://github.com/jordansissel/fpm • Simplified package building • Easy to

    script and automate
  27. FPM EXAMPLE - FROM THE DOCS # Normal build steps.

    % wget http://nodejs.org/dist/v0.6.0/node-v0.6.0.tar.gz % tar -zxf node-v0.6.0.tar.gz % cd node-v0.6.0 % ./configure --prefix=/usr % make # Install to a separate directory for capture. % mkdir /tmp/installdir % make install DESTDIR=/tmp/installdir # Create a nodejs deb with only bin and lib directories: # The 'VERSION' and 'ARCH' strings are automatically filled in # for you based on the other arguments given. % fpm -s dir -t deb -n nodejs -v 0.6.0 -C /tmp/installdir \ -p nodejs-VERSION_ARCH.deb \ -d "libssl0.9.8 (> 0)" \ -d "libstdc++6 (>= 4.4.3)" \ usr/bin usr/lib
  28. TEST YOUR PUPPET CODE • Beaker - cloud enabled acceptance

    testing tool • https://github.com/puppetlabs/beaker/ • Uses RSpec
  29. BEAKER - EXAMPLE describe 'mysql::server::account_security class' do describe 'running puppet

    code' do it 'should work with no errors' do pp = <<-EOS class { 'mysql::server': remove_default_accounts => true } EOS # Run it twice and test for idempotency apply_manifest(pp, :catch_failures => true) expect(apply_manifest(pp, :catch_failures => true).exit_code).to be_zero end end end
  30. YOU CAN KEEP PRODUCTION RESPONSIBILITIES • Developers can still own

    creating a release • QA can still own certifying a release • Operations can still own operational stability and monitoring • Responsbilities != silos
  31. THANK YOU Questions?