Upgrade to Pro — share decks privately, control downloads, hide ads and more …

PuppetConf 2016: Scaling Puppet and Puppet Cult...

PuppetConf 2016: Scaling Puppet and Puppet Culture at GitHub

Watch the talk: https://www.youtube.com/watch?v=H7cQcoXVacU

Slides from PuppetConf (presented from main stage on October 20, 2016). This talk describes the cultural and technical considerations that enable 2/3 of all engineers at GitHub contribute to the Puppet code base. Topics include ownership, workflow, collaboration, technical debt, managing chaos, and the initial open-source release of the "octocatalog-diff" tool.

Kevin Paulisse

October 20, 2016
Tweet

More Decks by Kevin Paulisse

Other Decks in Technology

Transcript

  1. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    1 Scaling Puppet and Puppet Culture Kevin Paulisse SRE @ GitHub
  2. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    About Kevin Paulisse 2 " ! Text @kpaulisse kpaulisse [email protected] x # ↑ Work: GitHub ↓ Live: Madison, Wisc.
  3. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Agenda 3 ! Culture • Ownership • Collaboration • Chaos Tools • Workflow • Technical debt • Productivity
  4. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Puppet at GitHub 5 ! • First Commit: September 8, 2008 (GitHub launched on April 10, 2008) • Puppet Versions: 0.24.x - 4.5.x • Lines of Code: 500,000+ • Puppet Resources: 3,000+ per node
  5. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Puppet at GitHub 7 ! of GitHub engineers have contributed to github/puppet of all GitHubbers have committed to github/puppet 35% 67% pull requests are merged each day in github/puppet 15+
  6. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Who owns your Puppet? 10 ! OPS SILO Ops team owns the Puppet code. Others submit tickets.
  7. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Who owns your Puppet? 11 ! OPS SILO Ops team owns the Puppet code. Others submit tickets. WILD WEST Everyone owns the Puppet code. The two extremes
  8. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Who owns your Puppet? 12 ! WILD WEST Everyone owns the Puppet code. SELF SERVE One group owns the Puppet code but others can modify Hiera data. OPS SILO Ops team owns the Puppet code. Others submit tickets.
  9. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Who owns your Puppet? 13 ! WILD WEST Everyone owns the Puppet code. SELF SERVE One group owns the Puppet code but others can modify Hiera data. INNER SOURCE One group maintains the project but others can contribute. OPS SILO Ops team owns the Puppet code. Others submit tickets.
  10. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    WILD WEST Everyone owns the Puppet code. Who owns your Puppet? 14 ! SELF SERVE One group owns the Puppet code but others can modify Hiera data. INNER SOURCE One group maintains the project but others can contribute. OPS SILO Ops team owns the Puppet code. Others submit tickets. WE WERE HERE
  11. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Who owns your Puppet? 15 ! SELF SERVE One group owns the Puppet code but others can modify Hiera data. INNER SOURCE One group maintains the project but others can contribute. OPS SILO Ops team owns the Puppet code. Others submit tickets. WE WERE HERE WE ARE HERE WILD WEST Everyone owns the Puppet code.
  12. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Effective workflow checklist 17 ! A Puppet contributor needs to... Check out the Puppet code Bootstrap a Puppet development environment Run unit tests / CI Run Puppet in no-op mode to see changes Deploy Puppet changes to a host Deploy Puppet changes everywhere (???)
  13. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    GitHub's Puppet development workflow 18 ! $ Branch % Pull Request & Review ' Test (CI) ( Branch Deploy ) Ship * Clone (Just like nearly every other project GitHub develops)
  14. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Clone and bootstrap 19 ! ./script/bootstrap • Ruby gems (bundler) • librarian-puppet • Pre-commit hooks $ Branch % Pull Request & Review ' Test (CI) ( Branch Deploy ) Ship * Clone
  15. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Pull request 20 ! $ Branch % Pull Request & Review ' Test (CI) ( Branch Deploy ) Ship * Clone
  16. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Test (CI) 21 ! • puppet spec tests with rspec-puppet • puppet-catalogs for ~50 key roles • puppet-lint to enforce style • puppet-real-host-compile to test facter • puppet-utility to verify project setup $ Branch % Pull Request & Review ' Test (CI) ( Branch Deploy ) Ship * Clone
  17. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Review 22 ! $ Branch % Pull Request & Review ' Test (CI) ( Branch Deploy ) Ship * Clone
  18. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Review 23 ! Types of review: • Team review • Puppet review $ Branch % Pull Request & Review ' Test (CI) ( Branch Deploy ) Ship * Clone
  19. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Branch deploy 24 ! $ Branch % Pull Request & Review ' Test (CI) ( Branch Deploy ) Ship * Clone
  20. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Ship it! 25 ! $ Branch % Pull Request & Review ' Test (CI) ( Branch Deploy ) Ship * Clone
  21. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Welcome? 27 ! This repository runs the production infrastructure of github.com and many other services. Any changes that break production infrastructure will page the Hubber on call, interrupt their flow, and possibly wake them up. -- CONTRIBUTING.md in github/puppet, October 2015
  22. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Welcome!! 28 ! This repository runs the production
 infrastructure of github.com and many other
 - services. Any changes that break production
 - infrastructure will page the Hubber on call,
 - interrupt their flow, and possibly wake them up. + services. We encourage (and expect) Puppet
 + contributions from everyone within GitHub
 + engineering. Changes made here can break
 + production infrastructure, so you need to be
 + careful. If you need help or a second pair of
 + eyes, please feel free to visit us in the
 + `#puppet` channel...
  23. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Anti-pattern of engagement 31 ! "Good programmers know what to write. Great ones know what to rewrite (and reuse)." -- Eric S. Raymond, The Cathedral and the Bazaar Brand new programmers will copy the first thing that looks close. But also consider...
  24. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Supporting your contributors - education 35 ! Learn Puppet... • Book • Tutorial • Formal training • Hack time + mentor
  25. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Supporting your contributors - education 36 ! Our Puppet repo... • Where is it? • How to test? • How to deploy? • Where to get help? Learn Puppet... • Book • Tutorial • Formal training • Hack time + mentor
  26. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Supporting your contributors - documentation 37 !
  27. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Technical debt 39 ! Technical debt is the enemy of scalability
  28. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Watch the technical debt grow 43 ! "If this host is not in physical data center 1, it must be in AWS!"
  29. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Modularize for future growth 44 ! Caution: Over-simplified pseudo-code!
  30. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    "Bad clever" example 46 ! ipaddress = '172.17.0.1' WITH DOCKER RUNNING m = 25 fqdn_rand(60) A BETTER WAY
  31. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Is your solution "good clever"? 47 ! Evaluate your clever solution: • Will you remember how it works? • Will others understand how it works? • What advantages does this have over the
 generally accepted path? • If you were on stage at PuppetConf,
 would the audience or ?
  32. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Paying down technical debt 48 ! • Do you set aside time to work on... • Refactoring • Deprecating • Deleting • Enforcing code quality
  33. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Paying down technical debt 49 ! • Do you set aside time to work on... • Refactoring • Deprecating • Deleting • Enforcing code quality • Do you praise the engineers who do? • Raises and promotions • Performance reviews
  34. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Monitoring, metrics, and alerting 51 ! Monitors & Metrics
  35. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Monitoring, metrics, and alerting 52 ! Monitors: • Catalog fails to generate • Failing resources on host • Host hasn't reported in Metrics: • Catalog compile time • Catalog run time • CI build time
  36. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Sometimes things go horribly wrong 53 ! .puppet panic
  37. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Development cycle before catalog analysis 56 !
  38. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Agent testing 57 ! MASTER AGENT OS VERSION CPUS MEMORY DATACENTER MAC ADDRESS DISKS FQDN HW MODEL EC2? IP ADDR FACTS 1. Agent sends
 facts to master 2. Master returns
 catalog to agent 3. Agent applies catalog to system code hiera puppetdb ENC logs
  39. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    My first catalog difference analysis 58 ! for host in $hosts ; do git checkout master puppet master --compile $host > /tmp/old.json git checkout my-changed-branch puppet master --compile $host > /tmp/new.json diff /tmp/old.json /tmp/new.json > /tmp/$host.diff done Caution: Over-simplified pseudo-code!
  40. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Catalog testing 59 ! MASTER AGENT 1. Agent sends
 facts to master 2. Master returns
 catalog to agent 3. Agent applies catalog to system code hiera puppetdb ENC logs OS VERSION CPUS MEMORY DATACENTER MAC ADDRESS DISKS FQDN HW MODEL EC2? IP ADDR FACTS
  41. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Developing a catalog difference tool 60 !
  42. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Catalog differences with octocatalog-diff 61 ! Catalog compilation: • Check out "from" and "to" branches • Munge hiera config, ENC, etc. • Facts from PuppetDB • Build catalogs using Puppet Comparison analysis: • Resources added, removed, changed • Display human-readable output
  43. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Detecting collateral damage 62 ! class mysql::params { $mysql_port = 3306 $mysql_version = '5.6.20-1' }
  44. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Detecting collateral damage 63 ! class mysql::params { $mysql_port = 3306 $mysql_version = '5.6.20-1' } class profile::frontend { package { 'mysql-client': ensure => $mysql::params::mysql_version, } }
  45. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Detecting collateral damage 64 ! class mysql::params { $mysql_port = 3306 $mysql_version = '5.5.20-1' } class profile::frontend { package { 'mysql-client': ensure => $mysql::params::mysql_version, } } 5.6.24-7 5.6.24-7
  46. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Running octocatalog-diff across a fleet 65 !
  47. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Running octocatalog-diff across a fleet 66 !
  48. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Productivity improvements with octocatalog-diff 67 ! With octocatalog-diff
  49. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Productivity improvements with octocatalog-diff 68 ! With octocatalog-diff
  50. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    We have open-sourced octocatalog-diff 69 ! https://github.com/github/octocatalog-diff
  51. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    70 + , ! CULTURAL TECHNICAL • Ownership • Participation • Manage chaos • Workflow • Eliminating technical debt • https://github.com/github/octocatalog-diff x Session summary
  52. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Scaling Puppet and Puppet Culture 71 ! @kpaulisse kpaulisse x [email protected] # kpaulisse (puppetcommunity.slack.com) Kevin Paulisse (SRE @ GitHub) https://github.com/github/octocatalog-diff https://linkedin.com/in/kpaulisse