Upgrade to Pro — share decks privately, control downloads, hide ads and more …

PuppetConf 2016: Scaling Puppet and Puppet Culture at GitHub

PuppetConf 2016: Scaling Puppet and Puppet Culture at GitHub

Watch the talk: https://www.youtube.com/watch?v=H7cQcoXVacU

Slides from PuppetConf (presented from main stage on October 20, 2016). This talk describes the cultural and technical considerations that enable 2/3 of all engineers at GitHub contribute to the Puppet code base. Topics include ownership, workflow, collaboration, technical debt, managing chaos, and the initial open-source release of the "octocatalog-diff" tool.

0a8b38ac18506421bcd88074c9a1a386?s=128

Kevin Paulisse

October 20, 2016
Tweet

Transcript

  1. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    1 Scaling Puppet and Puppet Culture Kevin Paulisse SRE @ GitHub
  2. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    About Kevin Paulisse 2 " ! Text @kpaulisse kpaulisse kpaulisse@github.com x # ↑ Work: GitHub ↓ Live: Madison, Wisc.
  3. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Agenda 3 ! Culture • Ownership • Collaboration • Chaos Tools • Workflow • Technical debt • Productivity
  4. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Puppet at GitHub 4 !
  5. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Puppet at GitHub 5 ! • First Commit: September 8, 2008 (GitHub launched on April 10, 2008) • Puppet Versions: 0.24.x - 4.5.x • Lines of Code: 500,000+ • Puppet Resources: 3,000+ per node
  6. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Daily deployments of GitHub.com 6 !
  7. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Puppet at GitHub 7 ! of GitHub engineers have contributed to github/puppet of all GitHubbers have committed to github/puppet 35% 67% pull requests are merged each day in github/puppet 15+
  8. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Puppet at GitHub 8 !
  9. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Ownership 9 !
  10. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Who owns your Puppet? 10 ! OPS SILO Ops team owns the Puppet code. Others submit tickets.
  11. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Who owns your Puppet? 11 ! OPS SILO Ops team owns the Puppet code. Others submit tickets. WILD WEST Everyone owns the Puppet code. The two extremes
  12. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Who owns your Puppet? 12 ! WILD WEST Everyone owns the Puppet code. SELF SERVE One group owns the Puppet code but others can modify Hiera data. OPS SILO Ops team owns the Puppet code. Others submit tickets.
  13. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Who owns your Puppet? 13 ! WILD WEST Everyone owns the Puppet code. SELF SERVE One group owns the Puppet code but others can modify Hiera data. INNER SOURCE One group maintains the project but others can contribute. OPS SILO Ops team owns the Puppet code. Others submit tickets.
  14. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    WILD WEST Everyone owns the Puppet code. Who owns your Puppet? 14 ! SELF SERVE One group owns the Puppet code but others can modify Hiera data. INNER SOURCE One group maintains the project but others can contribute. OPS SILO Ops team owns the Puppet code. Others submit tickets. WE WERE HERE
  15. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Who owns your Puppet? 15 ! SELF SERVE One group owns the Puppet code but others can modify Hiera data. INNER SOURCE One group maintains the project but others can contribute. OPS SILO Ops team owns the Puppet code. Others submit tickets. WE WERE HERE WE ARE HERE WILD WEST Everyone owns the Puppet code.
  16. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Workflow 16 !
  17. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Effective workflow checklist 17 ! A Puppet contributor needs to... Check out the Puppet code Bootstrap a Puppet development environment Run unit tests / CI Run Puppet in no-op mode to see changes Deploy Puppet changes to a host Deploy Puppet changes everywhere (???)
  18. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    GitHub's Puppet development workflow 18 ! $ Branch % Pull Request & Review ' Test (CI) ( Branch Deploy ) Ship * Clone (Just like nearly every other project GitHub develops)
  19. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Clone and bootstrap 19 ! ./script/bootstrap • Ruby gems (bundler) • librarian-puppet • Pre-commit hooks $ Branch % Pull Request & Review ' Test (CI) ( Branch Deploy ) Ship * Clone
  20. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Pull request 20 ! $ Branch % Pull Request & Review ' Test (CI) ( Branch Deploy ) Ship * Clone
  21. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Test (CI) 21 ! • puppet spec tests with rspec-puppet • puppet-catalogs for ~50 key roles • puppet-lint to enforce style • puppet-real-host-compile to test facter • puppet-utility to verify project setup $ Branch % Pull Request & Review ' Test (CI) ( Branch Deploy ) Ship * Clone
  22. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Review 22 ! $ Branch % Pull Request & Review ' Test (CI) ( Branch Deploy ) Ship * Clone
  23. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Review 23 ! Types of review: • Team review • Puppet review $ Branch % Pull Request & Review ' Test (CI) ( Branch Deploy ) Ship * Clone
  24. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Branch deploy 24 ! $ Branch % Pull Request & Review ' Test (CI) ( Branch Deploy ) Ship * Clone
  25. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Ship it! 25 ! $ Branch % Pull Request & Review ' Test (CI) ( Branch Deploy ) Ship * Clone
  26. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Collaboration 26 !
  27. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Welcome? 27 ! This repository runs the production infrastructure of github.com and many other services. Any changes that break production infrastructure will page the Hubber on call, interrupt their flow, and possibly wake them up. -- CONTRIBUTING.md in github/puppet, October 2015
  28. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Welcome!! 28 ! This repository runs the production
 infrastructure of github.com and many other
 - services. Any changes that break production
 - infrastructure will page the Hubber on call,
 - interrupt their flow, and possibly wake them up. + services. We encourage (and expect) Puppet
 + contributions from everyone within GitHub
 + engineering. Changes made here can break
 + production infrastructure, so you need to be
 + careful. If you need help or a second pair of
 + eyes, please feel free to visit us in the
 + `#puppet` channel...
  29. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Getting contributors on board 29 !
  30. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Engaged contributors 30 !
  31. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Anti-pattern of engagement 31 ! "Good programmers know what to write. Great ones know what to rewrite (and reuse)." -- Eric S. Raymond, The Cathedral and the Bazaar Brand new programmers will copy the first thing that looks close. But also consider...
  32. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Which code is the right code? 32 !
  33. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Perpetuating technical debt 33 !
  34. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Perpetuating technical debt 34 !
  35. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Supporting your contributors - education 35 ! Learn Puppet... • Book • Tutorial • Formal training • Hack time + mentor
  36. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Supporting your contributors - education 36 ! Our Puppet repo... • Where is it? • How to test? • How to deploy? • Where to get help? Learn Puppet... • Book • Tutorial • Formal training • Hack time + mentor
  37. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Supporting your contributors - documentation 37 !
  38. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Technical debt 38 !
  39. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Technical debt 39 ! Technical debt is the enemy of scalability
  40. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Watch the technical debt grow 40 !
  41. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Watch the technical debt grow 41 !
  42. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Watch the technical debt grow 42 !
  43. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Watch the technical debt grow 43 ! "If this host is not in physical data center 1, it must be in AWS!"
  44. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Modularize for future growth 44 ! Caution: Over-simplified pseudo-code!
  45. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    "Good clever" vs. "Bad clever" 45 !
  46. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    "Bad clever" example 46 ! ipaddress = '172.17.0.1' WITH DOCKER RUNNING m = 25 fqdn_rand(60) A BETTER WAY
  47. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Is your solution "good clever"? 47 ! Evaluate your clever solution: • Will you remember how it works? • Will others understand how it works? • What advantages does this have over the
 generally accepted path? • If you were on stage at PuppetConf,
 would the audience or ?
  48. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Paying down technical debt 48 ! • Do you set aside time to work on... • Refactoring • Deprecating • Deleting • Enforcing code quality
  49. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Paying down technical debt 49 ! • Do you set aside time to work on... • Refactoring • Deprecating • Deleting • Enforcing code quality • Do you praise the engineers who do? • Raises and promotions • Performance reviews
  50. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Managing chaos 50 !
  51. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Monitoring, metrics, and alerting 51 ! Monitors & Metrics
  52. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Monitoring, metrics, and alerting 52 ! Monitors: • Catalog fails to generate • Failing resources on host • Host hasn't reported in Metrics: • Catalog compile time • Catalog run time • CI build time
  53. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Sometimes things go horribly wrong 53 ! .puppet panic
  54. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Sometimes things go horribly wrong 54 !
  55. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Productivity 55 !
  56. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Development cycle before catalog analysis 56 !
  57. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Agent testing 57 ! MASTER AGENT OS VERSION CPUS MEMORY DATACENTER MAC ADDRESS DISKS FQDN HW MODEL EC2? IP ADDR FACTS 1. Agent sends
 facts to master 2. Master returns
 catalog to agent 3. Agent applies catalog to system code hiera puppetdb ENC logs
  58. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    My first catalog difference analysis 58 ! for host in $hosts ; do git checkout master puppet master --compile $host > /tmp/old.json git checkout my-changed-branch puppet master --compile $host > /tmp/new.json diff /tmp/old.json /tmp/new.json > /tmp/$host.diff done Caution: Over-simplified pseudo-code!
  59. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Catalog testing 59 ! MASTER AGENT 1. Agent sends
 facts to master 2. Master returns
 catalog to agent 3. Agent applies catalog to system code hiera puppetdb ENC logs OS VERSION CPUS MEMORY DATACENTER MAC ADDRESS DISKS FQDN HW MODEL EC2? IP ADDR FACTS
  60. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Developing a catalog difference tool 60 !
  61. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Catalog differences with octocatalog-diff 61 ! Catalog compilation: • Check out "from" and "to" branches • Munge hiera config, ENC, etc. • Facts from PuppetDB • Build catalogs using Puppet Comparison analysis: • Resources added, removed, changed • Display human-readable output
  62. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Detecting collateral damage 62 ! class mysql::params { $mysql_port = 3306 $mysql_version = '5.6.20-1' }
  63. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Detecting collateral damage 63 ! class mysql::params { $mysql_port = 3306 $mysql_version = '5.6.20-1' } class profile::frontend { package { 'mysql-client': ensure => $mysql::params::mysql_version, } }
  64. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Detecting collateral damage 64 ! class mysql::params { $mysql_port = 3306 $mysql_version = '5.5.20-1' } class profile::frontend { package { 'mysql-client': ensure => $mysql::params::mysql_version, } } 5.6.24-7 5.6.24-7
  65. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Running octocatalog-diff across a fleet 65 !
  66. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Running octocatalog-diff across a fleet 66 !
  67. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Productivity improvements with octocatalog-diff 67 ! With octocatalog-diff
  68. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Productivity improvements with octocatalog-diff 68 ! With octocatalog-diff
  69. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    We have open-sourced octocatalog-diff 69 ! https://github.com/github/octocatalog-diff
  70. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    70 + , ! CULTURAL TECHNICAL • Ownership • Participation • Manage chaos • Workflow • Eliminating technical debt • https://github.com/github/octocatalog-diff x Session summary
  71. @kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016

    Scaling Puppet and Puppet Culture 71 ! @kpaulisse kpaulisse x kpaulisse@github.com # kpaulisse (puppetcommunity.slack.com) Kevin Paulisse (SRE @ GitHub) https://github.com/github/octocatalog-diff https://linkedin.com/in/kpaulisse