Tweet
Tweet

Slide 1

Slide 1 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 1 Scaling Puppet and Puppet Culture Kevin Paulisse SRE @ GitHub

Slide 2

Slide 2 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 About Kevin Paulisse 2 " ! Text @kpaulisse kpaulisse kpaulisse@github.com x # ↑ Work: GitHub ↓ Live: Madison, Wisc.

Slide 3

Slide 3 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Agenda 3 ! Culture • Ownership • Collaboration • Chaos Tools • Workflow • Technical debt • Productivity

Slide 4

Slide 4 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Puppet at GitHub 4 !

Slide 5

Slide 5 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Puppet at GitHub 5 ! • First Commit: September 8, 2008 (GitHub launched on April 10, 2008) • Puppet Versions: 0.24.x - 4.5.x • Lines of Code: 500,000+ • Puppet Resources: 3,000+ per node

Slide 6

Slide 6 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Daily deployments of GitHub.com 6 !

Slide 7

Slide 7 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Puppet at GitHub 7 ! of GitHub engineers have contributed to github/puppet of all GitHubbers have committed to github/puppet 35% 67% pull requests are merged each day in github/puppet 15+

Slide 8

Slide 8 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Puppet at GitHub 8 !

Slide 9

Slide 9 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Ownership 9 !

Slide 10

Slide 10 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Who owns your Puppet? 10 ! OPS SILO Ops team owns the Puppet code. Others submit tickets.

Slide 11

Slide 11 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Who owns your Puppet? 11 ! OPS SILO Ops team owns the Puppet code. Others submit tickets. WILD WEST Everyone owns the Puppet code. The two extremes

Slide 12

Slide 12 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Who owns your Puppet? 12 ! WILD WEST Everyone owns the Puppet code. SELF SERVE One group owns the Puppet code but others can modify Hiera data. OPS SILO Ops team owns the Puppet code. Others submit tickets.

Slide 13

Slide 13 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Who owns your Puppet? 13 ! WILD WEST Everyone owns the Puppet code. SELF SERVE One group owns the Puppet code but others can modify Hiera data. INNER SOURCE One group maintains the project but others can contribute. OPS SILO Ops team owns the Puppet code. Others submit tickets.

Slide 14

Slide 14 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 WILD WEST Everyone owns the Puppet code. Who owns your Puppet? 14 ! SELF SERVE One group owns the Puppet code but others can modify Hiera data. INNER SOURCE One group maintains the project but others can contribute. OPS SILO Ops team owns the Puppet code. Others submit tickets. WE WERE HERE

Slide 15

Slide 15 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Who owns your Puppet? 15 ! SELF SERVE One group owns the Puppet code but others can modify Hiera data. INNER SOURCE One group maintains the project but others can contribute. OPS SILO Ops team owns the Puppet code. Others submit tickets. WE WERE HERE WE ARE HERE WILD WEST Everyone owns the Puppet code.

Slide 16

Slide 16 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Workflow 16 !

Slide 17

Slide 17 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Effective workflow checklist 17 ! A Puppet contributor needs to... Check out the Puppet code Bootstrap a Puppet development environment Run unit tests / CI Run Puppet in no-op mode to see changes Deploy Puppet changes to a host Deploy Puppet changes everywhere (???)

Slide 18

Slide 18 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 GitHub's Puppet development workflow 18 ! $ Branch % Pull Request & Review ' Test (CI) ( Branch Deploy ) Ship * Clone (Just like nearly every other project GitHub develops)

Slide 19

Slide 19 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Clone and bootstrap 19 ! ./script/bootstrap • Ruby gems (bundler) • librarian-puppet • Pre-commit hooks $ Branch % Pull Request & Review ' Test (CI) ( Branch Deploy ) Ship * Clone

Slide 20

Slide 20 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Pull request 20 ! $ Branch % Pull Request & Review ' Test (CI) ( Branch Deploy ) Ship * Clone

Slide 21

Slide 21 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Test (CI) 21 ! • puppet spec tests with rspec-puppet • puppet-catalogs for ~50 key roles • puppet-lint to enforce style • puppet-real-host-compile to test facter • puppet-utility to verify project setup $ Branch % Pull Request & Review ' Test (CI) ( Branch Deploy ) Ship * Clone

Slide 22

Slide 22 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Review 22 ! $ Branch % Pull Request & Review ' Test (CI) ( Branch Deploy ) Ship * Clone

Slide 23

Slide 23 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Review 23 ! Types of review: • Team review • Puppet review $ Branch % Pull Request & Review ' Test (CI) ( Branch Deploy ) Ship * Clone

Slide 24

Slide 24 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Branch deploy 24 ! $ Branch % Pull Request & Review ' Test (CI) ( Branch Deploy ) Ship * Clone

Slide 25

Slide 25 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Ship it! 25 ! $ Branch % Pull Request & Review ' Test (CI) ( Branch Deploy ) Ship * Clone

Slide 26

Slide 26 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Collaboration 26 !

Slide 27

Slide 27 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Welcome? 27 ! This repository runs the production infrastructure of github.com and many other services. Any changes that break production infrastructure will page the Hubber on call, interrupt their flow, and possibly wake them up. -- CONTRIBUTING.md in github/puppet, October 2015

Slide 28

Slide 28 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Welcome!! 28 ! This repository runs the production
 infrastructure of github.com and many other
 - services. Any changes that break production
 - infrastructure will page the Hubber on call,
 - interrupt their flow, and possibly wake them up. + services. We encourage (and expect) Puppet
 + contributions from everyone within GitHub
 + engineering. Changes made here can break
 + production infrastructure, so you need to be
 + careful. If you need help or a second pair of
 + eyes, please feel free to visit us in the
 + `#puppet` channel...

Slide 29

Slide 29 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Getting contributors on board 29 !

Slide 30

Slide 30 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Engaged contributors 30 !

Slide 31

Slide 31 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Anti-pattern of engagement 31 ! "Good programmers know what to write. Great ones know what to rewrite (and reuse)." -- Eric S. Raymond, The Cathedral and the Bazaar Brand new programmers will copy the first thing that looks close. But also consider...

Slide 32

Slide 32 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Which code is the right code? 32 !

Slide 33

Slide 33 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Perpetuating technical debt 33 !

Slide 34

Slide 34 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Perpetuating technical debt 34 !

Slide 35

Slide 35 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Supporting your contributors - education 35 ! Learn Puppet... • Book • Tutorial • Formal training • Hack time + mentor

Slide 36

Slide 36 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Supporting your contributors - education 36 ! Our Puppet repo... • Where is it? • How to test? • How to deploy? • Where to get help? Learn Puppet... • Book • Tutorial • Formal training • Hack time + mentor

Slide 37

Slide 37 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Supporting your contributors - documentation 37 !

Slide 38

Slide 38 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Technical debt 38 !

Slide 39

Slide 39 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Technical debt 39 ! Technical debt is the enemy of scalability

Slide 40

Slide 40 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Watch the technical debt grow 40 !

Slide 41

Slide 41 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Watch the technical debt grow 41 !

Slide 42

Slide 42 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Watch the technical debt grow 42 !

Slide 43

Slide 43 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Watch the technical debt grow 43 ! "If this host is not in physical data center 1, it must be in AWS!"

Slide 44

Slide 44 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Modularize for future growth 44 ! Caution: Over-simplified pseudo-code!

Slide 45

Slide 45 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 "Good clever" vs. "Bad clever" 45 !

Slide 46

Slide 46 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 "Bad clever" example 46 ! ipaddress = '172.17.0.1' WITH DOCKER RUNNING m = 25 fqdn_rand(60) A BETTER WAY

Slide 47

Slide 47 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Is your solution "good clever"? 47 ! Evaluate your clever solution: • Will you remember how it works? • Will others understand how it works? • What advantages does this have over the
 generally accepted path? • If you were on stage at PuppetConf,
 would the audience or ?

Slide 48

Slide 48 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Paying down technical debt 48 ! • Do you set aside time to work on... • Refactoring • Deprecating • Deleting • Enforcing code quality

Slide 49

Slide 49 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Paying down technical debt 49 ! • Do you set aside time to work on... • Refactoring • Deprecating • Deleting • Enforcing code quality • Do you praise the engineers who do? • Raises and promotions • Performance reviews

Slide 50

Slide 50 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Managing chaos 50 !

Slide 51

Slide 51 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Monitoring, metrics, and alerting 51 ! Monitors & Metrics

Slide 52

Slide 52 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Monitoring, metrics, and alerting 52 ! Monitors: • Catalog fails to generate • Failing resources on host • Host hasn't reported in Metrics: • Catalog compile time • Catalog run time • CI build time

Slide 53

Slide 53 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Sometimes things go horribly wrong 53 ! .puppet panic

Slide 54

Slide 54 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Sometimes things go horribly wrong 54 !

Slide 55

Slide 55 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Productivity 55 !

Slide 56

Slide 56 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Development cycle before catalog analysis 56 !

Slide 57

Slide 57 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Agent testing 57 ! MASTER AGENT OS VERSION CPUS MEMORY DATACENTER MAC ADDRESS DISKS FQDN HW MODEL EC2? IP ADDR FACTS 1. Agent sends
 facts to master 2. Master returns
 catalog to agent 3. Agent applies catalog to system code hiera puppetdb ENC logs

Slide 58

Slide 58 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 My first catalog difference analysis 58 ! for host in $hosts ; do git checkout master puppet master --compile $host > /tmp/old.json git checkout my-changed-branch puppet master --compile $host > /tmp/new.json diff /tmp/old.json /tmp/new.json > /tmp/$host.diff done Caution: Over-simplified pseudo-code!

Slide 59

Slide 59 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Catalog testing 59 ! MASTER AGENT 1. Agent sends
 facts to master 2. Master returns
 catalog to agent 3. Agent applies catalog to system code hiera puppetdb ENC logs OS VERSION CPUS MEMORY DATACENTER MAC ADDRESS DISKS FQDN HW MODEL EC2? IP ADDR FACTS

Slide 60

Slide 60 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Developing a catalog difference tool 60 !

Slide 61

Slide 61 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Catalog differences with octocatalog-diff 61 ! Catalog compilation: • Check out "from" and "to" branches • Munge hiera config, ENC, etc. • Facts from PuppetDB • Build catalogs using Puppet Comparison analysis: • Resources added, removed, changed • Display human-readable output

Slide 62

Slide 62 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Detecting collateral damage 62 ! class mysql::params { $mysql_port = 3306 $mysql_version = '5.6.20-1' }

Slide 63

Slide 63 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Detecting collateral damage 63 ! class mysql::params { $mysql_port = 3306 $mysql_version = '5.6.20-1' } class profile::frontend { package { 'mysql-client': ensure => $mysql::params::mysql_version, } }

Slide 64

Slide 64 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Detecting collateral damage 64 ! class mysql::params { $mysql_port = 3306 $mysql_version = '5.5.20-1' } class profile::frontend { package { 'mysql-client': ensure => $mysql::params::mysql_version, } } 5.6.24-7 5.6.24-7

Slide 65

Slide 65 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Running octocatalog-diff across a fleet 65 !

Slide 66

Slide 66 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Running octocatalog-diff across a fleet 66 !

Slide 67

Slide 67 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Productivity improvements with octocatalog-diff 67 ! With octocatalog-diff

Slide 68

Slide 68 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Productivity improvements with octocatalog-diff 68 ! With octocatalog-diff

Slide 69

Slide 69 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 We have open-sourced octocatalog-diff 69 ! https://github.com/github/octocatalog-diff

Slide 70

Slide 70 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 70 + , ! CULTURAL TECHNICAL • Ownership • Participation • Manage chaos • Workflow • Eliminating technical debt • https://github.com/github/octocatalog-diff x Session summary

Slide 71

Slide 71 text

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016 Scaling Puppet and Puppet Culture 71 ! @kpaulisse kpaulisse x kpaulisse@github.com # kpaulisse (puppetcommunity.slack.com) Kevin Paulisse (SRE @ GitHub) https://github.com/github/octocatalog-diff https://linkedin.com/in/kpaulisse