PuppetConf 2017: Beyond rspec - Innovative Strategies for Confident CI

Slide 1

Slide 1 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 1 Beyond rspec: Innovative Strategies for Conﬁdent CI Kevin Paulisse SRE @ GitHub

Slide 2

Slide 2 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Introduction: About GitHub 2 !

Slide 3

Slide 3 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Introduction: About GitHub 3 !

Slide 4

Slide 4 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Introduction: About Kevin Paulisse 4 " ! Text @kpaulisse kpaulisse [email protected] x # ↑ Work: GitHub ↓ Live: Madison, Wisc.

Slide 5

Slide 5 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Agenda 5 ! 1. Introduction   2. Traditional Puppet Testing - Unit Tests - Integration Tests 3. Less Traditional Puppet Testing - Tools - Techniques - Practical Applications 4. Time for Questions

Slide 6

Slide 6 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Deﬁnition: What is CI? 6 ! CI = Continuous Integration Verifying each code commit with  one or more automated tests.

Slide 7

Slide 7 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Deﬁnition: What is CI? 7 ! CI = Continuous Integration Verifying each code commit with  one or more automated tests. CD = Continuous Delivery Continuous Deployment Producing software in short cycles  so that software can be released  at any time.

Slide 8

Slide 8 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Introduction: Puppet at GitHub 8 ! • First Puppet Commit: September 8, 2008 (GitHub launched on April 10, 2008) • Puppet Versions: 0.24.x - 4.8.x • Lines of Code: 200,000+ • Puppet Resources: 2,000+ per node

Slide 9

Slide 9 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Introduction: Puppet Culture 9 ! https://www.youtube.com/watch?v=H7cQcoXVacU PuppetConf 2016: Scaling Puppet (and Puppet Culture)

Slide 10

Slide 10 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Introduction: GitHub Flow 10 ! $ Branch % Pull Request & Review ' Test (CI) ( Branch Deploy ) Ship * Clone

Slide 11

Slide 11 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Introduction: Kubernetes at GitHub 11 ! https://githubengineering.com/kubernetes-at-github/

Slide 12

Slide 12 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 CI is for Humans 12 !

Slide 13

Slide 13 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 CI is for Humans 13 ! Good CI stops humans from shipping bad code. Great CI helps humans to ship good code.

Slide 14

Slide 14 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Puppet CI at GitHub 14 ! • Run 15 CI jobs with each push

Slide 15

Slide 15 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 rspec-puppet 15 ! https://github.com/rodjek/rspec-puppet

Slide 16

Slide 16 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 rspec-puppet 16 ! puppet-distributed: rspec-puppet for classes and deﬁnes Distributed in parallel across 6 containers puppet-functions: rspec-puppet for custom functions puppet-ops-mon: Split out tests for our monitoring system hosts because those catalogs take 2+ minutes to compile puppet-windows: rspec-puppet tests for the one Windows system that we manage (being retired)

Slide 17

Slide 17 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Example: rspec-puppet 17 !

Slide 18

Slide 18 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Problems 18 !

Slide 19

Slide 19 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Problems 19 !

Slide 20

Slide 20 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Problems 20 !

Slide 21

Slide 21 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 The Fix ... Until Next Time 21 !

Slide 22

Slide 22 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 The Hard Truth 22 !

Slide 23

Slide 23 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Octofacts 23 ! Octofacts Automated, real fact ﬁxtures for rspec-puppet Authors: @kpaulisse, @antonio License: MIT URL: https://github.com/github/octofacts

Slide 24

Slide 24 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Octofacts in Action 24 !

Slide 25

Slide 25 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Octofacts Fixture 25 !

Slide 26

Slide 26 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Octofacts Updates 26 !

Slide 27

Slide 27 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Octofacts Updates 27 !

Slide 28

Slide 28 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Octofacts Conclusion 28 ! Octofacts Automated, real fact ﬁxtures for rspec-puppet Authors: @kpaulisse, @antonio License: MIT URL: https://github.com/github/octofacts

Slide 29

Slide 29 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Puppet CI at GitHub 29 ! • Run 15 CI jobs with each push • "Traditional" CI jobs • rspec-puppet • integration

Slide 30

Slide 30 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Puppet CI at GitHub 30 ! • Run 15 CI jobs with each push • "Traditional" CI jobs • rspec-puppet • integration

Slide 31

Slide 31 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Integration Tests Powered by Docker-Compose 31 !

Slide 32

Slide 32 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Integration Tests Powered by Docker-Compose 32 !

Slide 33

Slide 33 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Provisioning: Old Style 33 ! Base Operating System One-Shot Provisioning Final System Run Puppet 45+ Minutes

Slide 34

Slide 34 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Image Based Provisioning Workﬂow 34 ! tar.gz Integration Test Docker Container Export Hiera Data Puppet Code Stub Data Provisioning Provisioning Physical Nodes EC2 Nodes S3 AMI * Extract * Snapshot * Build AMI

Slide 35

Slide 35 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Provisioning Methods Compared 35 ! Base Operating System One-Shot Provisioning Image Based Provisioning Final System Run Puppet 45+ Minutes Base Operating System Tested Image (AMI or S3 tar.gz) Integration Tests Final System Run Puppet 5 Minutes CI

Slide 36

Slide 36 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Puppet CI at GitHub 36 ! • Run 15 CI jobs with each push • "Traditional" CI jobs • rspec-puppet • integration • "Non-traditional" CI jobs • puppet-lint • puppet-catalogs • puppet-real-host-compile • puppet-utility

Slide 37

Slide 37 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 "puppet-lint" CI Job 37 ! Puppet Lint Check that your Puppet manifests conform to the style guide Author: Tim Sharpe License: MIT URL: https://github.com/rodjek/puppet-lint

Slide 38

Slide 38 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Example Output from "puppet-lint" 38 !

Slide 39

Slide 39 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 "puppet-catalogs" CI Job 39 ! PuppetDB Puppet Agent Catalog (JSON) Hiera Data Puppet Code Inventory Service Host Filter All Hosts Host Recent Facts

Slide 40

Slide 40 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 "puppet-catalogs" CI Job 40 ! PuppetDB Puppet Agent Catalog (JSON) Hiera Data Puppet Code Inventory Service Host Filter All Hosts Host Recent Facts

Slide 41

Slide 41 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Output from "puppet-catalogs" 41 !

Slide 42

Slide 42 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Limitations of "puppet-catalogs" 42 ! 1. Pulling facts from PuppetDB Adding, removing, or changing facts in your new code will not be reﬂected.

Slide 43

Slide 43 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Limitations of "puppet-catalogs" 43 ! 1. Pulling facts from PuppetDB Adding, removing, or changing facts in your new code will not be reﬂected. 2. Successful compile != correct resources Just because the catalog compiles, that doesn't mean it's what you expect.

Slide 44

Slide 44 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Limitations of "puppet-catalogs" 44 ! 1. Pulling facts from PuppetDB Adding, removing, or changing facts in your new code will not be reﬂected. 3. Successful compile != successful apply Compiling the catalog does not mean the catalog will apply correctly on the agents. 2. Successful compile != correct resources Just because the catalog compiles, that doesn't mean it's what you expect.

Slide 45

Slide 45 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Summary of "puppet-catalogs" 45 ! Strengths Weaknesses • Detects problems across multiple roles • Does not touch production nodes • Pass/fail output • Does not exercise actual facts • Does not verify catalog is as desired • Does not verify catalog will apply Sales Pitch Conﬁrm via a pass-fail test that your changes didn't break critical roles

Slide 46

Slide 46 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 "puppet-real-host-compile" CI Job 46 ! Puppet Agent Catalog (JSON) Hiera Data Puppet Code Facter

Slide 47

Slide 47 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Limitations of "puppet-real-host-compile" 47 ! 1. Only covers one role We don't want our CI environment to be able to touch real servers.

Slide 48

Slide 48 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Limitations of "puppet-real-host-compile" 48 ! 1. Only covers one role We don't want our CI environment to be able to touch real servers. 3. Successful compile != successful apply Compiling the catalog does not mean the catalog will apply correctly on the agents. 2. Successful compile != correct resources Just because the catalog compiles, that doesn't mean it's what you expect.

Slide 49

Slide 49 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Summary of "puppet-real-host-compile" 49 ! Strengths Weaknesses • Exercises custom facts • Does not touch production systems • Pass/fail output • Only covers one role • Does not verify catalog is as desired • Does not verify catalog will apply Sales Pitch Compile an actual Puppet catalog on an actual node with no shortcuts

Slide 50

Slide 50 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 "puppet-utility" CI Job 50 !

Slide 51

Slide 51 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 "puppet-utility": Hiera YAML Validation 51 !

Slide 52

Slide 52 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 "puppet-utility": Hiera YAML Validation 52 ! Test strategy: 1. Parse YAML to hash 2. Regenerate YAML 3. Parse regenerated YAML 4. Compare key counts

Slide 53

Slide 53 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 "puppet-utility": Project Setup 53 !

Slide 54

Slide 54 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 What Changed? 54 !

Slide 55

Slide 55 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Octocatalog-Diff 55 ! Octocatalog-Diff Compile and Compare Puppet Catalogs Author: @kpaulisse License: MIT URL: https://github.com/github/octocatalog-diff

Slide 56

Slide 56 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Octocatalog-Diﬀ: History 56 !

Slide 57

Slide 57 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Octocatalog-Diff: History 57 ! for host in $hosts ; do git checkout master puppet master --compile $host > /tmp/old.json git checkout my-changed-branch puppet master --compile $host > /tmp/new.json diff /tmp/old.json /tmp/new.json > /tmp/$host.diff done Caution: Over-simplified pseudo-code!

Slide 58

Slide 58 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Octocatalog-Diﬀ 58 ! Recent Facts Puppet Agent Catalog (JSON) Hiera Data Puppet Code Master Branch Puppet Agent Catalog (JSON) Hiera Data Puppet Code Feature Branch PuppetDB ENC ENC Data

Slide 59

Slide 59 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Octocatalog-Diff Command Line Usage 59 ! Catalog compilation: • Check out "from" and "to" branches • Munge hiera config, ENC, etc. • Facts from PuppetDB • Build catalogs using Puppet Comparison analysis: • Resources added, removed, changed • Display human-readable output

Slide 60

Slide 60 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Octocatalog-Diﬀ Across the Fleet 60 !

Slide 61

Slide 61 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Octocatalog-Diﬀ and Code Reviews 61 !

Slide 62

Slide 62 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Octocatalog-Diﬀ and Code Reviews 62 !

Slide 63

Slide 63 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Optimizing Octocatalog-Diﬀ 63 !

Slide 64

Slide 64 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Optimizing Octocatalog-Diﬀ 64 !

Slide 65

Slide 65 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Load on PuppetDB 65 ! CI Node CI Node CI Node CI Node CI Node CI Node CI Node CI Node PuppetDB PuppetDB PuppetDB catalog-diff #1 catalog-diff #2 catalog-diff #3 ELB Old Catalog New Catalog Old Catalog New Catalog Queries: -Facts -Exported Resources postgresql

Slide 66

Slide 66 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Optimizing (or, Working Around) PuppetDB 66 !

Slide 67

Slide 67 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Optimizing (or, Working Around) PuppetDB 67 ! Not recommended for production use

Slide 68

Slide 68 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Octocatalog-Diﬀ Limitation: Underlying Providers 68 !

Slide 69

Slide 69 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Octocatalog-Diﬀ Limitation: Underlying Providers 69 !

Slide 70

Slide 70 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Octocatalog-Diﬀ Limitation: Agents Apply Catalogs 70 !

Slide 71

Slide 71 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Octocatalog-Diﬀ Limitation: Fact Gathering 71 ! Recent Facts Puppet Agent Catalog (JSON) Hiera Data Puppet Code Master Branch Puppet Agent Catalog (JSON) Hiera Data Puppet Code Feature Branch PuppetDB ENC ENC Data From Last Puppet Run (not from your branch) From Your Branch

Slide 72

Slide 72 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Octocatalog-Diff: What's an ENC? 72 ! ENC: External Node Classifier A script that runs on the Puppet server and queries an external service. It gathers classification information and parameters for a host.

Slide 73

Slide 73 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Octocatalog-Diﬀ: Validating ENC Changes 73 !

Slide 74

Slide 74 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Octocatalog-Diﬀ: Validating ENC Changes 74 !

Slide 75

Slide 75 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Octocatalog-Diﬀ: Validating Fact Changes 75 !

Slide 76

Slide 76 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Octocatalog-Diﬀ: Validating Fact Changes 76 !

Slide 77

Slide 77 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Octocatalog-Diﬀ: Validating Fact Changes 77 ! --to-fact-override 'gh_host_app=(nil)' --to-fact-override 'gh_host_role=(nil)'

Slide 78

Slide 78 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Octocatalog-Diﬀ: Validating Puppet Server Upgrade 78 ! https://puppet.com/blog/upgrading-to-puppet-4-at-github

Slide 79

Slide 79 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Octocatalog-Diﬀ: Validating Puppet Agent Upgrade 79 ! 4.x Fact Files in S3 Recent Facts Puppet 3.x Puppet Agent Catalog (JSON) Hiera Data Puppet Code Master Branch Puppet Agent Catalog (JSON) Hiera Data Puppet Code Master or Feature Branch PuppetDB ENC

Slide 80

Slide 80 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Octocatalog-Diﬀ: Validating Puppet Agent Upgrade 80 !

Slide 81

Slide 81 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 When CI is not Enough... 81 !

Slide 82

Slide 82 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Branch Deploys 82 !

Slide 83

Slide 83 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Real Deploys 83 !

Slide 84

Slide 84 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Other Tips... 84 !

Slide 85

Slide 85 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Protected Branches and Code Review 85 ! • Protected branches: • Require certain CI jobs to pass before merging • Require an approved code review before merging • Require branches to be  up-to-date before merging

Slide 86

Slide 86 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Time to Completion 86 !

Slide 87

Slide 87 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Time to Completion 87 !

Slide 88

Slide 88 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Re-Evaluate Necessity 88 !

Slide 89

Slide 89 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Conﬁdent CI PUPPETCONF 2017 Conclusion 89 !

Slide 90

Slide 90 text

! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Beyond rspec: Innovative Strategies for Confident CI 90 ! @kpaulisse kpaulisse x [email protected] # kpaulisse (puppetcommunity.slack.com) Kevin Paulisse - Contact Info & Links https://github.com/github/octofacts https://linkedin.com/in/kpaulisse https://github.com/github/octocatalog-diff x