PuppetConf 2017: Beyond rspec - Innovative Strategies for Confident CI

Continuous Integration (CI) testing provides instant feedback as you develop Puppet code. While it’s great that your rspec-puppet and beaker tests pass using fixtures and stubs, you can get a false sense of security from unanticipated edge cases, unrealistic assumptions, or the lack of complete coverage. You may be tempted to address these limitations by writing even more tests, but that would only exacerbate your frustration. Instead, you could use some alternate techniques to make your CI pipeline predict real-world behavior more accurately, and thereby gain confidence in your test results. In this session, we will explore some less traditional yet more powerful techniques for automated testing of Puppet code. Real-time catalog compilation testing obtains more realistic results by simulating your production environment with the latest data within CI. Learn how GitHub scaled this strategy to test hundreds of unique combinations while keeping build times reasonable. Catalog difference testing with GitHub’s open source catalog difference tool (octocatalog-diff) increases development velocity and reduces deployment risk. The speaker, who is the original author of the tool, will explain why and how octocatalog-diff was developed and demonstrate how it can be used to streamline code reviews and validate Puppet upgrades. Real host testing of catalog compilation on actual nodes can exercise external dependencies and custom facts to provide the most accurate results. Overcome the limitations of human-defined fact fixtures and assumptions, avoiding situations that defy fixtures without you realizing it. Throughout the session, pick up other useful ideas, including how and why to run puppet-lint as a CI job, how to test project setup, and how Hubot keeps Dockerfiles up to date at GitHub. You will leave this session with fresh ideas to improve your CI pipeline, which will lead to confident and reliable deployments.

Transcript

1. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 1 Beyond rspec: Innovative Strategies for Confident CI Kevin Paulisse SRE @ GitHub
2. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Introduction: About GitHub 2 !
3. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Introduction: About GitHub 3 !
4. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Introduction: About Kevin Paulisse 4 " ! Text @kpaulisse kpaulisse kpaulisse@gmail.com x # ↑ Work: GitHub ↓ Live: Madison, Wisc.
5. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Agenda 5 ! 1. Introduction 
 2. Traditional Puppet Testing - Unit Tests - Integration Tests 3. Less Traditional Puppet Testing - Tools - Techniques - Practical Applications 4. Time for Questions
6. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Definition: What is CI? 6 ! CI = Continuous Integration Verifying each code commit with
 one or more automated tests.
7. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Definition: What is CI? 7 ! CI = Continuous Integration Verifying each code commit with
 one or more automated tests. CD = Continuous Delivery Continuous Deployment Producing software in short cycles
 so that software can be released
 at any time.
8. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Introduction: Puppet at GitHub 8 ! • First Puppet Commit: September 8, 2008 (GitHub launched on April 10, 2008) • Puppet Versions: 0.24.x - 4.8.x • Lines of Code: 200,000+ • Puppet Resources: 2,000+ per node
9. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Introduction: Puppet Culture 9 ! https://www.youtube.com/watch?v=H7cQcoXVacU PuppetConf 2016: Scaling Puppet (and Puppet Culture)
10. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Introduction: GitHub Flow 10 ! $ Branch % Pull Request & Review ' Test (CI) ( Branch Deploy ) Ship * Clone
11. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Introduction: Kubernetes at GitHub 11 ! https://githubengineering.com/kubernetes-at-github/
12. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 CI is for Humans 12 !
13. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 CI is for Humans 13 ! Good CI stops humans from shipping bad code. Great CI helps humans to ship good code.
14. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Puppet CI at GitHub 14 ! • Run 15 CI jobs with each push
15. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 rspec-puppet 15 ! https://github.com/rodjek/rspec-puppet
16. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 rspec-puppet 16 ! puppet-distributed: rspec-puppet for classes and defines Distributed in parallel across 6 containers puppet-functions: rspec-puppet for custom functions puppet-ops-mon: Split out tests for our monitoring system hosts because those catalogs take 2+ minutes to compile puppet-windows: rspec-puppet tests for the one Windows system that we manage (being retired)
17. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Example: rspec-puppet 17 !
18. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Problems 18 !
19. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Problems 19 !
20. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Problems 20 !
21. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 The Fix ... Until Next Time 21 !
22. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 The Hard Truth 22 !
23. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Octofacts 23 ! Octofacts Automated, real fact fixtures for rspec-puppet Authors: @kpaulisse, @antonio License: MIT URL: https://github.com/github/octofacts
24. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Octofacts in Action 24 !
25. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Octofacts Fixture 25 !
26. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Octofacts Updates 26 !
27. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Octofacts Updates 27 !
28. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Octofacts Conclusion 28 ! Octofacts Automated, real fact fixtures for rspec-puppet Authors: @kpaulisse, @antonio License: MIT URL: https://github.com/github/octofacts
29. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Puppet CI at GitHub 29 ! • Run 15 CI jobs with each push • "Traditional" CI jobs • rspec-puppet • integration
30. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Puppet CI at GitHub 30 ! • Run 15 CI jobs with each push • "Traditional" CI jobs • rspec-puppet • integration
31. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Integration Tests Powered by Docker-Compose 31 !
32. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Integration Tests Powered by Docker-Compose 32 !
33. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Provisioning: Old Style 33 ! Base Operating System One-Shot Provisioning Final System Run Puppet 45+ Minutes
34. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Image Based Provisioning Workflow 34 ! tar.gz Integration Test Docker Container Export Hiera Data Puppet Code Stub Data Provisioning Provisioning Physical Nodes EC2 Nodes S3 AMI * Extract * Snapshot * Build AMI
35. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Provisioning Methods Compared 35 ! Base Operating System One-Shot Provisioning Image Based Provisioning Final System Run Puppet 45+ Minutes Base Operating System Tested Image (AMI or S3 tar.gz) Integration Tests Final System Run Puppet 5 Minutes CI
36. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Puppet CI at GitHub 36 ! • Run 15 CI jobs with each push • "Traditional" CI jobs • rspec-puppet • integration • "Non-traditional" CI jobs • puppet-lint • puppet-catalogs • puppet-real-host-compile • puppet-utility
37. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 "puppet-lint" CI Job 37 ! Puppet Lint Check that your Puppet manifests conform to the style guide Author: Tim Sharpe License: MIT URL: https://github.com/rodjek/puppet-lint
38. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Example Output from "puppet-lint" 38 !
39. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 "puppet-catalogs" CI Job 39 ! PuppetDB Puppet Agent Catalog (JSON) Hiera Data Puppet Code Inventory Service Host Filter All Hosts Host Recent Facts
40. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 "puppet-catalogs" CI Job 40 ! PuppetDB Puppet Agent Catalog (JSON) Hiera Data Puppet Code Inventory Service Host Filter All Hosts Host Recent Facts
41. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Output from "puppet-catalogs" 41 !
42. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Limitations of "puppet-catalogs" 42 ! 1. Pulling facts from PuppetDB Adding, removing, or changing facts in your new code will not be reflected.
43. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Limitations of "puppet-catalogs" 43 ! 1. Pulling facts from PuppetDB Adding, removing, or changing facts in your new code will not be reflected. 2. Successful compile != correct resources Just because the catalog compiles, that doesn't mean it's what you expect.
44. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Limitations of "puppet-catalogs" 44 ! 1. Pulling facts from PuppetDB Adding, removing, or changing facts in your new code will not be reflected. 3. Successful compile != successful apply Compiling the catalog does not mean the catalog will apply correctly on the agents. 2. Successful compile != correct resources Just because the catalog compiles, that doesn't mean it's what you expect.
45. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Summary of "puppet-catalogs" 45 ! Strengths Weaknesses • Detects problems across multiple roles • Does not touch production nodes • Pass/fail output • Does not exercise actual facts • Does not verify catalog is as desired • Does not verify catalog will apply Sales Pitch Confirm via a pass-fail test that your changes didn't break critical roles
46. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 "puppet-real-host-compile" CI Job 46 ! Puppet Agent Catalog (JSON) Hiera Data Puppet Code Facter
47. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Limitations of "puppet-real-host-compile" 47 ! 1. Only covers one role We don't want our CI environment to be able to touch real servers.
48. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Limitations of "puppet-real-host-compile" 48 ! 1. Only covers one role We don't want our CI environment to be able to touch real servers. 3. Successful compile != successful apply Compiling the catalog does not mean the catalog will apply correctly on the agents. 2. Successful compile != correct resources Just because the catalog compiles, that doesn't mean it's what you expect.
49. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Summary of "puppet-real-host-compile" 49 ! Strengths Weaknesses • Exercises custom facts • Does not touch production systems • Pass/fail output • Only covers one role • Does not verify catalog is as desired • Does not verify catalog will apply Sales Pitch Compile an actual Puppet catalog on an actual node with no shortcuts
50. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 "puppet-utility" CI Job 50 !
51. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 "puppet-utility": Hiera YAML Validation 51 !
52. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 "puppet-utility": Hiera YAML Validation 52 ! Test strategy: 1. Parse YAML to hash 2. Regenerate YAML 3. Parse regenerated YAML 4. Compare key counts
53. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 "puppet-utility": Project Setup 53 !
54. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 What Changed? 54 !
55. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Octocatalog-Diff 55 ! Octocatalog-Diff Compile and Compare Puppet Catalogs Author: @kpaulisse License: MIT URL: https://github.com/github/octocatalog-diff
56. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Octocatalog-Diff: History 56 !
57. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Octocatalog-Diff: History 57 ! for host in $hosts ; do git checkout master puppet master --compile $host > /tmp/old.json git checkout my-changed-branch puppet master --compile $host > /tmp/new.json diff /tmp/old.json /tmp/new.json > /tmp/$host.diff done Caution: Over-simplified pseudo-code!
58. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Octocatalog-Diff 58 ! Recent Facts Puppet Agent Catalog (JSON) Hiera Data Puppet Code Master Branch Puppet Agent Catalog (JSON) Hiera Data Puppet Code Feature Branch PuppetDB ENC ENC Data
59. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Octocatalog-Diff Command Line Usage 59 ! Catalog compilation: • Check out "from" and "to" branches • Munge hiera config, ENC, etc. • Facts from PuppetDB • Build catalogs using Puppet Comparison analysis: • Resources added, removed, changed • Display human-readable output
60. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Octocatalog-Diff Across the Fleet 60 !
61. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Octocatalog-Diff and Code Reviews 61 !
62. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Octocatalog-Diff and Code Reviews 62 !
63. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Optimizing Octocatalog-Diff 63 !
64. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Optimizing Octocatalog-Diff 64 !
65. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Load on PuppetDB 65 ! CI Node CI Node CI Node CI Node CI Node CI Node CI Node CI Node PuppetDB PuppetDB PuppetDB catalog-diff #1 catalog-diff #2 catalog-diff #3 ELB Old Catalog New Catalog Old Catalog New Catalog Queries: -Facts -Exported Resources postgresql
66. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Optimizing (or, Working Around) PuppetDB 66 !
67. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Optimizing (or, Working Around) PuppetDB 67 ! Not recommended for production use
68. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Octocatalog-Diff Limitation: Underlying Providers 68 !
69. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Octocatalog-Diff Limitation: Underlying Providers 69 !
70. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Octocatalog-Diff Limitation: Agents Apply Catalogs 70 !
71. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Octocatalog-Diff Limitation: Fact Gathering 71 ! Recent Facts Puppet Agent Catalog (JSON) Hiera Data Puppet Code Master Branch Puppet Agent Catalog (JSON) Hiera Data Puppet Code Feature Branch PuppetDB ENC ENC Data From Last Puppet Run (not from your branch) From Your Branch
72. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Octocatalog-Diff: What's an ENC? 72 ! ENC: External Node Classifier A script that runs on the Puppet server and queries an external service. It gathers classification information and parameters for a host.
73. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Octocatalog-Diff: Validating ENC Changes 73 !
74. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Octocatalog-Diff: Validating ENC Changes 74 !
75. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Octocatalog-Diff: Validating Fact Changes 75 !
76. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Octocatalog-Diff: Validating Fact Changes 76 !
77. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Octocatalog-Diff: Validating Fact Changes 77 ! --to-fact-override 'gh_host_app=(nil)' --to-fact-override 'gh_host_role=(nil)'
78. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Octocatalog-Diff: Validating Puppet Server Upgrade 78 ! https://puppet.com/blog/upgrading-to-puppet-4-at-github
79. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Octocatalog-Diff: Validating Puppet Agent Upgrade 79 ! 4.x Fact Files in S3 Recent Facts Puppet 3.x Puppet Agent Catalog (JSON) Hiera Data Puppet Code Master Branch Puppet Agent Catalog (JSON) Hiera Data Puppet Code Master or Feature Branch PuppetDB ENC
80. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Octocatalog-Diff: Validating Puppet Agent Upgrade 80 !
81. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 When CI is not Enough... 81 !
82. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Branch Deploys 82 !
83. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Real Deploys 83 !
84. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Other Tips... 84 !
85. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Protected Branches and Code Review 85 ! • Protected branches: • Require certain CI jobs to pass before merging • Require an approved code review before merging • Require branches to be
 up-to-date before merging
86. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Time to Completion 86 !
87. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Time to Completion 87 !
88. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Re-Evaluate Necessity 88 !
89. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Conclusion 89 !
90. ! @kpaulisse - Beyond rspec: Innovative Strategies for Confident CI PUPPETCONF 2017 Beyond rspec: Innovative Strategies for Confident CI 90 ! @kpaulisse kpaulisse x kpaulisse@gmail.com # kpaulisse (puppetcommunity.slack.com) Kevin Paulisse - Contact Info & Links https://github.com/github/octofacts https://linkedin.com/in/kpaulisse https://github.com/github/octocatalog-diff x