The Future of Testing Puppet Code

A talk from #PuppetConf 2016 covering the state of affairs with testing Puppet code. Mainly a talk about what we, as a community, can do to improve the tools and adoption of those tools. Lots of survey data and bigquery research.

Transcript

1. (without introducing more risk) The Future of Testing Puppet Code Puppet Gareth Rushgrove Who, What, Why, How
2. (without introducing more risk) @garethr
3. (without introducing more risk) Gareth Rushgrove
4. (without introducing more risk) This Talk What we’ll cover
5. - Why test Puppet anyway - Existing Puppet testing tools - Different users and usecases - Future challenges Gareth Rushgrove
6. Mainly a talk about problems, rather than solutions Gareth Rushgrove
7. Not a talk about how to test Puppet code. Lot of existing content on that subject Gareth Rushgrove
8. (without introducing more risk) Gareth Rushgrove PuppetConf 2014
9. (without introducing more risk) Gareth Rushgrove Survey of 300 Puppet users
10. (without introducing more risk) Gareth Rushgrove
11. (without introducing more risk) Why Test Puppet And a little history
12. Gareth Rushgrove
13. - You have lots of Puppet code - You’re targeting a wide range of OS’s - The Puppet language has more features Gareth Rushgrove
14. (without introducing more risk) Gareth Rushgrove Lots of code But how much is lots
15. Note that this just represents public code on GitHub. Lots of private Puppet code out there too Gareth Rushgrove
16. (without introducing more risk) Gareth Rushgrove How many .pp files on GitHub? SELECT COUNT(*) FROM [puppet.puppet_files] -- 489725
17. (without introducing more risk) Gareth Rushgrove How many lines of Puppet code? SELECT COUNT(line) total_lines FROM ( SELECT SPLIT(content, '\n') AS line FROM [puppet.puppet_content] ) -- 6697099
18. (without introducing more risk) Gareth Rushgrove How big are .pp files? SELECT QUANTILES(total_lines, 10) AS q FROM ( SELECT COUNT(line) total_lines FROM ( SELECT SPLIT(content, '\n') AS line, id FROM [puppet.puppet_content] ) GROUP BY id )
19. 50% of Puppet files are 20 lines or less. 90% of Puppet files are 100 lines or less Gareth Rushgrove
20. The largest Puppet file public on GitHub has 25,238 lines Gareth Rushgrove
21. (without introducing more risk) Gareth Rushgrove Mixed environments So many platforms
22. Gareth Rushgrove
23. A third of respondents run more than one OS on there development machine Gareth Rushgrove
24. Gareth Rushgrove
25. 60% of respondents run more than one OS in production Gareth Rushgrove
26. (without introducing more risk) Gareth Rushgrove The Puppet language Types, lambdas, iterators, functions
27. Gareth Rushgrove
28. (without introducing more risk) Existing Tools Providing some context
29. Gareth Rushgrove
30. (without introducing more risk) Core tools Common starting points Gareth Rushgrove
31. (without introducing more risk) Gareth Rushgrove puppet-lint
32. (without introducing more risk) Gareth Rushgrove Style conformance $ puppet-lint /etc/puppet/modules foo/manifests/bar.pp - ERROR: trailing whitespace found on li apache/manifests/server.pp - WARNING: variable not enclosed i ...
33. (without introducing more risk) Gareth Rushgrove rspec-puppet
34. (without introducing more risk) Gareth Rushgrove Unit tests for Puppet it { is_expected.to compile } it { is_expected.to contain_class('db') } it do is_expected.to contain_package('mysql-server') .with_ensure('present') end it { is_expected.to have_exec_resource_count(0) }
35. (without introducing more risk) Gareth Rushgrove puppet-syntax
36. (without introducing more risk) Gareth Rushgrove Popular Puppet gems SELECT package, COUNT(*) count FROM ( SELECT REGEXP_EXTRACT(line, r'gem [\'|"](.*)[\'|"]') packag FROM ( SELECT SPLIT(content, '\n') line, id FROM [puppet.gemfile_contents] WHERE content CONTAINS 'gem' ) GROUP BY package, id ) GROUP BY 1 ORDER BY count DESC LIMIT 30;
37. (without introducing more risk) Gareth Rushgrove Popular gem results puppetlabs_spec_helper 1157 puppet 1142 rake 1135 puppet-lint 768 beaker-rspec 610 rspec-puppet 576 metadata-json-lint 566 beaker 440 puppet-blacksmith 425 serverspec 359
38. (without introducing more risk) Gareth Rushgrove puppetlabs_spec_helper
39. (without introducing more risk) Gareth Rushgrove Advanced tools Next steps
40. (without introducing more risk) Gareth Rushgrove beaker-rspec
41. (without introducing more risk) Gareth Rushgrove Serverspec
42. (without introducing more risk) Gareth Rushgrove rubocop
43. (without introducing more risk) Gareth Rushgrove Other tools Alternative approaches
44. (without introducing more risk) Gareth Rushgrove puppet-catalog-test
45. (without introducing more risk) Gareth Rushgrove Test catalogue compilation $ puppet-catalog-test -m modules -M site.pp [INFO] Using puppet 3.0.2 [PASSED] foo (compile time: 0.168182 seconds) [PASSED] default (compile time: 0.003451 seconds) ---------------------------------------- Compiled 2 catalogs in 0.1717 seconds (avg: 0.0858 seconds)
46. (without introducing more risk) Gareth Rushgrove Test Kitchen
47. (without introducing more risk) Different Users And different usecases
48. Gareth Rushgrove
49. Half of respondents regularly write tests for Puppet code Gareth Rushgrove
50. A quarter of respondents never write tests for Puppet code Gareth Rushgrove
51. The problem with a tool-centric world view Gareth Rushgrove
52. Understanding user needs means starting with users Gareth Rushgrove
53. Gareth Rushgrove - The Vox Pupuli folks - Lots of Puppet employees - You in-house Puppet expert - Custom functions The module authors
54. The regular Puppet users Gareth Rushgrove - Roles and profiles - Using Puppet weekly or more - PuppetDB, Puppet Server - Custom facts
55. The occasional Puppet user Gareth Rushgrove - Using Puppet as needed - Basic Puppet code - Hieradata
56. Today different types of Puppet users have a very different experience Gareth Rushgrove
57. Gareth Rushgrove
58. For those with no Ruby experience the average was 2 out of 5 Gareth Rushgrove
59. For those with a little Ruby experience the average was still only 2.5 out of 5 Gareth Rushgrove
60. Of the people who love the testing experience 33% are self-described power users of both Puppet and Ruby Gareth Rushgrove
61. Gareth Rushgrove
62. But more than 50% of respondents have little or no Ruby experience Gareth Rushgrove
63. (without introducing more risk) Challenges What we need for the future
64. (without introducing more risk) Gareth Rushgrove Improved getting started experience Helping a new generation of users
65. - Getting started testing documentation - Single entry point for users - Opinionated workflow Gareth Rushgrove
66. (without introducing more risk) Gareth Rushgrove Adopting acceptance testing Still too hard
67. 27 of the 67 Serverspec users are not using either Beaker or Test Kitchen. Gareth Rushgrove
68. Only 5 of the Beaker (without rspec) users are not also using beaker-rspec Gareth Rushgrove
69. 3 of the Test Kitchen users are also using some flavour of Beaker Gareth Rushgrove
70. So only 26% to 35% of respondents are using an acceptance testing tool with there Puppet code Gareth Rushgrove
71. (without introducing more risk) Gareth Rushgrove Experience matters Consistence interfaces
72. (without introducing more risk) Gareth Rushgrove Inconsistencies $ puppet-lint vs rake lint vs rake validate $ rake syntax vs puppet parser validate $ rspec vs rake spec $ BEAKER_destroy=no rake beaker vs beaker
73. We have a language style guide. What about other Puppet user interfaces? Gareth Rushgrove
74. (without introducing more risk) Gareth Rushgrove Ruby heritage Ruby as a development environment
75. 60% of respondents described themselves as Puppet power users Gareth Rushgrove
76. Only 12% of respondents described themselves as Ruby power users. A third of those work for Puppet! Gareth Rushgrove
77. - More help with using Ruby for testing? - Support for testing in other languages? - Native Puppet support for testing? - Containerised testing abstraction? Gareth Rushgrove
78. (without introducing more risk) Gareth Rushgrove jordan/spec
79. (without introducing more risk) Gareth Rushgrove Test Puppet with Puppet file { '/tmp/test': ensure => present, content => 'The file content', } assertion { 'that the file has the correct contents': subject => File['/tmp/test'], attribute => 'content', expectation => 'The file content', }
80. (without introducing more risk) Conclusions Where to go from here
81. Thanks to the community we have a collection of powerful testing tools in the Puppet ecosystem Gareth Rushgrove
82. But we don’t have consistent user interfaces or the opinionated workflows useful for beginners Gareth Rushgrove
83. Testing shouldn’t just be a power user feature Gareth Rushgrove
84. As a community we should talk about design more often Gareth Rushgrove
85. (without introducing more risk) Gareth Rushgrove https://goo.gl/forms/nchOp5hYdLLgmJ8z1
86. Gareth Rushgrove
87. (without introducing more risk) Questions? And thanks for listening