Test Driven Infrastructure - ChefConf Online 2020

This presentation was given at ChefConf Online 2020 (and other conferences previously). It included a live coding section which is represented by code slides. You can also see the commit stream from start to end here: https://github.com/amaltson/test-driven-redis/compare/chef-conf-2020-start...chef-conf-2020-end

Test Driven Development is a popular concept in Software Development, leading to higher quality code that's easier to maintain. Automated testing is normally a foreign concept in the Operations world, but as you ssh into your servers to make that quick fix or run your updated script (fingers crossed), you might be wondering if there's a better way. A way that gives you the confidence in your script and lets you test those scripts in isolation. Well I have good news for you, there is a better way! Test Driven Infrastructure (TDI) is now possible. I know, it sounds crazy.

At this session you'll learn the how, and more importantly the why, of TDI. You'll see how Chef (or any other Config Management framework) can be tested with Test Kitchen and ServerSpec. You'll also learn how to improve your feedback cycle with Docker, and using the Docker approach on a CI server. There may even be some live demos!

Finally, the Ops world collides with the Dev world in true DevOps testing bliss.

Transcript

3. Speaker Note: I’ll be talking about a problem that plagues….
4. Speaker Note: Both large companies like big financials.
5. Speaker Note: And small, like the hip startups in Liberty Village.
6. Speaker Note: That problem is untested infrastructure. If you’re experiencing this, what might be the symptoms?
7. Speaker Note: Do you cross your fingers before running a script in production?
8. Speaker Note: Or cradle a bottle of wine under your desk after another failed deploy?
9. Speaker Note: Or cradle a bottle of wine under your desk after another failed deploy?
10. Speaker Note: Do you find your infrastructure has stability issues?
11. Speaker Note: Or that automation you’ve written behaves in surprising ways? If so, you’re probably experiencing untested infrastructure.
12. Speaker Note: If you happen to talk to your friendly neighbourhood DevOps Unicorn….
13. Speaker Note: They might tell you about Test Driven Infrastructure. But what is Test Driven Infrastructure (TDI)?
14. Refactor Green Red Speaker Note: TDI comes from a process in Software Development known as Test Driven Development (TDD). This is a popular technique that has been shown to lead to higher quality code, that’s more stable and easier to maintain.
15. Speaker Note: If you listen to those.. interesting.. DevOps unicorns, you might expect to experience….
16. Speaker Note: Extreme confidence in your automation.
17. Speaker Note: Better infrastructure stability.
18. Speaker Note: And general operations and developer happiness. Great, but what does it take to get here?
19. Speaker Note: It doesn’t come free. Getting to TDI takes a number of tools.
20. Speaker Note: To start, we need a Configuration Management system. Doesn’t have to be Chef, that’s just the example here. At the end of the day you could use Bash scripts, but CM will probably works better.
21. Speaker Note: We’ll use a default cookbook generated by Chef DK using the chef generate cookbook command.
22. Speaker Note: In our example we’ll set up Redis. In Chef, using the redisio cookbook from the Supermarket, it might look like this.
23. Speaker Note: The other tool we’ll need is Test Kitchen (TK). TK is going to be our primary testing work horse.
24. Speaker Note: TK is the test runner, using it’s massively pluggable architecture to run tests on any platform, framework, etc.
25. Drivers: docker, Vagrant, AWS … Speaker Note: TK is the test runner, using it’s massively pluggable architecture to run tests on any platform, framework, etc.
26. Drivers: docker, Vagrant, AWS … Communication: ssh, winrm, … Speaker Note: TK is the test runner, using it’s massively pluggable architecture to run tests on any platform, framework, etc.
27. Drivers: docker, Vagrant, AWS … Communication: ssh, winrm, … Provisioners: Chef, Ansible, Puppet … Speaker Note: TK is the test runner, using it’s massively pluggable architecture to run tests on any platform, framework, etc.
28. Drivers: docker, Vagrant, AWS … Communication: ssh, winrm, … Provisioners: Chef, Ansible, Puppet … Testing: InSpec, Pester, BATS… Speaker Note: TK is the test runner, using it’s massively pluggable architecture to run tests on any platform, framework, etc.
29. Drivers: docker, Vagrant, AWS … Communication: ssh, winrm, … Provisioners: Chef, Ansible, Puppet … Testing: InSpec, Pester, BATS… Platform: CentOS, Ubuntu, Windows … Speaker Note: TK is the test runner, using it’s massively pluggable architecture to run tests on any platform, framework, etc.
30. kitchen converge kitchen verify kitchen login Speaker Note: The three commands we’ll be looking at are kitchen login/converge/verify. Login lets you poke around the server TK starts up. Converge will execute the provisioner against the server to put it into the desired state. Finally, verify will run all the tests inside the server.
31. Speaker Note: Speaking of tests, this is where InSpec comes in. InSpec is an extension of RSpec, a Ruby BDD testing library. It specifically focuses on server testing. InSpec uses the underlying OS commands to verify the state of the system.
32. Speaker Note: Speaking of tests, this is where InSpec comes in. InSpec is an extension of RSpec, a Ruby BDD testing library. It specifically focuses on server testing. InSpec uses the underlying OS commands to verify the state of the system.
33. Speaker Note: This is an example of how to test whether a system is listening on a specific port. InSpec will then use the underlying netstat command to check if the port is being listened to. You can also make sure it’s NOT listening on specific ranges ports.
34. Speaker Note: This is how to check if a service exists or is enabled. InSpec will use the proper OS level check, like chkconfig on CentOS.
35. Speaker Note: You can use it to check if a user exists. In this case it’ll use id on Linux OSes.
36. Speaker Note: There are many more resources, but InSpec offers the command resource which provides the ultimate flexibility. You can execute any command and then inspect its standard out, standard error and exit status.
37. Speaker Note: Docker is perfect for infrastructure testing. You want to spin up a server, very quickly, provision it and then tear it down. In addition, the biggest benefit is the portability.
38. Speaker Note: We can leverage Test Kitchen’s plugin ecosystem to use Docker by depending on the Kitchen Dokken plugin that ships with ChefDK. In fact, this is the officially supported way to run Chef Kitchen Tests in Docker and it’s easy to see why. It optimizes for performance by using the official Chef Infra Client Docker images, has caching built in, and does a lot of the heavy lifting for infra handing init processes.
39. 0 35s 1m 10s 1m 45s 2m 20s Vagrant Docker @amaltson Speaker Note: Now a days on modern infrastructure the performance difference isn’t as drastic. The bigger difference come in the portability of running in Docker and getting the same experience locally and on CI builds.
40. 0 35s 1m 10s 1m 45s 2m 20s Vagrant Docker @amaltson Speaker Note: Now a days on modern infrastructure the performance difference isn’t as drastic. The bigger difference come in the portability of running in Docker and getting the same experience locally and on CI builds.
41. 0 35s 1m 10s 1m 45s 2m 20s Vagrant Docker @amaltson Speaker Note: Now a days on modern infrastructure the performance difference isn’t as drastic. The bigger difference come in the portability of running in Docker and getting the same experience locally and on CI builds.
42. Speaker Note: With Chef, Test Kitchen, InSpec and Docker in our tool belt, we put on our safety goggles and ask “what does the process look like?”
43. Speaker Note: To talk about the TDI process, we first need to discuss the TDD process. In TDD you first write the failing test, then you write the code to make it pass, and especially in software development, you refactor. You can safely refactor your code because you have the tests to back you up. I’m not religious about the order, as long as you write the tests close to the code under test.
44. Red Speaker Note: To talk about the TDI process, we first need to discuss the TDD process. In TDD you first write the failing test, then you write the code to make it pass, and especially in software development, you refactor. You can safely refactor your code because you have the tests to back you up. I’m not religious about the order, as long as you write the tests close to the code under test.
45. Green Red Speaker Note: To talk about the TDI process, we first need to discuss the TDD process. In TDD you first write the failing test, then you write the code to make it pass, and especially in software development, you refactor. You can safely refactor your code because you have the tests to back you up. I’m not religious about the order, as long as you write the tests close to the code under test.
46. Refactor Green Red Speaker Note: To talk about the TDI process, we first need to discuss the TDD process. In TDD you first write the failing test, then you write the code to make it pass, and especially in software development, you refactor. You can safely refactor your code because you have the tests to back you up. I’m not religious about the order, as long as you write the tests close to the code under test.
47. Speaker Note: The approach for TDI is very similar. You write a failing InSpec test, you make it pass with a Chef recipe/Ansible playbook/Puppet manifest, and then you refactor if necessary. You won’t refactor as often because the code is generally simpler. However, if you’re depending on an open source cookbook, like redisio, and sometime down the road you decide to write your own Redis cookbook, you have the tests to back you up.
48. InSpec Speaker Note: The approach for TDI is very similar. You write a failing InSpec test, you make it pass with a Chef recipe/Ansible playbook/Puppet manifest, and then you refactor if necessary. You won’t refactor as often because the code is generally simpler. However, if you’re depending on an open source cookbook, like redisio, and sometime down the road you decide to write your own Redis cookbook, you have the tests to back you up.
49. InSpec Recipe Speaker Note: The approach for TDI is very similar. You write a failing InSpec test, you make it pass with a Chef recipe/Ansible playbook/Puppet manifest, and then you refactor if necessary. You won’t refactor as often because the code is generally simpler. However, if you’re depending on an open source cookbook, like redisio, and sometime down the road you decide to write your own Redis cookbook, you have the tests to back you up.
50. Refactor InSpec Recipe Speaker Note: The approach for TDI is very similar. You write a failing InSpec test, you make it pass with a Chef recipe/Ansible playbook/Puppet manifest, and then you refactor if necessary. You won’t refactor as often because the code is generally simpler. However, if you’re depending on an open source cookbook, like redisio, and sometime down the road you decide to write your own Redis cookbook, you have the tests to back you up.
51. Speaker Note: Enough slides, let’s see Test Driven Infrastructure in action. We’ll get Redis installed practicing TDI.
52. Speaker Note: We start with an empty recipe, and an empty CentOS Docker container.
53. Speaker Note: And also the default generated InSpec test.
54. Speaker Note: When installing Redis, we want to make sure Redis runs under it’s own user as is standard practice in Linux. We also want make sure Redis stores it’s database in the default /var/lib/redis directory. We write out these tests in InSpec. Make it fail.
55. Speaker Note: We need to add the redisio cookbook as a dependency to our Policyfile.rb and then run `chef update` to pull it from Supermarket.
56. Speaker Note: This is our default metadata.rb, no dependencies yet.
57. Speaker Note: We then add a dependency on the redisio Supermarket cookbook. This will help us set up Redis.
58. Speaker Note: And finally, Chef specific, we include the default recipe to get Redis installed. Now our test passes.
59. Speaker Note:The tests passed, but we don’t know if Redis is actually started and running. Let’s write some failing tests that make sure Redis is listening on the default 6379 port and has a service to ensure Redis starts back up on reboots. Redisio names the service with the port number.
60. Speaker Note: To make the test pass, we just need to include the ‘enable’ recipe.
61. Speaker Note: Redis being a database, you want to be able to store and retrieve data from it. Using the command resource in InSpec, we can call any command on the OS. We can use the redis-cli command to put data into Redis and get data out. Let’s write those tests. They pass.
62. Speaker Note: Finally, we discussed how Test Kitchen has support for multiple platforms. Let’s say tomorrow your CIO comes down and says “there’s this great new Linux server out there called Ubuntu Server, we should use it.” Fortunately, since we’ve followed a Test Drive Infrastructure approach, we just need to add the Ubuntu platform (in this case an offline cache version), run kitchen converge and then kitchen verify to see a passing build of Redis installed and working on Ubuntu. This is kind of a “refactor” to add Ubuntu support.
63. Refactor InSpec Recipe Speaker Note: We now saw the full virtuous cycle of writing a failing ServerSpec test, writing the recipe/playbook/etc to make that test pass, and then even “refactor” by changing the platform we support.
64. Speaker Note: Of course nothing has all upsides, there are some downsides with testing too. If you’re cookbook does too much, your feedback cycle can get really long. If the tests you execute take a long time to return, and you do a lot of them, that also increases the feedback cycle. Testing also adds more process, so shipping takes longer. Just like with software development 10 years ago, there was questions on “why would I test, I never did that before”. 10 years later, we’re terrified to touch code that doesn’t have tests. My recommendation would be, keep your cookbooks/playbooks/etc small and focused. You should really only need 30-100 tests.
65. Speaker Note: At the end of the day, the tradeoffs and gotchas are well worth it. It’s all about safety and confidence in making changes to your infrastructure. We’ve found tests catch a range of issues, like wrong configurations for websites, before it ever hits the development or production environment. The end goal is to move fast and continuous deliver.
66. Speaker Note: continuously deliver value, not downtime. This whole time we’ve been talking about development on our local workstation…
67. Speaker Note: This is where Continuous Integration (CI) comes into play. With something like CircleCI or Jenkins, you get a central place that verifies the tests continue passing. With Docker, running these TK tests in CI is really easy.
68. Speaker Note: This is where Continuous Integration (CI) comes into play. With something like CircleCI or Jenkins, you get a central place that verifies the tests continue passing. With Docker, running these TK tests in CI is really easy.
69. Speaker Note: What does the full workflow look like? You follow the TDI cycle locally, commit and push to your central repo, that triggers a build, which fires up a Docker image and runs Test Kitchen and InSpec tests. If build fails, you deal with it on your workstation and continue the cycle again.
70. Refactor Recipe ServerSpec Speaker Note: What does the full workflow look like? You follow the TDI cycle locally, commit and push to your central repo, that triggers a build, which fires up a Docker image and runs Test Kitchen and InSpec tests. If build fails, you deal with it on your workstation and continue the cycle again.
71. git push Refactor Recipe ServerSpec Speaker Note: What does the full workflow look like? You follow the TDI cycle locally, commit and push to your central repo, that triggers a build, which fires up a Docker image and runs Test Kitchen and InSpec tests. If build fails, you deal with it on your workstation and continue the cycle again.
72. git push Refactor Recipe ServerSpec Speaker Note: What does the full workflow look like? You follow the TDI cycle locally, commit and push to your central repo, that triggers a build, which fires up a Docker image and runs Test Kitchen and InSpec tests. If build fails, you deal with it on your workstation and continue the cycle again.
73. git push Refactor Recipe ServerSpec triggers Speaker Note: What does the full workflow look like? You follow the TDI cycle locally, commit and push to your central repo, that triggers a build, which fires up a Docker image and runs Test Kitchen and InSpec tests. If build fails, you deal with it on your workstation and continue the cycle again.
74. git push Refactor Recipe ServerSpec triggers Speaker Note: What does the full workflow look like? You follow the TDI cycle locally, commit and push to your central repo, that triggers a build, which fires up a Docker image and runs Test Kitchen and InSpec tests. If build fails, you deal with it on your workstation and continue the cycle again.
75. git push Refactor Recipe ServerSpec triggers build Speaker Note: What does the full workflow look like? You follow the TDI cycle locally, commit and push to your central repo, that triggers a build, which fires up a Docker image and runs Test Kitchen and InSpec tests. If build fails, you deal with it on your workstation and continue the cycle again.
76. git push Refactor Recipe ServerSpec triggers build Speaker Note: What does the full workflow look like? You follow the TDI cycle locally, commit and push to your central repo, that triggers a build, which fires up a Docker image and runs Test Kitchen and InSpec tests. If build fails, you deal with it on your workstation and continue the cycle again.
77. git push Refactor Recipe ServerSpec triggers build ❌ Speaker Note: What does the full workflow look like? You follow the TDI cycle locally, commit and push to your central repo, that triggers a build, which fires up a Docker image and runs Test Kitchen and InSpec tests. If build fails, you deal with it on your workstation and continue the cycle again.
78. triggers build ✅ Speaker Note: But what happens when the build passes? This is where you can use the Continuous Delivery features available in your CI system, most of them have something built in (eg Bamboo). A successful CI build can automatically trigger a deployment to development. With enough comfort level, you could even trigger an automatic deployment to production.
79. triggers build ✅ deploy Speaker Note: But what happens when the build passes? This is where you can use the Continuous Delivery features available in your CI system, most of them have something built in (eg Bamboo). A successful CI build can automatically trigger a deployment to development. With enough comfort level, you could even trigger an automatic deployment to production.
80. triggers build ✅ deploy Speaker Note: But what happens when the build passes? This is where you can use the Continuous Delivery features available in your CI system, most of them have something built in (eg Bamboo). A successful CI build can automatically trigger a deployment to development. With enough comfort level, you could even trigger an automatic deployment to production.
81. triggers build ✅ deploy ✅ Speaker Note: But what happens when the build passes? This is where you can use the Continuous Delivery features available in your CI system, most of them have something built in (eg Bamboo). A successful CI build can automatically trigger a deployment to development. With enough comfort level, you could even trigger an automatic deployment to production.
82. triggers build ✅ deploy deploy ✅ Speaker Note: But what happens when the build passes? This is where you can use the Continuous Delivery features available in your CI system, most of them have something built in (eg Bamboo). A successful CI build can automatically trigger a deployment to development. With enough comfort level, you could even trigger an automatic deployment to production.
83. triggers build ✅ deploy deploy ✅ Speaker Note: But what happens when the build passes? This is where you can use the Continuous Delivery features available in your CI system, most of them have something built in (eg Bamboo). A successful CI build can automatically trigger a deployment to development. With enough comfort level, you could even trigger an automatic deployment to production.
84. triggers build ✅ deploy deploy ✅ ✅ Speaker Note: But what happens when the build passes? This is where you can use the Continuous Delivery features available in your CI system, most of them have something built in (eg Bamboo). A successful CI build can automatically trigger a deployment to development. With enough comfort level, you could even trigger an automatic deployment to production.
85. Speaker Note: What’s the path ahead?
86. Speaker Note: Another interesting use of TK is multi-server testing. You can have TK spin up several nodes, and have them all talk to each other on a private local network. We’ve had success testing Redis primary, replica and sentinel configurations as well as testing the entire ELK stack.
87. Speaker Note: But ultimately you’ll have to run this Redis server somewhere, and most likely you’re going to do it in the Cloud.
88. Speaker Note: But is it possible to do TDI against Cloud resources???
89. Speaker Note: It is, because InSpec supports AWS (and Azure) resources out of the box. You can check on AWS EC2 instances.
90. Speaker Note: And ELBs that point at those EC2 instances.
91. Speaker Note: And even S3 buckets. You can make sure your buckets are never publicly exposed!
92. Speaker Note: If you use the popular Terraform tool to create that infrastructure, you can use the awesome kitchen-terraform plugin to tie this all together!
93. Speaker Note: Mind… blown!
94. Speaker Note: Mind… blown!
95. Speaker Note: If you take one thing away, please test your infrastructure and be a super hero.
97. Arthur Maltson @amaltson maltson.com Capital One Distinguished Engineer 70% Dev, 30% Ops Full Time DadOps @amaltson
98. • Slides: https://bit.ly/tdi-chefconf-2020-slides • test-driven-redis, start to end commits: https://bit.ly/tdi-chefconf-2020-code • Test Kitchen: https://kitchen.ci • InSpec: https://www.inspec.io • Kitchen Dokken: https://github.com/test-kitchen/kitchen-dokken • Kitchen Terraform: https://github.com/newcontext-oss/kitchen-terraform @amaltson Arthur Maltson
99. Credits • Slide 1 - Riccardo Cuppini, Zen [Explored], https://flic.kr/p/5ehoTC • Slide 2 - CollegeDegrees360, Computer Problems, https://flic.kr/p/cEJpCY • Slide 3 - Greg Heo, Big banks, https://flic.kr/p/dfb13h • Slide 4 - Heisenberg Media, Berlin Startup Tour, https://flic.kr/p/dP6W49 • Slide 6 - Will Humes Follow, crossed fingers, https://flic.kr/p/4s5kZ5 • Slide 7 - Crying Under the Table With a Bottle of Wine GIF, https://mashable.com/2013/08/20/gif-origins/#3PUHZ0bAVPqj • Slide 10, 13 - Matthew Frederickson, Unicorns, https://flic.kr/p/5jrvmr • Slide 11, yosuke muroya, Unicorn, https://flic.kr/p/bpQFTw • Slide 14, 59, Chris & Karen Highland, consumer confidence!, https://flic.kr/p/qKcmR2 • Slide 15 - Quentin Meulepas, Whistler: Inukshuk, https://flic.kr/p/6izmiv • Slide 16 - Moyan Brenn, Happiness, https://flic.kr/p/nMmBGs • Slide 17, 32 - Bre Pettis, Dave’s Bike Tools, https://flic.kr/p/QMVMw • Slide 31 - F Delventhal, Safety First, https://flic.kr/p/EmGgn • Slide 34 - MsSaraKelly, Take one: Sarah's hen do, https://flic.kr/p/fsKWAi • Slide 47 - Simon Harrod, Strawberry Snail, https://flic.kr/p/9XkFkY • Slide 49 - GotCredit, Safety, https://flic.kr/p/qHCmfo • Slide 51 - Lawrence Whittemore, basement.jpg, https://flic.kr/p/c84PL • Slide 53 - Joseph Thornton, 2013 Retina Macbook Pro, https://flic.kr/p/eu3G38 • Slide 54, 57 - DeclanTM, Home Server, https://flic.kr/p/4PGBb5 • Slide 54, 57 - Quinn Dombrowski, Servers, https://flic.kr/p/cqqwcb • Slide 55 - Matthew Faltz, The Path, https://flic.kr/p/pA7dZQ • Slide 58 - tribp, Grapes, https://flic.kr/p/dcZUgY • Slide 60 - Nate Grigg, Thank You, https://flic.kr/p/6K41qv • Slide 62 - woodleywonderworks, my son's teacher is a supermodel, https://flic.kr/p/auPuAq • Slide 63 - Harry (Howard) Potts Follow, Zulu pensioners - drums, https://flic.kr/p/81TmRT