Test Driven Infrastructure

Test Driven Development is a popular concept in Software Development, leading to higher quality code that's easier to maintain. Automated testing is normally a foreign concept in the Operations world, but as you ssh into your servers to make that quick fix or run your updated script (fingers crossed), you might be wondering if there's a better way. A way that gives you the confidence in your script and lets you test those scripts in isolation. Well I have good news for you, there is a better way! Test Driven Infrastructure (TDI) is now possible. I know, it sounds crazy.

At this session you'll learn the how, and more importantly the why, of TDI. You'll see how Chef (or any other Config Management framework) can be tested with Test Kitchen and ServerSpec. You'll also learn how to improve your feedback cycle with Docker, and using the Docker approach on a CI server. There may even be some live demos!

Finally, the Ops world collides with the Dev world in true DevOps testing bliss.

This presentation was given at a number of conferences:

- All Day DevOps 2020
- Chef Conf 2020
- Web Unleashed 2018
- Full Stack TO 2015
- DevOps Days 2014

It included a live coding section which is represented by code slides.

1. Test Driven Infrastructure Bring Tranquility To Your Infrastructure Arthur Maltson @amaltson
2. Speaker Note: I’ll be talking about a problem that plagues….
3. Speaker Note: Both large companies like big financials.
4. Speaker Note: And small, like the hip startups.
5. Speaker Note: That problem is untested infrastructure. If you’re experiencing this, what might be the symptoms?
6. Speaker Note: Do you cross your fingers before running a script in production?
7. Speaker Note: Or cradle a bottle of wine under your desk after another failed deploy?
8. Speaker Note: Or cradle a bottle of wine under your desk after another failed deploy?
9. Speaker Note: Do you find your infrastructure has stability issues?
10. Speaker Note: Or that automation you’ve written behaves in surprising ways? If so, you’re probably experiencing untested infrastructure.
11. Speaker Note: If you happen to talk to your friendly neighbourhood DevOps Unicorn….
12. Speaker Note: They might tell you about Test Driven Infrastructure. But what is Test Driven Infrastructure (TDI)?
13. Refactor Green Red Speaker Note: TDI comes from a process in Software Development known as Test Driven Development (TDD). This is a popular technique that has been shown to lead to higher quality code, that’s more stable and easier to maintain.
14. Speaker Note: If you listen to those.. interesting.. DevOps unicorns, you might expect to experience….
15. Speaker Note: Extreme confidence in your automation.
16. Speaker Note: Better infrastructure stability.
17. Speaker Note: And general operations and developer happiness. Great, but what does it take to get here?
18. Speaker Note: It doesn’t come free. Getting to TDI takes a number of tools.
19. Speaker Note: To start, we need a Configuration Management system. Doesn’t have to be Chef, that’s just the example here. At the end of the day you could use Bash scripts, but CM will probably works better.
20. Speaker Note: We’ll use a default cookbook generated by Chef DK using the chef generate cookbook command.
21. Speaker Note: In our example we’ll set up Redis. In Chef, using the redisio cookbook from the Supermarket, it might look like this.
22. Speaker Note: The other tool we’ll need is Test Kitchen (TK). TK is going to be our primary testing work horse.
23. Speaker Note: TK is the test runner, using it’s massively pluggable architecture to run tests on any platform, framework, etc.
24. Drivers: docker, Vagrant, AWS … Speaker Note: TK is the test runner, using it’s massively pluggable architecture to run tests on any platform, framework, etc.
25. Drivers: docker, Vagrant, AWS … Communication: ssh, winrm, … Speaker Note: TK is the test runner, using it’s massively pluggable architecture to run tests on any platform, framework, etc.
26. Drivers: docker, Vagrant, AWS … Communication: ssh, winrm, … Provisioners: Chef, Ansible, Puppet … Speaker Note: TK is the test runner, using it’s massively pluggable architecture to run tests on any platform, framework, etc.
27. Drivers: docker, Vagrant, AWS … Communication: ssh, winrm, … Provisioners: Chef, Ansible, Puppet … Testing: InSpec, Pester, BATS… Speaker Note: TK is the test runner, using it’s massively pluggable architecture to run tests on any platform, framework, etc.
28. Drivers: docker, Vagrant, AWS … Communication: ssh, winrm, … Provisioners: Chef, Ansible, Puppet … Testing: InSpec, Pester, BATS… Platform: CentOS, Ubuntu, Windows … Speaker Note: TK is the test runner, using it’s massively pluggable architecture to run tests on any platform, framework, etc.
29. Speaker Note: The four commands we’ll be looking at are kitchen create/login/converge/ verify. Create will create the VM/container. Login lets you poke around the server TK starts up. Converge will execute the provisioner against the server to put it into the desired state. Finally, verify will run all the tests inside the server.
30. Speaker Note: The four commands we’ll be looking at are kitchen create/login/converge/ verify. Create will create the VM/container. Login lets you poke around the server TK starts up. Converge will execute the provisioner against the server to put it into the desired state. Finally, verify will run all the tests inside the server. kitchen create
31. Speaker Note: The four commands we’ll be looking at are kitchen create/login/converge/ verify. Create will create the VM/container. Login lets you poke around the server TK starts up. Converge will execute the provisioner against the server to put it into the desired state. Finally, verify will run all the tests inside the server. kitchen login kitchen create
32. Speaker Note: The four commands we’ll be looking at are kitchen create/login/converge/ verify. Create will create the VM/container. Login lets you poke around the server TK starts up. Converge will execute the provisioner against the server to put it into the desired state. Finally, verify will run all the tests inside the server. kitchen converge kitchen login kitchen create
33. Speaker Note: The four commands we’ll be looking at are kitchen create/login/converge/ verify. Create will create the VM/container. Login lets you poke around the server TK starts up. Converge will execute the provisioner against the server to put it into the desired state. Finally, verify will run all the tests inside the server. kitchen converge kitchen verify kitchen login kitchen create
34. Speaker Note: Speaking of tests, this is where InSpec comes in. InSpec is an extension of RSpec, a Ruby BDD testing library. It specifically focuses on server testing. InSpec uses the underlying OS commands to verify the state of the system.
35. Speaker Note: Speaking of tests, this is where InSpec comes in. InSpec is an extension of RSpec, a Ruby BDD testing library. It specifically focuses on server testing. InSpec uses the underlying OS commands to verify the state of the system.
36. Speaker Note: This is an example of how to test whether a system is listening on a specific port. InSpec will then use the underlying netstat command to check if the port is being listened to. You can also make sure it’s NOT listening on specific ranges ports.
37. Speaker Note: This is how to check if a service exists or is enabled. InSpec will use the proper OS level check, like chkconfig on CentOS.
38. Speaker Note: You can use it to check if a user exists. In this case it’ll use id on Linux OSes.
39. Speaker Note: There are many more resources, but InSpec offers the command resource which provides the ultimate flexibility. You can execute any command and then inspect its standard out, standard error and exit status.
40. Speaker Note: I’d be remiss if I didn’t mention Docker in a DevOps themed talk. But Docker is perfect for testing. You want to spin up a server, very quickly, provision it and then tear it down.
41. Speaker Note: Using Test Kitchen’s pluggable architecture, we can customize the kitchen.yml file to test against Docker. The easiest path in a Chef world is to use kitchen-dokken, which ships custom Docker images set up to build in Chef and configures SystemD so it looks like a full OS making the test more realistic. @amaltson
42. 0 35s 1m 10s 1m 45s 2m 20s Vagrant Docker @amaltson Speaker Note: Get a huge performance gain. With that small change, we get over 30% performance boost. If we cache the resources offline, we can tighten our feedback cycle from initial boot to converge to verify in under one minute.
43. 0 35s 1m 10s 1m 45s 2m 20s Vagrant Docker @amaltson Speaker Note: Get a huge performance gain. With that small change, we get over 30% performance boost. If we cache the resources offline, we can tighten our feedback cycle from initial boot to converge to verify in under one minute.
44. 0 35s 1m 10s 1m 45s 2m 20s Vagrant Docker @amaltson Speaker Note: Get a huge performance gain. With that small change, we get over 30% performance boost. If we cache the resources offline, we can tighten our feedback cycle from initial boot to converge to verify in under one minute.
45. Speaker Note: With Chef, Test Kitchen, InSpec and Docker in our tool belt, we put on our safety goggles and ask “what does the process look like?”
46. Speaker Note: To talk about the TDI process, we first need to discuss the TDD process. In TDD you first write the failing test, then you write the code to make it pass, and especially in software development, you refactor. You can safely refactor your code because you have the tests to back you up. I’m not religious about the order, as long as you write the tests close to the code under test.
47. Red Speaker Note: To talk about the TDI process, we first need to discuss the TDD process. In TDD you first write the failing test, then you write the code to make it pass, and especially in software development, you refactor. You can safely refactor your code because you have the tests to back you up. I’m not religious about the order, as long as you write the tests close to the code under test.
48. Green Red Speaker Note: To talk about the TDI process, we first need to discuss the TDD process. In TDD you first write the failing test, then you write the code to make it pass, and especially in software development, you refactor. You can safely refactor your code because you have the tests to back you up. I’m not religious about the order, as long as you write the tests close to the code under test.
49. Refactor Green Red Speaker Note: To talk about the TDI process, we first need to discuss the TDD process. In TDD you first write the failing test, then you write the code to make it pass, and especially in software development, you refactor. You can safely refactor your code because you have the tests to back you up. I’m not religious about the order, as long as you write the tests close to the code under test.
50. Speaker Note: The approach for TDI is very similar. You write a failing InSpec test, you make it pass with a Chef recipe/Ansible playbook/Puppet manifest, and then you refactor if necessary. You won’t refactor as often because the code is generally simpler. However, if you’re depending on an open source cookbook, like redisio, and sometime down the road you decide to write your own Redis cookbook, you have the tests to back you up.
51. InSpec Speaker Note: The approach for TDI is very similar. You write a failing InSpec test, you make it pass with a Chef recipe/Ansible playbook/Puppet manifest, and then you refactor if necessary. You won’t refactor as often because the code is generally simpler. However, if you’re depending on an open source cookbook, like redisio, and sometime down the road you decide to write your own Redis cookbook, you have the tests to back you up.
52. InSpec Recipe Speaker Note: The approach for TDI is very similar. You write a failing InSpec test, you make it pass with a Chef recipe/Ansible playbook/Puppet manifest, and then you refactor if necessary. You won’t refactor as often because the code is generally simpler. However, if you’re depending on an open source cookbook, like redisio, and sometime down the road you decide to write your own Redis cookbook, you have the tests to back you up.
53. Refactor InSpec Recipe Speaker Note: The approach for TDI is very similar. You write a failing InSpec test, you make it pass with a Chef recipe/Ansible playbook/Puppet manifest, and then you refactor if necessary. You won’t refactor as often because the code is generally simpler. However, if you’re depending on an open source cookbook, like redisio, and sometime down the road you decide to write your own Redis cookbook, you have the tests to back you up.
54. Speaker Note: Enough slides, let’s see Test Driven Infrastructure in action. We’ll get Redis installed practicing TDI.
55. Speaker Note: We start with an empty recipe, and an empty Dokken Docker container.
56. Speaker Note: And also the default generated InSpec test.
57. Speaker Note: When installing Redis, we want to make sure Redis runs under it’s own user as is standard practice in Linux. We also want make sure Redis stores it’s database in the default /var/lib/redis directory. We write out these tests in InSpec. Make it fail.
58. Speaker Note: We need to add a dependency on the Supermarket redisio cookbook in Chef’s Policyfile.rb and execute the ‘chef update’ command.
59. Speaker Note: We then add a dependency on the redisio cookbook in metadata.rb. This will have our cookbook pull in that dependency.
60. Speaker Note: And finally, Chef specific, we include the default recipe to get Redis installed. Now our test passes.
61. Speaker Note:The tests passed, but we don’t know if Redis is actually started and running. Let’s write some failing tests that make sure Redis is listening on the default 6379 port and has a service to ensure Redis starts back up on reboots. Redisio names the service with the port number.
62. Speaker Note: To make the test pass, we just need to include the ‘enable’ recipe.
63. Speaker Note: Redis being a database, you want to be able to store and retrieve data from it. Using the command resource in InSpec, we can call any command on the OS. We can use the redis-cli command to put data into Redis and get data out. Let’s write those tests. They pass.
64. Speaker Note: Finally, we discussed how Test Kitchen has support for multiple platforms. Let’s say tomorrow your CIO comes down and says “there’s this great new Linux server out there called Ubuntu Server, we should use it.” Fortunately, since we’ve followed a Test Drive Infrastructure approach, we just need to add the Ubuntu platform (in this case an offline cache version), run kitchen converge and then kitchen verify to see a passing build of Redis installed and working on Ubuntu. This is kind of a “refactor” to add Ubuntu support.
65. Refactor InSpec Recipe Speaker Note: We now saw the full virtuous cycle of writing a failing ServerSpec test, writing the recipe/playbook/etc to make that test pass, and then even “refactor” by changing the platform we support.
66. Speaker Note: Of course nothing has all upsides, there are some downsides with testing too. If you’re cookbook does too much, your feedback cycle can get really long. If the tests you execute take a long time to return, and you do a lot of them, that also increases the feedback cycle. Testing also adds more process, so shipping takes longer. Just like with software development 10 years ago, there was questions on “why would I test, I never did that before”. 10 years later, we’re terrified to touch code that doesn’t have tests. My recommendation would be, keep your cookbooks/playbooks/etc small and focused. You should really only need 30-100 tests.
67. Speaker Note: At the end of the day, the tradeoffs and gotchas are well worth it. It’s all about safety and confidence in making changes to your infrastructure. We’ve found tests catch a range of issues, like wrong configurations for websites, before it ever hits the development or production environment. The end goal is to move fast and continuous deliver.
68. Speaker Note: continuously deliver value, not downtime. This whole time we’ve been talking about development on our local workstation…
69. Speaker Note: This is where Continuous Integration (CI) comes into play. With something like CircleCI or Jenkins, you get a central place that verifies the tests continue passing. With Docker, running these TK tests in CI is really easy.
70. Speaker Note: This is where Continuous Integration (CI) comes into play. With something like CircleCI or Jenkins, you get a central place that verifies the tests continue passing. With Docker, running these TK tests in CI is really easy.
71. Speaker Note: What does the full workflow look like? You follow the TDI cycle locally, commit and push to your central repo, that triggers a build, which fires up a Docker image and runs Test Kitchen and InSpec tests. If build fails, you deal with it on your workstation and continue the cycle again.
72. Refactor Recipe ServerSpec Speaker Note: What does the full workflow look like? You follow the TDI cycle locally, commit and push to your central repo, that triggers a build, which fires up a Docker image and runs Test Kitchen and InSpec tests. If build fails, you deal with it on your workstation and continue the cycle again.
73. git push Refactor Recipe ServerSpec Speaker Note: What does the full workflow look like? You follow the TDI cycle locally, commit and push to your central repo, that triggers a build, which fires up a Docker image and runs Test Kitchen and InSpec tests. If build fails, you deal with it on your workstation and continue the cycle again.
74. git push Refactor Recipe ServerSpec Speaker Note: What does the full workflow look like? You follow the TDI cycle locally, commit and push to your central repo, that triggers a build, which fires up a Docker image and runs Test Kitchen and InSpec tests. If build fails, you deal with it on your workstation and continue the cycle again.
75. git push Refactor Recipe ServerSpec triggers Speaker Note: What does the full workflow look like? You follow the TDI cycle locally, commit and push to your central repo, that triggers a build, which fires up a Docker image and runs Test Kitchen and InSpec tests. If build fails, you deal with it on your workstation and continue the cycle again.
76. git push Refactor Recipe ServerSpec triggers Speaker Note: What does the full workflow look like? You follow the TDI cycle locally, commit and push to your central repo, that triggers a build, which fires up a Docker image and runs Test Kitchen and InSpec tests. If build fails, you deal with it on your workstation and continue the cycle again.
77. git push Refactor Recipe ServerSpec triggers build Speaker Note: What does the full workflow look like? You follow the TDI cycle locally, commit and push to your central repo, that triggers a build, which fires up a Docker image and runs Test Kitchen and InSpec tests. If build fails, you deal with it on your workstation and continue the cycle again.
78. git push Refactor Recipe ServerSpec triggers build Speaker Note: What does the full workflow look like? You follow the TDI cycle locally, commit and push to your central repo, that triggers a build, which fires up a Docker image and runs Test Kitchen and InSpec tests. If build fails, you deal with it on your workstation and continue the cycle again.
79. git push Refactor Recipe ServerSpec triggers build ❌ Speaker Note: What does the full workflow look like? You follow the TDI cycle locally, commit and push to your central repo, that triggers a build, which fires up a Docker image and runs Test Kitchen and InSpec tests. If build fails, you deal with it on your workstation and continue the cycle again.
80. triggers build ✅ Speaker Note: But what happens when the build passes? This is where you can use the Continuous Delivery features available in your CI system, most of them have something built in (eg Bamboo). A successful CI build can automatically trigger a deployment to development. With enough comfort level, you could even trigger an automatic deployment to production.
81. triggers build ✅ deploy Speaker Note: But what happens when the build passes? This is where you can use the Continuous Delivery features available in your CI system, most of them have something built in (eg Bamboo). A successful CI build can automatically trigger a deployment to development. With enough comfort level, you could even trigger an automatic deployment to production.
82. triggers build ✅ deploy Speaker Note: But what happens when the build passes? This is where you can use the Continuous Delivery features available in your CI system, most of them have something built in (eg Bamboo). A successful CI build can automatically trigger a deployment to development. With enough comfort level, you could even trigger an automatic deployment to production.
83. triggers build ✅ deploy ✅ Speaker Note: But what happens when the build passes? This is where you can use the Continuous Delivery features available in your CI system, most of them have something built in (eg Bamboo). A successful CI build can automatically trigger a deployment to development. With enough comfort level, you could even trigger an automatic deployment to production.
84. triggers build ✅ deploy deploy ✅ Speaker Note: But what happens when the build passes? This is where you can use the Continuous Delivery features available in your CI system, most of them have something built in (eg Bamboo). A successful CI build can automatically trigger a deployment to development. With enough comfort level, you could even trigger an automatic deployment to production.
85. triggers build ✅ deploy deploy ✅ Speaker Note: But what happens when the build passes? This is where you can use the Continuous Delivery features available in your CI system, most of them have something built in (eg Bamboo). A successful CI build can automatically trigger a deployment to development. With enough comfort level, you could even trigger an automatic deployment to production.
86. triggers build ✅ deploy deploy ✅ ✅ Speaker Note: But what happens when the build passes? This is where you can use the Continuous Delivery features available in your CI system, most of them have something built in (eg Bamboo). A successful CI build can automatically trigger a deployment to development. With enough comfort level, you could even trigger an automatic deployment to production.
87. Speaker Note: What’s the path ahead?
88. Speaker Note: Another interesting use of TK is multi-server testing. You can have TK spin up several nodes, and have them all talk to each other on a private local network. We’ve had success testing Redis primary, replica and sentinel configurations as well as testing the entire ELK stack.
89. Speaker Note: But ultimately you’ll have to run this Redis server somewhere, and most likely you’re going to do it in the Cloud.
90. Speaker Note: But is it possible to do TDI against Cloud resources???
91. Speaker Note: It is, because InSpec supports AWS (and Azure) resources out of the box. You can check on AWS EC2 instances.
92. Speaker Note: And ELBs that point at those EC2 instances.
93. Speaker Note: And even S3 buckets. You can make sure your buckets are never publicly exposed!
94. Speaker Note: If you use the popular Terraform tool to create that infrastructure, you can use the awesome kitchen-terraform plugin to tie this all together!
95. Speaker Note: Mind… blown!
96. Speaker Note: Mind… blown!
97. @amaltson Speaker Note: And if you think that’s mind blowing, you can take it to the next level by building Test Kitchen plugins.
98. @amaltson Speaker Note: Remember the various concepts in Test Kitchen? They’re all extensible by inheriting from Test Kitchen’s base classes and packaging as a gem.
99. @amaltson Drivers: custom lifecycle Speaker Note: Remember the various concepts in Test Kitchen? They’re all extensible by inheriting from Test Kitchen’s base classes and packaging as a gem.
100. @amaltson Drivers: custom lifecycle Communication: custom remote login Speaker Note: Remember the various concepts in Test Kitchen? They’re all extensible by inheriting from Test Kitchen’s base classes and packaging as a gem.
101. @amaltson Drivers: custom lifecycle Communication: custom remote login Provisioners: custom provisioning Speaker Note: Remember the various concepts in Test Kitchen? They’re all extensible by inheriting from Test Kitchen’s base classes and packaging as a gem.
102. @amaltson Drivers: custom lifecycle Communication: custom remote login Provisioners: custom provisioning Testing: custom verification Speaker Note: Remember the various concepts in Test Kitchen? They’re all extensible by inheriting from Test Kitchen’s base classes and packaging as a gem.
103. @amaltson Drivers: custom lifecycle Communication: custom remote login Provisioners: custom provisioning Testing: custom verification Platform: custom deployment platforms Speaker Note: Remember the various concepts in Test Kitchen? They’re all extensible by inheriting from Test Kitchen’s base classes and packaging as a gem.
104. Speaker Note: If you take one thing away, please test your infrastructure and be a super hero.
106. Arthur Maltson @amaltson maltson.com Capital One Distinguished Engineer 70% Dev, 30% Ops Full Time DadOps @amaltson
107. • Slides: https://speakerdeck.com/amaltson/test-driven-infrastructure • test-driven-redis: https://github.com/amaltson/test-driven-redis • Test Kitchen: https://kitchen.ci • InSpec: https://www.inspec.io • Kitchen Dokken: https://github.com/test-kitchen/kitchen-dokken • Kitchen Terraform: https://github.com/newcontext-oss/kitchen-terraform @amaltson Arthur Maltson
108. Credits • Riccardo Cuppini, Zen [Explored], https://flic.kr/p/5ehoTC • CollegeDegrees360, Computer Problems, https://flic.kr/p/cEJpCY • Greg Heo, Big banks, https://flic.kr/p/dfb13h • Heisenberg Media, Berlin Startup Tour, https://flic.kr/p/dP6W49 • Will Humes Follow, crossed fingers, https://flic.kr/p/4s5kZ5 • Crying Under the Table With a Bottle of Wine GIF, https://mashable.com/2013/08/20/gif-origins/#3PUHZ0bAVPqj • Matthew Frederickson, Unicorns, https://flic.kr/p/5jrvmr • yosuke muroya, Unicorn, https://flic.kr/p/bpQFTw • Chris & Karen Highland, consumer confidence!, https://flic.kr/p/qKcmR2 • Quentin Meulepas, Whistler: Inukshuk, https://flic.kr/p/6izmiv • Moyan Brenn, Happiness, https://flic.kr/p/nMmBGs • Bre Pettis, Dave’s Bike Tools, https://flic.kr/p/QMVMw • F Delventhal, Safety First, https://flic.kr/p/EmGgn • MsSaraKelly, Take one: Sarah's hen do, https://flic.kr/p/fsKWAi • Simon Harrod, Strawberry Snail, https://flic.kr/p/9XkFkY • GotCredit, Safety, https://flic.kr/p/qHCmfo • Lawrence Whittemore, basement.jpg, https://flic.kr/p/c84PL • Joseph Thornton, 2013 Retina Macbook Pro, https://flic.kr/p/eu3G38 • DeclanTM, Home Server, https://flic.kr/p/4PGBb5 • Quinn Dombrowski, Servers, https://flic.kr/p/cqqwcb • Matthew Faltz, The Path, https://flic.kr/p/pA7dZQ • Anita Sollars, Niche Chat, https://www.pinterest.ca/pin/138063544803937259 • tribp, Grapes, https://flic.kr/p/dcZUgY • Nate Grigg, Thank You, https://flic.kr/p/6K41qv