Chef, Test Kitchen, Docker, CI... Oh My!

This presentation was given at DevOps Days Toronto 2015. The video is available here: https://youtu.be/IEQUfo0eUiI?t=241

Test Driven Development is a popular concept in Software Development, leading to higher quality code that's easier to maintain. Automated testing is normally a foreign concept in the Operations world, but as you ssh into your servers to make that quick fix or run your updated script (fingers crossed), you might be wondering if there's a better way. A way that gives you the confidence in your script and lets you test those scripts in isolation. Well I have good news for you, there is a better way! Test Driven Infrastructure (TDI) is now possible. I know, it sounds crazy.

At this session you'll learn the how, and more importantly the why, of TDI. You'll see how Chef (or any other Config Management framework) can be tested with Test Kitchen and ServerSpec. You'll also learn how to improve your feedback cycle with Docker, and using the Docker approach on a CI server. There may even be some live demos!

Finally, the Ops world collides with the Dev world in true DevOps testing bliss.

Transcript

1. Chef, Test Kitchen, Docker, CI… Oh My! Bring Tranquility To Your Infrastructure Arthur Maltson @amaltson
2. Arthur Maltson ! @amaltson http://onthejvm.com ! ! ! ! Software Developer ! Ops Curious ! Full Time DevOp
3. Speaker Note: I work at Ontario Teachers’
4. Speaker Note: Pension Plan
5. Speaker Note: And I’ll be talking about a problem that plagues….
6. Speaker Note: Both large companies like big financials.
7. Speaker Note: And small, like the hip startups in Liberty Village.
8. Speaker Note: That problem is untested infrastructure. If you’re experiencing this, what might be the symptoms?
9. Speaker Note: Do you cross your fingers before running a script in production?
10. Speaker Note: Or get paged late at night?
11. Speaker Note: Do you find your infrastructure has stability issues?
12. Speaker Note: Or that automation you’ve written behaves in surprising ways? If so, you’re probably experiencing untested infrastructure.
13. Speaker Note: If you happen to talk to your friendly neighbourhood DevOps Unicorn….
14. Speaker Note: They might tell you about Test Driven Infrastructure. But what is Test Driven Infrastructure (TDI)?
15. Refactor Green Red Speaker Note: TDI comes from a process in Software Development called Test Driven Development (TDD). This is a popular technic that has been shown to lead to higher quality code, that’s more stable and easier to maintain.
16. Speaker Note: If you listen to those.. interesting.. DevOps unicorns, you might expect to experience….
17. Speaker Note: Extreme confidence in your automation.
18. Speaker Note: Better infrastructure stability.
19. Speaker Note: And generation operations and developer happiness. Great, but what does it take to get here?
20. Speaker Note: It doesn’t come free. Getting to TDI takes a number of tools.
21. Speaker Note: To start, we need a Configuration Management system. Doesn’t have to be Chef, that’s just the example here. At the end of the day you could use Bash scripts, but CM will probably works better.
22. metadata.rb recipe/default.rb Speaker Note: In our example we’ll set up Redis. In Chef, using the redisio cookbook from the Supermarket, it might look like this.
23. Speaker Note: The other tool we’ll need is Test Kitchen (TK). TK is going to be our primary testing work horse.
24. Speaker Note: TK is the test runner, using it’s massively pluggable architecture to run tests on any platform, framework, etc.
25. Drivers: docker, Vagrant, AWS … Speaker Note: TK is the test runner, using it’s massively pluggable architecture to run tests on any platform, framework, etc.
26. Drivers: docker, Vagrant, AWS … Communication: ssh, winrm, … Speaker Note: TK is the test runner, using it’s massively pluggable architecture to run tests on any platform, framework, etc.
27. Drivers: docker, Vagrant, AWS … Communication: ssh, winrm, … Provisioners: Chef, Puppet, Ansible … Speaker Note: TK is the test runner, using it’s massively pluggable architecture to run tests on any platform, framework, etc.
28. Drivers: docker, Vagrant, AWS … Communication: ssh, winrm, … Provisioners: Chef, Puppet, Ansible … Testing: ServerSpec, Pester, BATS… Speaker Note: TK is the test runner, using it’s massively pluggable architecture to run tests on any platform, framework, etc.
29. Drivers: docker, Vagrant, AWS … Communication: ssh, winrm, … Provisioners: Chef, Puppet, Ansible … Testing: ServerSpec, Pester, BATS… Platform: CentOS, Ubuntu, Windows … Speaker Note: TK is the test runner, using it’s massively pluggable architecture to run tests on any platform, framework, etc.
30. kitchen converge kitchen verify kitchen login Speaker Note: The three commands we’ll be looking at are kitchen login/converge/verify. Login lets you poke around the server TK starts up. Converge will execute the provisioner against the server to put it into the desired state. Finally, verify will run all the tests inside the server.
31. Speaker Note: Speaking of tests, this is where ServerSpec comes in. ServerSpec is an extension of RSpec, a Ruby BDD testing library. It specifically focuses on server testing. ServerSpec uses the underlying OS commands to verify the state of the system.
32. Speaker Note: Speaking of tests, this is where ServerSpec comes in. ServerSpec is an extension of RSpec, a Ruby BDD testing library. It specifically focuses on server testing. ServerSpec uses the underlying OS commands to verify the state of the system.
33. Speaker Note: This is an example of how to test whether a system is listening on a specific port. ServerSpec will then use the underlying netstat command to check if the port is being listened to.
34. Speaker Note: This is how to check if a service exists or is enabled. ServerSpec will use the proper OS level check, like chkconfig on CentOS.
35. Speaker Note: You can use it to check if a user exists. In this case it’ll use id on Linux OSes.
36. Speaker Note: There are many more resources, but ServerSpec offers the command resource which provides the ultimate flexibility. You can execute any command and then inspect its standard out, standard error and exit status.
37. Speaker Note: I’d be remiss in a DevOps Days if I didn’t mention Docker. But Docker is perfect for testing. You want to spin up a server, very quickly, provision it and then tear it down.
38. Speaker Note: Using Test Kitchen’s pluggable architecture, we can pull in the third party kitchen-docker gem, make a minor change to the configuration file, and….
39. 0 57s 500ms 1m 55s 2m 52s 500ms 3m 50s Vagrant Docker Docker + Cache Speaker Note: Get a huge performance gain. With that small change, we get over 30% performance boost. If we cache the resources offline, we can tighten our feedback cycle from initial boot to converge to verify in under one minute.
40. Speaker Note: With Chef, Test Kitchen, ServerSpec and Docker in our tool belt, we put on our safety goggles and ask “what does the process look like?”
41. Speaker Note: To talk about the TDI process, we first need to discuss the TDD process. In TDD you first write the failing test, then you write the code to make it pass, and especially in software development, you refactor. You can safely refactor your code because you have the tests to back you up. I’m not religious about the order, as long as you write the tests close to the code under test.
42. Red Speaker Note: To talk about the TDI process, we first need to discuss the TDD process. In TDD you first write the failing test, then you write the code to make it pass, and especially in software development, you refactor. You can safely refactor your code because you have the tests to back you up. I’m not religious about the order, as long as you write the tests close to the code under test.
43. Green Red Speaker Note: To talk about the TDI process, we first need to discuss the TDD process. In TDD you first write the failing test, then you write the code to make it pass, and especially in software development, you refactor. You can safely refactor your code because you have the tests to back you up. I’m not religious about the order, as long as you write the tests close to the code under test.
44. Refactor Green Red Speaker Note: To talk about the TDI process, we first need to discuss the TDD process. In TDD you first write the failing test, then you write the code to make it pass, and especially in software development, you refactor. You can safely refactor your code because you have the tests to back you up. I’m not religious about the order, as long as you write the tests close to the code under test.
45. Speaker Note: The approach for TDI is very similar. You write a failing ServerSpec test, you make it pass with a Chef recipe/Ansible playbook/Puppet manifest, and then you refactor if necessary. You won’t refactor as often because the code is generally simpler. However, if you’re depending on an open source cookbook, like redisio, and sometime down the road you decide to write your own Redis cookbook, you have the tests to back you up.
46. ServerSpec Speaker Note: The approach for TDI is very similar. You write a failing ServerSpec test, you make it pass with a Chef recipe/Ansible playbook/Puppet manifest, and then you refactor if necessary. You won’t refactor as often because the code is generally simpler. However, if you’re depending on an open source cookbook, like redisio, and sometime down the road you decide to write your own Redis cookbook, you have the tests to back you up.
47. ServerSpec Recipe Speaker Note: The approach for TDI is very similar. You write a failing ServerSpec test, you make it pass with a Chef recipe/Ansible playbook/Puppet manifest, and then you refactor if necessary. You won’t refactor as often because the code is generally simpler. However, if you’re depending on an open source cookbook, like redisio, and sometime down the road you decide to write your own Redis cookbook, you have the tests to back you up.
48. Refactor ServerSpec Recipe Speaker Note: The approach for TDI is very similar. You write a failing ServerSpec test, you make it pass with a Chef recipe/Ansible playbook/Puppet manifest, and then you refactor if necessary. You won’t refactor as often because the code is generally simpler. However, if you’re depending on an open source cookbook, like redisio, and sometime down the road you decide to write your own Redis cookbook, you have the tests to back you up.
49. Speaker Note: Enough slides, let’s see Test Driven Infrastructure in action. We’ll get Redis installed practicing TDI.
50. Speaker Note: We don’t all live in our mother’s basement, and normally collaborate with others. How does TDI help with that?
51. Speaker Note: This is where Continuous Integration (CI) comes into play. With something like Bamboo or Jenkins, you get a central place that verifies the tests continue passing. With Docker, running these TK tests in CI is really easy.
52. Speaker Note: This is where Continuous Integration (CI) comes into play. With something like Bamboo or Jenkins, you get a central place that verifies the tests continue passing. With Docker, running these TK tests in CI is really easy.
53. Speaker Note: What does the full workflow look like? You follow the TDI cycle locally, commit and push to your central repo, that triggers a build, which fires up a Docker image and runs Test Kitchen and ServerSpec tests.
54. Refactor Recipe ServerSpec Speaker Note: What does the full workflow look like? You follow the TDI cycle locally, commit and push to your central repo, that triggers a build, which fires up a Docker image and runs Test Kitchen and ServerSpec tests.
55. git push Refactor Recipe ServerSpec Speaker Note: What does the full workflow look like? You follow the TDI cycle locally, commit and push to your central repo, that triggers a build, which fires up a Docker image and runs Test Kitchen and ServerSpec tests.
56. git push Refactor Recipe ServerSpec Speaker Note: What does the full workflow look like? You follow the TDI cycle locally, commit and push to your central repo, that triggers a build, which fires up a Docker image and runs Test Kitchen and ServerSpec tests.
57. git push Refactor Recipe ServerSpec triggers Speaker Note: What does the full workflow look like? You follow the TDI cycle locally, commit and push to your central repo, that triggers a build, which fires up a Docker image and runs Test Kitchen and ServerSpec tests.
58. git push Refactor Recipe ServerSpec triggers Speaker Note: What does the full workflow look like? You follow the TDI cycle locally, commit and push to your central repo, that triggers a build, which fires up a Docker image and runs Test Kitchen and ServerSpec tests.
59. git push Refactor Recipe ServerSpec triggers build Speaker Note: What does the full workflow look like? You follow the TDI cycle locally, commit and push to your central repo, that triggers a build, which fires up a Docker image and runs Test Kitchen and ServerSpec tests.
60. git push Refactor Recipe ServerSpec triggers build Speaker Note: What does the full workflow look like? You follow the TDI cycle locally, commit and push to your central repo, that triggers a build, which fires up a Docker image and runs Test Kitchen and ServerSpec tests.
61. Speaker Note: What’s the path ahead?
62. Speaker Note: We’ve been running TK and ServerSpec locally, but what about running a subset of your ServerSpec on production after a provision. Chef Audit being an interesting example.
63. Speaker Note: Another interesting use of TK is multi-server testing. You can have TK spin up several nodes, and have them all talk to each other on a private local network. We’ve had success testing Redis master, slave and sentinel configurations as well as testing the entire ELK stack.
64. Speaker Note: If you take one thing away, please test your infrastructure and be a super hero.
66. • Slides: https://speakerdeck.com/amaltson • Test Kitchen: http://kitchen.ci/ • ServerSpec: http://serverspec.org/ • test-driven-redis: https://github.com/amaltson/test-driven-redis • offline docker images: https://github.com/amaltson/docker-redis-requirements @amaltson Arthur Maltson We’re Hiring!
67. Credits • Slide 1 - Riccardo Cuppini, Zen [Explored], https://flic.kr/p/5ehoTC • Slide 3 - woodleywonderworks, my son's teacher is a supermodel, https://flic.kr/p/auPuAq • Slide 4 - Harry (Howard) Potts Follow, Zulu pensioners - drums, https://flic.kr/p/81TmRT • Slide 5 - CollegeDegrees360, Computer Problems, https://flic.kr/p/cEJpCY • Slide 6 - Greg Heo, Big banks, https://flic.kr/p/dfb13h • Slide 7 - Heisenberg Media, Berlin Startup Tour, https://flic.kr/p/dP6W49 • Slide 9 - Will Humes Follow, crossed fingers, https://flic.kr/p/4s5kZ5 • Slide 10 - sheila miguez, on call, https://flic.kr/p/37xur8 • Slide 13, 16 - Matthew Frederickson, Unicorns, https://flic.kr/p/5jrvmr • Slide 14, yosuke muroya, Unicorn, https://flic.kr/p/bpQFTw • Slide 17, 44, Chris & Karen Highland, consumer confidence!, https://flic.kr/p/qKcmR2 • Slide 18 - Quentin Meulepas, Whistler: Inukshuk, https://flic.kr/p/6izmiv • Slide 19 - Moyan Brenn, Happiness, https://flic.kr/p/nMmBGs • Slide 20 - Bre Pettis, Dave’s Bike Tools, https://flic.kr/p/QMVMw • Slide 34 - F Delventhal, Safety First, https://flic.kr/p/EmGgn • Slide 37 - MsSaraKelly, Take one: Sarah's hen do, https://flic.kr/p/fsKWAi • Slide 38 - Lawrence Whittemore, basement.jpg, https://flic.kr/p/c84PL • Slide 40 - Joseph Thornton, 2013 Retina Macbook Pro, https://flic.kr/p/eu3G38 • Slide 41 - Matthew Faltz, The Path, https://flic.kr/p/pA7dZQ • Slide 43 - tribp, Grapes, https://flic.kr/p/dcZUgY • Slide 45 - Nate Grigg, Thank You, https://flic.kr/p/6K41qv