Test Driven Infrastructure

Test Driven Infrastructure

This presentation was given at Web Unleashed 2018 (and older version at Full Stack TO 2015). It included a live coding section which is represented by code slides.

Test Driven Development is a popular concept in Software Development, leading to higher quality code that's easier to maintain. Automated testing is normally a foreign concept in the Operations world, but as you ssh into your servers to make that quick fix or run your updated script (fingers crossed), you might be wondering if there's a better way. A way that gives you the confidence in your script and lets you test those scripts in isolation. Well I have good news for you, there is a better way! Test Driven Infrastructure (TDI) is now possible. I know, it sounds crazy.

At this session you'll learn the how, and more importantly the why, of TDI. You'll see how Chef (or any other Config Management framework) can be tested with Test Kitchen and ServerSpec. You'll also learn how to improve your feedback cycle with Docker, and using the Docker approach on a CI server. There may even be some live demos!

Finally, the Ops world collides with the Dev world in true DevOps testing bliss.

275e91e7de0d6321f3759c2d0789c6d2?s=128

Arthur Maltson

October 02, 2018
Tweet

Transcript

  1. 7.

    Speaker Note: Or cradle a bottle of wine under your

    desk after another failed deploy?
  2. 8.

    Speaker Note: Or cradle a bottle of wine under your

    desk after another failed deploy?
  3. 10.

    Speaker Note: Or that automation you’ve written behaves in surprising

    ways? If so, you’re probably experiencing untested infrastructure.
  4. 11.

    Speaker Note: If you happen to talk to your friendly

    neighbourhood DevOps Unicorn….
  5. 12.

    Speaker Note: They might tell you about Test Driven Infrastructure.

    But what is Test Driven Infrastructure (TDI)?
  6. 13.

    Refactor Green Red Speaker Note: TDI comes from a process

    in Software Development known as Test Driven Development (TDD). This is a popular technique that has been shown to lead to higher quality code, that’s more stable and easier to maintain.
  7. 19.

    Speaker Note: To start, we need a Configuration Management system.

    Doesn’t have to be Chef, that’s just the example here. At the end of the day you could use Bash scripts, but CM will probably works better.
  8. 20.

    Speaker Note: We’ll use a default cookbook generated by Chef

    DK using the chef generate cookbook command.
  9. 21.

    Speaker Note: In our example we’ll set up Redis. In

    Chef, using the redisio cookbook from the Supermarket, it might look like this.
  10. 22.

    Speaker Note: The other tool we’ll need is Test Kitchen

    (TK). TK is going to be our primary testing work horse.
  11. 23.

    Speaker Note: TK is the test runner, using it’s massively

    pluggable architecture to run tests on any platform, framework, etc.
  12. 24.

    Drivers: docker, Vagrant, AWS … Speaker Note: TK is the

    test runner, using it’s massively pluggable architecture to run tests on any platform, framework, etc.
  13. 25.

    Drivers: docker, Vagrant, AWS … Communication: ssh, winrm, … Speaker

    Note: TK is the test runner, using it’s massively pluggable architecture to run tests on any platform, framework, etc.
  14. 26.

    Drivers: docker, Vagrant, AWS … Communication: ssh, winrm, … Provisioners:

    Chef, Ansible, Puppet … Speaker Note: TK is the test runner, using it’s massively pluggable architecture to run tests on any platform, framework, etc.
  15. 27.

    Drivers: docker, Vagrant, AWS … Communication: ssh, winrm, … Provisioners:

    Chef, Ansible, Puppet … Testing: InSpec, Pester, BATS… Speaker Note: TK is the test runner, using it’s massively pluggable architecture to run tests on any platform, framework, etc.
  16. 28.

    Drivers: docker, Vagrant, AWS … Communication: ssh, winrm, … Provisioners:

    Chef, Ansible, Puppet … Testing: InSpec, Pester, BATS… Platform: CentOS, Ubuntu, Windows … Speaker Note: TK is the test runner, using it’s massively pluggable architecture to run tests on any platform, framework, etc.
  17. 29.

    kitchen converge kitchen verify kitchen login Speaker Note: The three

    commands we’ll be looking at are kitchen login/converge/verify. Login lets you poke around the server TK starts up. Converge will execute the provisioner against the server to put it into the desired state. Finally, verify will run all the tests inside the server.
  18. 30.

    Speaker Note: Speaking of tests, this is where InSpec comes

    in. InSpec is an extension of RSpec, a Ruby BDD testing library. It specifically focuses on server testing. InSpec uses the underlying OS commands to verify the state of the system.
  19. 31.

    Speaker Note: Speaking of tests, this is where InSpec comes

    in. InSpec is an extension of RSpec, a Ruby BDD testing library. It specifically focuses on server testing. InSpec uses the underlying OS commands to verify the state of the system.
  20. 32.

    Speaker Note: This is an example of how to test

    whether a system is listening on a specific port. InSpec will then use the underlying netstat command to check if the port is being listened to. You can also make sure it’s NOT listening on specific ranges ports.
  21. 33.

    Speaker Note: This is how to check if a service

    exists or is enabled. InSpec will use the proper OS level check, like chkconfig on CentOS.
  22. 34.

    Speaker Note: You can use it to check if a

    user exists. In this case it’ll use id on Linux OSes.
  23. 35.

    Speaker Note: There are many more resources, but InSpec offers

    the command resource which provides the ultimate flexibility. You can execute any command and then inspect its standard out, standard error and exit status.
  24. 36.

    Speaker Note: I’d be remiss if I didn’t mention Docker

    in a DevOps themed talk. But Docker is perfect for testing. You want to spin up a server, very quickly, provision it and then tear it down.
  25. 37.

    Speaker Note: Using Test Kitchen’s pluggable architecture, we can pull

    in the third party kitchen-docker gem, make a minor change to the configuration file, and….
  26. 38.

    0 57s 500ms 1m 55s 2m 52s 500ms 3m 50s

    Vagrant Docker Docker + Cache Speaker Note: Get a huge performance gain. With that small change, we get over 30% performance boost. If we cache the resources offline, we can tighten our feedback cycle from initial boot to converge to verify in under one minute.
  27. 39.

    Speaker Note: With Chef, Test Kitchen, InSpec and Docker in

    our tool belt, we put on our safety goggles and ask “what does the process look like?”
  28. 40.

    Speaker Note: To talk about the TDI process, we first

    need to discuss the TDD process. In TDD you first write the failing test, then you write the code to make it pass, and especially in software development, you refactor. You can safely refactor your code because you have the tests to back you up. I’m not religious about the order, as long as you write the tests close to the code under test.
  29. 41.

    Red Speaker Note: To talk about the TDI process, we

    first need to discuss the TDD process. In TDD you first write the failing test, then you write the code to make it pass, and especially in software development, you refactor. You can safely refactor your code because you have the tests to back you up. I’m not religious about the order, as long as you write the tests close to the code under test.
  30. 42.

    Green Red Speaker Note: To talk about the TDI process,

    we first need to discuss the TDD process. In TDD you first write the failing test, then you write the code to make it pass, and especially in software development, you refactor. You can safely refactor your code because you have the tests to back you up. I’m not religious about the order, as long as you write the tests close to the code under test.
  31. 43.

    Refactor Green Red Speaker Note: To talk about the TDI

    process, we first need to discuss the TDD process. In TDD you first write the failing test, then you write the code to make it pass, and especially in software development, you refactor. You can safely refactor your code because you have the tests to back you up. I’m not religious about the order, as long as you write the tests close to the code under test.
  32. 44.

    Speaker Note: The approach for TDI is very similar. You

    write a failing InSpec test, you make it pass with a Chef recipe/Ansible playbook/Puppet manifest, and then you refactor if necessary. You won’t refactor as often because the code is generally simpler. However, if you’re depending on an open source cookbook, like redisio, and sometime down the road you decide to write your own Redis cookbook, you have the tests to back you up.
  33. 45.

    InSpec Speaker Note: The approach for TDI is very similar.

    You write a failing InSpec test, you make it pass with a Chef recipe/Ansible playbook/Puppet manifest, and then you refactor if necessary. You won’t refactor as often because the code is generally simpler. However, if you’re depending on an open source cookbook, like redisio, and sometime down the road you decide to write your own Redis cookbook, you have the tests to back you up.
  34. 46.

    InSpec Recipe Speaker Note: The approach for TDI is very

    similar. You write a failing InSpec test, you make it pass with a Chef recipe/Ansible playbook/Puppet manifest, and then you refactor if necessary. You won’t refactor as often because the code is generally simpler. However, if you’re depending on an open source cookbook, like redisio, and sometime down the road you decide to write your own Redis cookbook, you have the tests to back you up.
  35. 47.

    Refactor InSpec Recipe Speaker Note: The approach for TDI is

    very similar. You write a failing InSpec test, you make it pass with a Chef recipe/Ansible playbook/Puppet manifest, and then you refactor if necessary. You won’t refactor as often because the code is generally simpler. However, if you’re depending on an open source cookbook, like redisio, and sometime down the road you decide to write your own Redis cookbook, you have the tests to back you up.
  36. 48.

    Speaker Note: Enough slides, let’s see Test Driven Infrastructure in

    action. We’ll get Redis installed practicing TDI.
  37. 49.
  38. 51.

    Speaker Note: When installing Redis, we want to make sure

    Redis runs under it’s own user as is standard practice in Linux. We also want make sure Redis stores it’s database in the default /var/lib/redis directory. We write out these tests in InSpec. Make it fail.
  39. 53.

    Speaker Note: We then add a dependency on the redisio

    Supermarket cookbook. This will help us set up Redis.
  40. 54.

    Speaker Note: And finally, Chef specific, we include the default

    recipe to get Redis installed. Now our test passes.
  41. 55.

    Speaker Note:The tests passed, but we don’t know if Redis

    is actually started and running. Let’s write some failing tests that make sure Redis is listening on the default 6379 port and has a service to ensure Redis starts back up on reboots. Redisio names the service with the port number.
  42. 56.

    Speaker Note: To make the test pass, we just need

    to include the ‘enable’ recipe.
  43. 57.

    Speaker Note: Redis being a database, you want to be

    able to store and retrieve data from it. Using the command resource in InSpec, we can call any command on the OS. We can use the redis-cli command to put data into Redis and get data out. Let’s write those tests. They pass.
  44. 58.

    Speaker Note: Finally, we discussed how Test Kitchen has support

    for multiple platforms. Let’s say tomorrow your CIO comes down and says “there’s this great new Linux server out there called Ubuntu Server, we should use it.” Fortunately, since we’ve followed a Test Drive Infrastructure approach, we just need to add the Ubuntu platform (in this case an offline cache version), run kitchen converge and then kitchen verify to see a passing build of Redis installed and working on Ubuntu. This is kind of a “refactor” to add Ubuntu support.
  45. 59.

    Refactor InSpec Recipe Speaker Note: We now saw the full

    virtuous cycle of writing a failing ServerSpec test, writing the recipe/playbook/etc to make that test pass, and then even “refactor” by changing the platform we support.
  46. 60.

    Speaker Note: Of course nothing has all upsides, there are

    some downsides with testing too. If you’re cookbook does too much, your feedback cycle can get really long. If the tests you execute take a long time to return, and you do a lot of them, that also increases the feedback cycle. Testing also adds more process, so shipping takes longer. Just like with software development 10 years ago, there was questions on “why would I test, I never did that before”. 10 years later, we’re terrified to touch code that doesn’t have tests. My recommendation would be, keep your cookbooks/playbooks/etc small and focused. You should really only need 30-100 tests.
  47. 61.

    Speaker Note: At the end of the day, the tradeoffs

    and gotchas are well worth it. It’s all about safety and confidence in making changes to your infrastructure. We’ve found tests catch a range of issues, like wrong configurations for websites, before it ever hits the development or production environment. The end goal is to move fast and continuous deliver.
  48. 62.

    Speaker Note: continuously deliver value, not downtime. This whole time

    we’ve been talking about development on our local workstation…
  49. 63.

    Speaker Note: This is where Continuous Integration (CI) comes into

    play. With something like CircleCI or Jenkins, you get a central place that verifies the tests continue passing. With Docker, running these TK tests in CI is really easy.
  50. 64.

    Speaker Note: This is where Continuous Integration (CI) comes into

    play. With something like CircleCI or Jenkins, you get a central place that verifies the tests continue passing. With Docker, running these TK tests in CI is really easy.
  51. 65.

    Speaker Note: What does the full workflow look like? You

    follow the TDI cycle locally, commit and push to your central repo, that triggers a build, which fires up a Docker image and runs Test Kitchen and InSpec tests. If build fails, you deal with it on your workstation and continue the cycle again.
  52. 66.

    Refactor Recipe ServerSpec Speaker Note: What does the full workflow

    look like? You follow the TDI cycle locally, commit and push to your central repo, that triggers a build, which fires up a Docker image and runs Test Kitchen and InSpec tests. If build fails, you deal with it on your workstation and continue the cycle again.
  53. 67.

    git push Refactor Recipe ServerSpec Speaker Note: What does the

    full workflow look like? You follow the TDI cycle locally, commit and push to your central repo, that triggers a build, which fires up a Docker image and runs Test Kitchen and InSpec tests. If build fails, you deal with it on your workstation and continue the cycle again.
  54. 68.

    git push Refactor Recipe ServerSpec Speaker Note: What does the

    full workflow look like? You follow the TDI cycle locally, commit and push to your central repo, that triggers a build, which fires up a Docker image and runs Test Kitchen and InSpec tests. If build fails, you deal with it on your workstation and continue the cycle again.
  55. 69.

    git push Refactor Recipe ServerSpec triggers Speaker Note: What does

    the full workflow look like? You follow the TDI cycle locally, commit and push to your central repo, that triggers a build, which fires up a Docker image and runs Test Kitchen and InSpec tests. If build fails, you deal with it on your workstation and continue the cycle again.
  56. 70.

    git push Refactor Recipe ServerSpec triggers Speaker Note: What does

    the full workflow look like? You follow the TDI cycle locally, commit and push to your central repo, that triggers a build, which fires up a Docker image and runs Test Kitchen and InSpec tests. If build fails, you deal with it on your workstation and continue the cycle again.
  57. 71.

    git push Refactor Recipe ServerSpec triggers build Speaker Note: What

    does the full workflow look like? You follow the TDI cycle locally, commit and push to your central repo, that triggers a build, which fires up a Docker image and runs Test Kitchen and InSpec tests. If build fails, you deal with it on your workstation and continue the cycle again.
  58. 72.

    git push Refactor Recipe ServerSpec triggers build Speaker Note: What

    does the full workflow look like? You follow the TDI cycle locally, commit and push to your central repo, that triggers a build, which fires up a Docker image and runs Test Kitchen and InSpec tests. If build fails, you deal with it on your workstation and continue the cycle again.
  59. 73.

    git push Refactor Recipe ServerSpec triggers build ❌ Speaker Note:

    What does the full workflow look like? You follow the TDI cycle locally, commit and push to your central repo, that triggers a build, which fires up a Docker image and runs Test Kitchen and InSpec tests. If build fails, you deal with it on your workstation and continue the cycle again.
  60. 74.

    triggers build ✅ Speaker Note: But what happens when the

    build passes? This is where you can use the Continuous Delivery features available in your CI system, most of them have something built in (eg Bamboo). A successful CI build can automatically trigger a deployment to development. With enough comfort level, you could even trigger an automatic deployment to production.
  61. 75.

    triggers build ✅ deploy Speaker Note: But what happens when

    the build passes? This is where you can use the Continuous Delivery features available in your CI system, most of them have something built in (eg Bamboo). A successful CI build can automatically trigger a deployment to development. With enough comfort level, you could even trigger an automatic deployment to production.
  62. 76.

    triggers build ✅ deploy Speaker Note: But what happens when

    the build passes? This is where you can use the Continuous Delivery features available in your CI system, most of them have something built in (eg Bamboo). A successful CI build can automatically trigger a deployment to development. With enough comfort level, you could even trigger an automatic deployment to production.
  63. 77.

    triggers build ✅ deploy ✅ Speaker Note: But what happens

    when the build passes? This is where you can use the Continuous Delivery features available in your CI system, most of them have something built in (eg Bamboo). A successful CI build can automatically trigger a deployment to development. With enough comfort level, you could even trigger an automatic deployment to production.
  64. 78.

    triggers build ✅ deploy deploy ✅ Speaker Note: But what

    happens when the build passes? This is where you can use the Continuous Delivery features available in your CI system, most of them have something built in (eg Bamboo). A successful CI build can automatically trigger a deployment to development. With enough comfort level, you could even trigger an automatic deployment to production.
  65. 79.

    triggers build ✅ deploy deploy ✅ Speaker Note: But what

    happens when the build passes? This is where you can use the Continuous Delivery features available in your CI system, most of them have something built in (eg Bamboo). A successful CI build can automatically trigger a deployment to development. With enough comfort level, you could even trigger an automatic deployment to production.
  66. 80.

    triggers build ✅ deploy deploy ✅ ✅ Speaker Note: But

    what happens when the build passes? This is where you can use the Continuous Delivery features available in your CI system, most of them have something built in (eg Bamboo). A successful CI build can automatically trigger a deployment to development. With enough comfort level, you could even trigger an automatic deployment to production.
  67. 82.

    Speaker Note: Another interesting use of TK is multi-server testing.

    You can have TK spin up several nodes, and have them all talk to each other on a private local network. We’ve had success testing Redis primary, replica and sentinel configurations as well as testing the entire ELK stack.
  68. 83.

    Speaker Note: But ultimately you’ll have to run this Redis

    server somewhere, and most likely you’re going to do it in the Cloud.
  69. 85.

    Speaker Note: It is, because InSpec supports AWS (and Azure)

    resources out of the box. You can check on AWS EC2 instances.
  70. 87.

    Speaker Note: And even S3 buckets. You can make sure

    your buckets are never publicly exposed!
  71. 88.

    Speaker Note: If you use the popular Terraform tool to

    create that infrastructure, you can use the awesome kitchen-terraform plugin to tie this all together!
  72. 91.

    Speaker Note: If you take one thing away, please test

    your infrastructure and be a super hero.
  73. 92.
  74. 93.
  75. 94.
  76. 95.
  77. 97.
  78. 98.
  79. 99.

    • Slides: https://speakerdeck.com/amaltson • Test Kitchen: https://kitchen.ci • InSpec: https://www.inspec.io

    • Kitchen Docker: https://github.com/test-kitchen/kitchen-docker • Kitchen Terraform: https://github.com/newcontext-oss/kitchen-terraform • test-driven-redis: https://github.com/amaltson/test-driven-redis @amaltson Arthur Maltson We’re Hiring!
  80. 100.

    Credits • Slide 1 - Riccardo Cuppini, Zen [Explored], https://flic.kr/p/5ehoTC

    • Slide 2 - CollegeDegrees360, Computer Problems, https://flic.kr/p/cEJpCY • Slide 3 - Greg Heo, Big banks, https://flic.kr/p/dfb13h • Slide 4 - Heisenberg Media, Berlin Startup Tour, https://flic.kr/p/dP6W49 • Slide 6 - Will Humes Follow, crossed fingers, https://flic.kr/p/4s5kZ5 • Slide 7 - Crying Under the Table With a Bottle of Wine GIF, https://mashable.com/2013/08/20/gif-origins/#3PUHZ0bAVPqj • Slide 10, 13 - Matthew Frederickson, Unicorns, https://flic.kr/p/5jrvmr • Slide 11, yosuke muroya, Unicorn, https://flic.kr/p/bpQFTw • Slide 14, 59, Chris & Karen Highland, consumer confidence!, https://flic.kr/p/qKcmR2 • Slide 15 - Quentin Meulepas, Whistler: Inukshuk, https://flic.kr/p/6izmiv • Slide 16 - Moyan Brenn, Happiness, https://flic.kr/p/nMmBGs • Slide 17, 32 - Bre Pettis, Dave’s Bike Tools, https://flic.kr/p/QMVMw • Slide 31 - F Delventhal, Safety First, https://flic.kr/p/EmGgn • Slide 34 - MsSaraKelly, Take one: Sarah's hen do, https://flic.kr/p/fsKWAi • Slide 47 - Simon Harrod, Strawberry Snail, https://flic.kr/p/9XkFkY • Slide 49 - GotCredit, Safety, https://flic.kr/p/qHCmfo • Slide 51 - Lawrence Whittemore, basement.jpg, https://flic.kr/p/c84PL • Slide 53 - Joseph Thornton, 2013 Retina Macbook Pro, https://flic.kr/p/eu3G38 • Slide 54, 57 - DeclanTM, Home Server, https://flic.kr/p/4PGBb5 • Slide 54, 57 - Quinn Dombrowski, Servers, https://flic.kr/p/cqqwcb • Slide 55 - Matthew Faltz, The Path, https://flic.kr/p/pA7dZQ • Slide 58 - tribp, Grapes, https://flic.kr/p/dcZUgY • Slide 60 - Nate Grigg, Thank You, https://flic.kr/p/6K41qv • Slide 62 - woodleywonderworks, my son's teacher is a supermodel, https://flic.kr/p/auPuAq • Slide 63 - Harry (Howard) Potts Follow, Zulu pensioners - drums, https://flic.kr/p/81TmRT