Simple system testing with Roundup and Ansible

Automated testing has become a natural part of toolboxes, not only for application developer's toolboxes but also for sysadmins sailing under the oh-so-misinterpreted DevOps flag. Testing of installed systems goes along with configuration management just nicely and has been introduced e.g. at CfgMgmtCamp for Ansible in combination with Inspec.

In this talk I want to introduce you to a much simpler alternative that doesn't require a dedicated programming language stack such as Python or Ruby: roundup. roundup is a really itsy-bitsy-tiny little rapper that adds just enough around your off-the-shelf Bash test functions to run isolated and get easy overviewable test results. If this doesn't make you immediately roll with your eyes for fear of a maintenance nightmare, let me remind you how our well-established Un*x heritage of tools and plumbing allows us to even test our mail server's basic functionality without needing 31337 add-on libraries. For the bravest (or craziest, depending on the viewpoint) among you, I will show how dynamic test generation takes us close to the border of language abuse.

In the second part I will then also show how I integrate roundup with my Ansible playbooks, giving instant feedback that desired configuration has not only been applied successfully but also yielded the intended system state. And this all without fancy setup steps and orphaned test artifacts lingering around on tested systems.

Transcript

1. Simple system testing with Roundup & Ansible Pieter Hollants LOADays pieter@hollants.com Antwerp, Belgium pfhllnts April 21st, 2018
2. Pieter who? Developer (by now primarily Python) but also… IT support in NDS/Win95 times for 3y (Senior) Intern at SUSE Consulting for 9y Linux System Engineer at German Air Trafic Control for 4y ...and Freelancer since over 15y dfs.de
3. My story goes like this: I... ...don‘t need much IT for my needs but some (e.g. file server, mail service) ...run IT mostly on my own for reasons ...don‘t do admin work ofen enough (and no, I don‘t want to either) ...thus keep forgetting things ...thus cfg-manage & automate ...because infra as code (done right) sort of documents itself. (yes, a very broad claim, but so there)
4. So I... ...began with Puppet but switched to Ansible ...wrote my own Ansible roles like everyone else („code reuse“ *cough*) ...actually realized I didn‘t always know when I‘d be finished, if I had forgotten something etc. → config management is making explicit the things I‘d otherwise do manually (implicit) and requires me to think and reflect about intent and desired state.
6. http://docs.ansible.com/ansible/latest/reference_appendices/test_strategies.html Twitter Emoji / CC-BY
7. Intent != end result Ansible, modules, roles surely get tested, that‘s not the point. Admin‘s intent vs. the actual execution is. Code is expression of intent. We can battle-test our playbook but we can not easily test for overseen/misinterpreted intent. But it helps to specifically try to code tests for intended end result (we think about „...to achieve what?“ instead of the „how“).
8. Testing all the things Modern Admins/DevOps/SREs/IT-Gods have adapted developer methodologies and tools (version control, continuous integration/deployment, containers etc.) including (automatic) testing. But what is subject and scope of a test? Testing is a wide field: Automanual tests vs. Generative tests Unit tests vs. Integration tests vs. Acceptance tests Playbook syntax testing vs. Role testing (→ Ton‘s talk!)
9. System testing On role testing... gives a good feeling about that role itself but says nothing about the state of a particular installed system. → Augment it with tests of the installed system (can be any of unit tests, integration tests, acceptance tests). Where do these run? On the system itself (testing from the inside). Could use Python, Ruby, Node… and some sophisticated testing frameworks. Or… Ansible again?
10. Testing from within Ansible itself wait_for: wait for port to open fail combined with when condition / assert with condition script: fail when script exits with non-zero return code http://docs.ansible.com/ansible/latest/reference_appendices/test_strategies.html We could do this (and it doesn‘t hurt) but...
11. Testing in same $PROGLANG? There are cases where it is natural to write tests in the same programming language as the implementation. There are also cases where it‘s apt not to do that. Installing systems is such a case because we care about using these systems, primarily by means of interactions. An Ansible playbook is not really the right place to model such interactions.
13. System tests as a Bash script (1/2) Short. Simple. Doesn‘t require a whole sofware stack. Can you do that in $PROGLANGUAGE?
14. System tests as a Bash script (2/2) Well, does the job but… - Output hard to grasp (no explicit success/failure) - Failing tests causes remaining tests to be skipped - No pre-tests setup/post-tests teardown functions - No isolation between single tests (think env vars) Flickr/zeitfaenger.at / CC-BY-SA
15. Flickr/Mike Mozart / CC-BY-SA
16. Blake Mizerany: Author of Sinatra for Ruby
17. Bash script 107 lines of code 307 lines total Well-commented A lot of shell magic
18. roundup Features (1/2) Roundup: wrapper, gets called in directory with „test plans“, optionally with name of single test plan to execute. Test plan: ordinary shell script, sourced by roundup, executes each test put in it_* function afer another in isolated sandbox fashion (e.g. with its own clean environment). Test output will be PASS or FAIL, in the latter case stdout/stderr shown.
19. System tests as roundup test plan (1/2)
20. System tests as roundup test plan (2/2)
21. roundup Features (2/2) Each test plan can set a description by calling the roundup- provided describe function, which will get shown instead of the test name. Pre-tests setup and post-tests teardown can be done in before and after functions. after will always be called, even if an error occured. And that‘s it. Just enough to spice up your tests.
22. roundup – The bad parts Last commit: January 2015. Open issues: 3. Pull requests: 11. Test selection limited to globbing. Tests can‘t signal to be skipped. No timing statistics. before() / after() have some issues (see Github). No programmatic test generation… (really? ;) 61 forks… which one to pick? But: usable enough for me.
23. roundup – Alternatives https://github.com/lehmannro/assert.sh Sourced into test scripts, assert* functions, no test isolation, no pre-test setup/post-test teardown https://github.com/kward/shunit2 Sourced into test scripts, assert* functions, oneTimeSetup()/setUp()/tearDown()/oneTimeTearDown(), skipping possible, test suites, inactive, multiple src repos https://github.com/sstephenson/bats @test „description“ {} syntax, .bats sufix, setup()/teardown(), code sharing, TAP (Test Anything Protocol)-compliant https://medium.com/wemake-services/testing-bash-applications-85512e7fe2de
24. roundup without test generation Suppose Toshy wanted to test whether ALL of his mail aliases were working. Repeating a variation of the same test is a bit cumbersome, right? Generates a unique subject for a test mail. Takes sender and recipient addresses and subject and sends a test mail using mailx. Checks using mailx whether test mail has arrived.
25. By a former colleague of mine. I smell a challenge here ;)
26. roundup & bash abuse 101 (1/2) Use for-loop to iterate over recipient addresses. Use eval() to dynamically define a test function for each recipient address. Nota bene: There is a reason there‘s only one letter diference from „eval“ to „evil“. See e.g. https://medium.com/dot-debug/the-perils-of-bash-eval-cc5f9e309cae Problem: roundup actually greps the test plan to obtain the list of test functions. Not really a problem, though. roundup also sources the test plan, so we simply expand the roundup_plan variable ourselves :)
27. roundup & bash abuse 101 (2/2) Might end up in hell for this… but there is actually an alternative if we combine roundup with Ansible... List of recipient address parts No dots allowed in shell function names Append at end of test plan.
29. 1st approach – Total independence Run Ansible playbook. Deploy roundup tests somehow to target system. ssh in ever afer and run roundup manually to test the installation. + roundup tests stay independent from cfgmgmt solution. + roundup tests can be run on the installed system at any time. - Lazy admin forgets/omits. And we‘re all lazy! - How to deal with sensitive data inside the tests, ofen required in cleartext (think email passwords)? Omit? - Ugly Bash abuse required for test generation.
30. 2nd approach – Let me help you Let Ansible do stuf, deploy (*cough* rsync) and run roundup tests. + roundup tests basically still stay independent from cfgmgmt solution. + Tests get automatically run. + roundup tests can still be run on the installed system at any time. - How to deal with sensitive data inside the tests, ofen required in cleartext (think email passwords). Omit? - Ugly Bash abuse required for test generation.
31. 3rd approach – A drone Let Ansible do stuf, deploy and run roundup tests but now also delete them aferwards. + roundup tests themselves still stay independent from cfgmgmt solution. + Tests get automatically run. + Sensitive data is no longer permanently lingering around on installed system. - roundup tests can now only be triggered externally from controlhost via Ansible playbook (think --tags roundup) - Ugly Bash abuse required for test generation.
32. 4th approach - Built-to-order Let Ansible now first template, then deploy and run roundup tests and delete them aferwards. + Tests get automatically run. + Sensitive data is no longer permanently lingering around on installed system. + Can use templating power to avoid ugly Bash abuse. - roundup tests can now only be triggered externally from controlhost via Ansible playbook (think --tags roundup) - roundup tests now entwined with cfgmgmt solution.
33. Quo vadis? I currently go with the 4th approach. But as always:
34. Winners use roles You didn‘t seriously think that we‘re going to repeat all those roundup steps in each and every playbook by hand, right? tests_dir: A directory with xxx-test.sh.j2 files preinstall_packages: Additional sofware otherwise not installed and required for the tests only (e.g. heirloom-mailx for testing mail server facilities)
35. roundup role anatomy What the associated little Ansible role does: - Downloads roundup.sh to /usr/local/bin/roundup with the get_url module - Installs preinstall_packages with the package module - block: - Creates a temporary directory for the tests with the tempfile module - Uses the template module to template/transfer tests - Uses the shell module to call roundup on the templated tests - always: - Deletes temporary directory
36. Example role output (1/2)
37. Example role output (2/2)
38. FIN One small gotcha: When using --tags, don‘t forget to include the roundup role. Or tag it with all tags used. Another gotcha: roundup role not yet available at Ansible Galaxy. Will make it available at http://github.com/pief if there is interest. Thank you!