Developers, Sysadmins, and everyone else: Why you should be using Serverspec (with speaker notes)

Sarah Kowalik and Jesse Reynolds
Jan 13 2015, LCA 2015, Auckland
A talk as part of the Miniconf: Developer, Testing, Release and Continuous Integration Automation http://linux.conf.au/schedule/30206/view_talk?day=tuesday

Transcript

1. #lca2015 Developers, Sysadmins, and everyone else: Why you should be using Serverspec Sarah Kowalik & Jesse Reynolds R&D @ Bulletproof
2. #lca2015 Let me paint you a picture… You're coming to the end of a deathmarch. The team has been slogging away tirelessly for the last 2 months getting all the required features ready for the prospective client. But there's a problem. How is the client going to get the software installed? You cobble together a tarball of the software, attach it to an email, hit send, and clock off. It worked in dev. Ops' problem now. You cruise into the office at 09.45 next Monday, and discover your inbox has exploded. Your boss wants your head because the customer couldn't install the release, and when they eventually got it running they got owned 15 minutes later by the latest OpenSSL exploit. How can you get better assurance that the software you're building and shipping does what it says on the box? For this, we're using Serverspec.
3. #lca2015 What is Serverspec? Serverspec is a framework for testing infrastructure and applications. Built on top of Rspec, a popular ruby testing framework. It uses standard Rspec syntax, so you use all the existing Rspec functions, and there’s lots of resources on the web. You can run it via ssh, using vagrant, or locally.
4. #lca2015 File "/var/log/flapjack/flapjack.log" should be file should be mode 644 should be owned by "flapjack" (FAILED - 1) 1) File "/var/log/flapjack/flapjack.log" should be owned by "flapjack" On host `127.0.0.1' Failure/Error: it { should be_owned_by 'flapjack' } expected `File "/var/log/flapjack/ flapjack.log".owned_by?("flapjack")` to return true, got false sudo -p 'Password: ' /bin/sh -c stat\ -c\ \%U\ / var/log/flapjack/flapjack.log\ \|\ grep\ --\ \\ \^flapjack\\\$ Example output…
5. #lca2015 Why Serverspec?
6. #lca2015 Automate your QA Humans can miss things in QA. Can only do QA during certain times of the day. Hard to scale - difficult to get more humans during busy periods. Serverspec will keep testing…and testing…the same way every time, at any time of the day or night.
7. #lca2015 Validate your expectations Run Serverspec against your development code to check that files are being created as you expect - a more concise form of cucumber tests!
8. #lca2015 Fail early Have Serverspec run as part of your build procedure, and when the tests fail, fail the entire build. Never publish a build that doesn’t work!
9. #lca2015 Validate your output For CLI tools, you can run commands, and check the standard out and error contain what you expect.
10. #lca2015 Works on my machine (and yours) Because Serverspec integrates with vagrant, you can easily test on multiple versions of multiple operating systems. This, of course, can be automated so you test all your operating systems with different vagrant boxes during one build.
11. #lca2015 Build confidence With all of this, you have a much better chance of your release working as you expect! As a bonus, you’ll free up your QA people to write more Serverspec tests (and Capybara tests for the GUI elements)!
12. #lca2015 So, how might you use Serverspec?
13. #lca2015 As a Developer…
14. #lca2015 Package installation describe package(’flapjack') do it { should be_installed } it { should be_enabled } end
15. #lca2015 Command Execution describe command(‘flapjack receiver httpbroker —help’) do its(:exit_status) { should eq 0 } its(:stdout) { should match /port/ } its(:stderr) { should match /interval/ } end
16. #lca2015 Running processes describe process("flapjack") do it { should be_running } its(:args) { should match /server start/ } end
17. #lca2015 File presence, perms describe file('/var/log/flapjack/flapjack.log') do it { should be_file } it { should be_mode 644 } end
18. #lca2015 File ownership describe file('/var/log/flapjack/flapjack.log') do it { should be_owned_by 'flapjack' } it { should be_grouped_into 'flapjack' } end
19. #lca2015 File hash describe file('/etc/services') do its(:sha256sum) { should eq ‘a861c…’ } end - ensure not corrupted
20. #lca2015 Listening on a port describe port(3080) do it { should be_listening } end
21. #lca2015 As a Sysadmin… Eg running Puppet on a server or making changes to infrastructure.
22. #lca2015 After running puppet on a server
23. #lca2015 File contents describe file('/etc/ssh/sshd_config') do its(:content) { should match /Port 22/ } end Serverspec doesn’t only check for file existence - it also checks the type (whether it’s is a file, symlink or a directory). You can also check the contents of files, both checking for strings and regular expressions. This can be restricted to finding strings before or after another string in the file.
24. #lca2015 Users & Groups describe user('flapjack') do it { should exist } it { should belong_to_group 'flapjack' } it { should have_home_directory ‘/opt/flapjack’ } it { should have_login_shell '/bin/bash' } end Serverspec provides full user and group support, including home directories, login shells and which users should belong to which group.
25. #lca2015 Mounted directories describe file('/') do it { should be_mounted.with( :type => 'ext4' )} it { should be_mounted.with( :options => { :rw => true } )} end You can check that filesystems are mounted as you expect, including the type and options. Here, the root filesystem - type ext4 - is mounted read-write.
26. #lca2015 PHP Configuration describe 'PHP config parameters' do context php_config('session.cache_expire') do its(:value) { should eq 180 } end end And, if you’re unfortunate enough to have to support PHP, you can check that its settings are correct too.
27. #lca2015 Making network changes
28. #lca2015 Iptables describe iptables do it { should have_rule('-P INPUT ACCEPT')} end Validate your IPtables rules. You can also limit the scope, and check that a rule is in a given chain.
29. #lca2015 Host Reachability describe host('target.example.jp') do it { should be_reachable } it { should be_reachable.with( :port => 22, :proto => 'tcp' )} end Serverspec doesn’t only check the host you’re on - it can also look at other hosts, and check that they’re reachable as well.
30. #lca2015 In the Windows world…
31. #lca2015 File versioning describe file('C:\\Windows\\System32\ \wuapi.dll') do it { should be_version('7.6.7600.256') } end
32. #lca2015 IIS Application Pool describe iis_app_pool('Default App Pool') do it { should exist } it { should have_dotnet_version('2.0') } end
33. #lca2015 IIS Website describe iis_website('Default Website') do it { should be_enabled } it { should be_running } it { should be_in_app_pool('Default App Pool') } it { should have_physical_path('C:\\inetpub\\www') } end
34. #lca2015 Windows Registry Key describe windows_registry_key('HKEY_USERS \S-1-5-21\Test MyKey') do it { should have_property_value('qword value', :type_qword, 'adff32') } it { should have_property_value('binary value', :type_binary, 'dfa0f066') } end Serverspec saves me remembering / looking up how to test these things myself.
35. #lca2015 Recap So you can see how Serverspec is great for validating your assumptions about how your servers are built, and how your application behaves after installation. For application developers, it’s not a replacement for the tests you may be doing already, such as unit tests, integration tests, but rather lets you test the overall behaviour of your app’s installation and that it runs successfully on all of your supported platforms.
36. #lca2015 We use Serverspec as part of the Flapjack project. Flapjack is a monitoring tool, it does notification routing + event processing. In lay-mans terms, it takes check output from Nagios, Sensu and friends, applies filters and works out who to notify and how. We are building packages for Debian, Ubuntu and CentOS using Omnibus. We use Serverspec to test our packages before they are made available to testers.
37. #lca2015 build package stop publish package Flapjack automated package testing Fail Pass We build packages, test them with Serverspec, then publish them. But what about the different operating systems and releases that we support? We use docker! (Yes, there’s the mandatory docker reference in this talk). We take a pristine image of each distribution, add the newly created package, and run Serverspec on each docker instance itself.
38. #lca2015 Serverspec deals with both RHEL and Debian families os[:family] # RedHat, Ubuntu, Debian and so on os[:release] # OS release version os[:arch] # i386 or x86_64 describe service('flapjack'), :if => os[:family] == 'ubuntu' do it { should be_enabled } end Not all tests are relevant to all operating systems. Serverspec allows you to add an if block to your test, which we use to run some of our tests only on Debian-based distros. Here’s a demonstration of our current serverspec test suite (including a few bonus fails).
39. Serverspec demo
40. #lca2015 Questions?
41. #lca2015 We’re Sarah Kowalik and Jesse Reynolds, engineers on the research and development team at Bulletproof. This is us on Karekare Beach in Auckland, a great place to go to avoid talk preparation. We highly recommend you visit while you’re in town if you’re not from these parts.
42. #lca2015 Big thanks to Bulletproof for sponsoring the development of Flapjack. Bulletproof uses flapjack in production.
43. #lca2015 Thank you! http://serverspec.org https://github.com/flapjack/omnibus-flapjack/tree/ master/spec/serverspec http://rspec.info/ http://flapjack.io http://bulletproof.net Liked the talk? Let @SarahKowalik & @jessereynolds know!