An introduction to testing infrastructure

Talk from #VelocityConf 2015 introducing testing infrastructure via simple unit tests to integrating with interactive tools, APIs and tools like serverspec, testinfra, BDD Security and more.

Transcript

1. Testing Infrastructure Puppet Labs Gareth Rushgrove An Introduction
2. If Infrastructure is Code… Then we’re programming like the 80s
3. Replace constant meetings interpreting policy with testable code Gareth Rushgrove
4. Replace occasionally accessed spreadsheets with constantly running programs Gareth Rushgrove
5. Replace a hand drawn map of your system with constant measurement of reality Gareth Rushgrove
6. A Simple Example That’s also pretty useful
7. Who operates a web server? Gareth Rushgrove
8. Who operates a correctly configured web server? Gareth Rushgrove
9. Gareth Rushgrove A project to catalogue sensible HTTP headers for ensuring security features are enabled in browsers
10. It doesn’t matter which web server you’re using Gareth Rushgrove
11. Access Control Allow Origin Content Security Policy Cross Domain Meta Policy NoSniff Server Information Gareth Rushgrove - - - - -
12. Strict Transport Security UTF-8 Character Encoding X-Frame-Options X-Powered-By X-XSS-Protection Gareth Rushgrove - - - - -
13. Black box testing is a method of software testing that examines the functionality of an application without peering into its internal structures or workings Gareth Rushgrove
14. HANDS ON
15. https://goo.gl/6HKO1n
16. Another Simple Example That’s maybe less useful but makes a point
17. White box testing is a method of testing software that tests internal structures or workings of an application Gareth Rushgrove
18. Is a particular service running on this host? Gareth Rushgrove
19. Is a user and group setup on particular machine? Gareth Rushgrove
20. Is a particular piece of software installed on a machine? Gareth Rushgrove
21. EXAMPLE 2 Test service status over SSH with Go
22. Making Tests Reusable Don’t repeat yourself too much
23. Lots of great interactive tools around for testing Gareth Rushgrove
24. Wrapping them in automated tests allows for sharing domain knowledge Gareth Rushgrove
25. Discovered open port 22/tcp on 45.56.74.113 Completed Connect Scan at 07:09, 3.31s elapsed (12 total ports) Nmap scan report for www.puppetlabs.com (45.56.74.113) Host is up (0.082s latency). rDNS record for 45.56.74.113: li924-113.members.linode.com PORT STATE SERVICE 20/tcp filtered ftp-data 21/tcp filtered ftp 22/tcp open ssh 23/tcp filtered telnet 25/tcp filtered smtp 80/tcp open http 110/tcp filtered pop3 443/tcp open https 512/tcp filtered exec 522/tcp filtered ulp 1080/tcp filtered socks 8080/tcp open http-proxy Standard nmap output requires manual analysis
26. it 'has only a limited number of open ports' do expect(@open_ports.count).to eq(3) end it 'exposes a web server' do expect(@open_ports).to include('80/tcp') expect(@open_ports).to include('443/tcp') end it 'exposes an SSH server' do expect(@open_ports).to include('22/tcp') end it 'rejects email traffic' do expect(@closed_ports).to include('25/tcp') end Using a unit testing framework we can make explicit assertions
27. www.puppetlabs.com from 192.168.1.10 has only a limited number of open ports (FAILED - 3) exposes a web server exposes an SSH server rejects accept email traffic (FAILED - 4) Anyone can run the tests and understand what is expected and what is currently broken
28. EXAMPLE 3 Test network ports with Nmap and Python
29. Make sharable tests suites that can take arguments Gareth Rushgrove
30. Taking Advantage of APIs Providing a surface area for tests
31. APIs provide a good interface for black box tests Gareth Rushgrove
32. CMDB assertions Gareth Rushgrove
33. PuppetDB assertions Gareth Rushgrove
34. (expect installed-on-all-clients? "auditd") A one line test to check auditd is installed everywhere
35. (expect (every? ubuntu? (facts "operatingsystem"))) A one line test to check all machines are using the permitted Operating System
36. (expect latest-on-all-clients? "openssh") A one line test to check we’re running the latest version of Open SSH everywhere
37. (expect running-on-all-clients? "selinux") Because people seem to like to disable SELinux
38. EXAMPLE 4 Test Cloud Infrastructure with Clojure
39. Ready Made Tools Open Source is pretty great
40. Gareth Rushgrove Serverspec makes assertions by running commands on remote machines via SSH, WinRM, Docker exec
41. EXAMPLE 5 Serverspec example using Docker
42. Gareth Rushgrove Infrataster provides common helpers for web infrastructure black box testing
43. EXAMPLE 6 Infrataster example
44. Gareth Rushgrove Servpeek is a new tool in Go, which would allow for shippable binaries with assertions
45. Servpeek test for asserting a package is installed
46. Gareth Rushgrove Testinfra is a Python tool for testing infrastrure, inspired by Serverspec
47. Testinfra provides primitives like File, Service, Package for making assertions against
48. Gareth Rushgrove BDD Security is a cucumber-based testing tool aimed specifically at testing security features of systems
49. An example test for checking for SQL injection vulnerabilities Scenario: The application should not contain SQL injection vulnerabilities Meta: @id scan_sql_injection @cwe-89 Given a scanner with all policies disabled And the SQL-Injection policy is enabled And the attack strength is set to High And the alert threshold is set to Low When the scanner is run And the XML report is written to the file sql_injection.xml Then no Medium or higher risk vulnerabilities should be present
50. An example test for checking for XSS. Lots more build-in Scenario: The application should not contain XSS vulnerabilities Meta: @id scan_sql_injection @cwe-89 Given a scanner with all policies disabled And the Cross-Site-Scripting policy is enabled And the attack strength is set to High When the scanner is run Then no Medium or higher risk vulnerabilities should be present
51. Gareth Rushgrove
52. Common cucumber stories with integration with nmap, sslyze, sqlmap, garmr and more
53. Testing Infrastructure as Code Mature testing tools in the configuration management space
54. EXAMPLE 7 An example using Test Kitchen, Bats and Serverspec
55. EXAMPLE 8 An example using Beaker and Azure
56. Real World Examples A few people are already sharing
57. Smokey a set of cucumber tests for GOV.UK features
58. Lots of scenarios for individual applications
59. Integrates those tests with the Icinga monitoring stack
60. Acceptance tests for the use of a CDN, again from GOV.UK
61. Serverspec tests for verifying an OpenStack setup
62. Conclusions Hopefully a few things to take away
63. Writing tests for infrastructure is easy Gareth Rushgrove
64. Deciding what tests to write is still hard Gareth Rushgrove
65. Retroactively writing tests is harder than writing them at the time Gareth Rushgrove
66. Sharing tests will help everyone learn (including you) Gareth Rushgrove
67. Questions? And thanks for listening