Pro Yearly is on sale from $80 to $50! »

An introduction to testing infrastructure

An introduction to testing infrastructure

Talk from #VelocityConf 2015 introducing testing infrastructure via simple unit tests to integrating with interactive tools, APIs and tools like serverspec, testinfra, BDD Security and more.


Gareth Rushgrove

October 28, 2015


  1. Testing Infrastructure Puppet Labs Gareth Rushgrove An Introduction

  2. If Infrastructure is Code… Then we’re programming like the 80s

  3. Replace constant meetings interpreting policy with testable code Gareth Rushgrove

  4. Replace occasionally accessed spreadsheets with constantly running programs Gareth Rushgrove

  5. Replace a hand drawn map of your system with constant

    measurement of reality Gareth Rushgrove
  6. A Simple Example That’s also pretty useful

  7. Who operates a web server? Gareth Rushgrove

  8. Who operates a correctly configured web server? Gareth Rushgrove

  9. Gareth Rushgrove A project to catalogue sensible HTTP headers for

    ensuring security features are enabled in browsers
  10. It doesn’t matter which web server you’re using Gareth Rushgrove

  11. Access Control Allow Origin Content Security Policy Cross Domain Meta

    Policy NoSniff Server Information Gareth Rushgrove - - - - -
  12. Strict Transport Security UTF-8 Character Encoding X-Frame-Options X-Powered-By X-XSS-Protection Gareth

    Rushgrove - - - - -
  13. Black box testing is a method of software testing that

    examines the functionality of an application without peering into its internal structures or workings Gareth Rushgrove
  14. HANDS ON


  16. Another Simple Example That’s maybe less useful but makes a

  17. White box testing is a method of testing software that

    tests internal structures or workings of an application Gareth Rushgrove
  18. Is a particular service running on this host? Gareth Rushgrove

  19. Is a user and group setup on particular machine? Gareth

  20. Is a particular piece of software installed on a machine?

    Gareth Rushgrove
  21. EXAMPLE 2 Test service status over SSH with Go

  22. Making Tests Reusable Don’t repeat yourself too much

  23. Lots of great interactive tools around for testing Gareth Rushgrove

  24. Wrapping them in automated tests allows for sharing domain knowledge

    Gareth Rushgrove
  25. Discovered open port 22/tcp on Completed Connect Scan at

    07:09, 3.31s elapsed (12 total ports) Nmap scan report for ( Host is up (0.082s latency). rDNS record for PORT STATE SERVICE 20/tcp filtered ftp-data 21/tcp filtered ftp 22/tcp open ssh 23/tcp filtered telnet 25/tcp filtered smtp 80/tcp open http 110/tcp filtered pop3 443/tcp open https 512/tcp filtered exec 522/tcp filtered ulp 1080/tcp filtered socks 8080/tcp open http-proxy Standard nmap output requires manual analysis
  26. it 'has only a limited number of open ports' do

    expect(@open_ports.count).to eq(3) end it 'exposes a web server' do expect(@open_ports).to include('80/tcp') expect(@open_ports).to include('443/tcp') end it 'exposes an SSH server' do expect(@open_ports).to include('22/tcp') end it 'rejects email traffic' do expect(@closed_ports).to include('25/tcp') end Using a unit testing framework we can make explicit assertions
  27. from has only a limited number of open

    ports (FAILED - 3) exposes a web server exposes an SSH server rejects accept email traffic (FAILED - 4) Anyone can run the tests and understand what is expected and what is currently broken
  28. EXAMPLE 3 Test network ports with Nmap and Python

  29. Make sharable tests suites that can take arguments Gareth Rushgrove

  30. Taking Advantage of APIs Providing a surface area for tests

  31. APIs provide a good interface for black box tests Gareth

  32. CMDB assertions Gareth Rushgrove

  33. PuppetDB assertions Gareth Rushgrove

  34. (expect installed-on-all-clients? "auditd") A one line test to check auditd

    is installed everywhere
  35. (expect (every? ubuntu? (facts "operatingsystem"))) A one line test to

    check all machines are using the permitted Operating System
  36. (expect latest-on-all-clients? "openssh") A one line test to check we’re

    running the latest version of Open SSH everywhere
  37. (expect running-on-all-clients? "selinux") Because people seem to like to disable

  38. EXAMPLE 4 Test Cloud Infrastructure with Clojure

  39. Ready Made Tools Open Source is pretty great

  40. Gareth Rushgrove Serverspec makes assertions by running commands on remote

    machines via SSH, WinRM, Docker exec
  41. EXAMPLE 5 Serverspec example using Docker

  42. Gareth Rushgrove Infrataster provides common helpers for web infrastructure black

    box testing
  43. EXAMPLE 6 Infrataster example

  44. Gareth Rushgrove Servpeek is a new tool in Go, which

    would allow for shippable binaries with assertions
  45. Servpeek test for asserting a package is installed

  46. Gareth Rushgrove Testinfra is a Python tool for testing infrastrure,

    inspired by Serverspec
  47. Testinfra provides primitives like File, Service, Package for making assertions

  48. Gareth Rushgrove BDD Security is a cucumber-based testing tool aimed

    specifically at testing security features of systems
  49. An example test for checking for SQL injection vulnerabilities Scenario:

    The application should not contain SQL injection vulnerabilities Meta: @id scan_sql_injection @cwe-89 Given a scanner with all policies disabled And the SQL-Injection policy is enabled And the attack strength is set to High And the alert threshold is set to Low When the scanner is run And the XML report is written to the file sql_injection.xml Then no Medium or higher risk vulnerabilities should be present
  50. An example test for checking for XSS. Lots more build-in

    Scenario: The application should not contain XSS vulnerabilities Meta: @id scan_sql_injection @cwe-89 Given a scanner with all policies disabled And the Cross-Site-Scripting policy is enabled And the attack strength is set to High When the scanner is run Then no Medium or higher risk vulnerabilities should be present
  51. Gareth Rushgrove

  52. Common cucumber stories with integration with nmap, sslyze, sqlmap, garmr

    and more
  53. Testing Infrastructure as Code Mature testing tools in the configuration

    management space
  54. EXAMPLE 7 An example using Test Kitchen, Bats and Serverspec

  55. EXAMPLE 8 An example using Beaker and Azure

  56. Real World Examples A few people are already sharing

  57. Smokey a set of cucumber tests for GOV.UK features

  58. Lots of scenarios for individual applications

  59. Integrates those tests with the Icinga monitoring stack

  60. Acceptance tests for the use of a CDN, again from

  61. Serverspec tests for verifying an OpenStack setup

  62. Conclusions Hopefully a few things to take away

  63. Writing tests for infrastructure is easy Gareth Rushgrove

  64. Deciding what tests to write is still hard Gareth Rushgrove

  65. Retroactively writing tests is harder than writing them at the

    time Gareth Rushgrove
  66. Sharing tests will help everyone learn (including you) Gareth Rushgrove

  67. Questions? And thanks for listening