An introduction to testing infrastructure

Transcript

1. Testing Infrastructure Puppet Labs Gareth Rushgrove An Introduction

2. If Infrastructure is Code… Then we’re programming like the 80s

3. Replace constant meetings interpreting policy with testable code Gareth Rushgrove

4. Replace occasionally accessed spreadsheets with constantly running programs Gareth Rushgrove

5. Replace a hand drawn map of your system with constant

   measurement of reality Gareth Rushgrove

6. A Simple Example That’s also pretty useful

7. Who operates a web server? Gareth Rushgrove

8. Who operates a correctly configured web server? Gareth Rushgrove

9. Gareth Rushgrove A project to catalogue sensible HTTP headers for

   ensuring security features are enabled in browsers

10. It doesn’t matter which web server you’re using Gareth Rushgrove

11. Access Control Allow Origin Content Security Policy Cross Domain Meta

    Policy NoSniff Server Information Gareth Rushgrove - - - - -

12. Strict Transport Security UTF-8 Character Encoding X-Frame-Options X-Powered-By X-XSS-Protection Gareth

    Rushgrove - - - - -

13. Black box testing is a method of software testing that

    examines the functionality of an application without peering into its internal structures or workings Gareth Rushgrove

14. HANDS ON

15. https://goo.gl/6HKO1n

16. Another Simple Example That’s maybe less useful but makes a

    point

17. White box testing is a method of testing software that

    tests internal structures or workings of an application Gareth Rushgrove

18. Is a particular service running on this host? Gareth Rushgrove

19. Is a user and group setup on particular machine? Gareth

    Rushgrove

20. Is a particular piece of software installed on a machine?

    Gareth Rushgrove

21. EXAMPLE 2 Test service status over SSH with Go

22. Making Tests Reusable Don’t repeat yourself too much

23. Lots of great interactive tools around for testing Gareth Rushgrove

24. Wrapping them in automated tests allows for sharing domain knowledge

    Gareth Rushgrove

25. Discovered open port 22/tcp on 45.56.74.113 Completed Connect Scan at

    07:09, 3.31s elapsed (12 total ports) Nmap scan report for www.puppetlabs.com (45.56.74.113) Host is up (0.082s latency). rDNS record for 45.56.74.113: li924-113.members.linode.com PORT STATE SERVICE 20/tcp filtered ftp-data 21/tcp filtered ftp 22/tcp open ssh 23/tcp filtered telnet 25/tcp filtered smtp 80/tcp open http 110/tcp filtered pop3 443/tcp open https 512/tcp filtered exec 522/tcp filtered ulp 1080/tcp filtered socks 8080/tcp open http-proxy Standard nmap output requires manual analysis

26. it 'has only a limited number of open ports' do

    expect(@open_ports.count).to eq(3) end it 'exposes a web server' do expect(@open_ports).to include('80/tcp') expect(@open_ports).to include('443/tcp') end it 'exposes an SSH server' do expect(@open_ports).to include('22/tcp') end it 'rejects email traffic' do expect(@closed_ports).to include('25/tcp') end Using a unit testing framework we can make explicit assertions

27. www.puppetlabs.com from 192.168.1.10 has only a limited number of open

    ports (FAILED - 3) exposes a web server exposes an SSH server rejects accept email traffic (FAILED - 4) Anyone can run the tests and understand what is expected and what is currently broken

28. EXAMPLE 3 Test network ports with Nmap and Python

29. Make sharable tests suites that can take arguments Gareth Rushgrove

30. Taking Advantage of APIs Providing a surface area for tests

31. APIs provide a good interface for black box tests Gareth

    Rushgrove

32. CMDB assertions Gareth Rushgrove

33. PuppetDB assertions Gareth Rushgrove

34. (expect installed-on-all-clients? "auditd") A one line test to check auditd

    is installed everywhere

35. (expect (every? ubuntu? (facts "operatingsystem"))) A one line test to

    check all machines are using the permitted Operating System

36. (expect latest-on-all-clients? "openssh") A one line test to check we’re

    running the latest version of Open SSH everywhere

37. (expect running-on-all-clients? "selinux") Because people seem to like to disable

    SELinux

38. EXAMPLE 4 Test Cloud Infrastructure with Clojure

39. Ready Made Tools Open Source is pretty great

40. Gareth Rushgrove Serverspec makes assertions by running commands on remote

    machines via SSH, WinRM, Docker exec

41. EXAMPLE 5 Serverspec example using Docker

42. Gareth Rushgrove Infrataster provides common helpers for web infrastructure black

    box testing

43. EXAMPLE 6 Infrataster example

44. Gareth Rushgrove Servpeek is a new tool in Go, which

    would allow for shippable binaries with assertions

45. Servpeek test for asserting a package is installed

46. Gareth Rushgrove Testinfra is a Python tool for testing infrastrure,

    inspired by Serverspec

47. Testinfra provides primitives like File, Service, Package for making assertions

    against

48. Gareth Rushgrove BDD Security is a cucumber-based testing tool aimed

    specifically at testing security features of systems

49. An example test for checking for SQL injection vulnerabilities Scenario:

    The application should not contain SQL injection vulnerabilities Meta: @id scan_sql_injection @cwe-89 Given a scanner with all policies disabled And the SQL-Injection policy is enabled And the attack strength is set to High And the alert threshold is set to Low When the scanner is run And the XML report is written to the file sql_injection.xml Then no Medium or higher risk vulnerabilities should be present

50. An example test for checking for XSS. Lots more build-in

    Scenario: The application should not contain XSS vulnerabilities Meta: @id scan_sql_injection @cwe-89 Given a scanner with all policies disabled And the Cross-Site-Scripting policy is enabled And the attack strength is set to High When the scanner is run Then no Medium or higher risk vulnerabilities should be present

51. Gareth Rushgrove

52. Common cucumber stories with integration with nmap, sslyze, sqlmap, garmr

    and more

53. Testing Infrastructure as Code Mature testing tools in the configuration

    management space

54. EXAMPLE 7 An example using Test Kitchen, Bats and Serverspec

55. EXAMPLE 8 An example using Beaker and Azure

56. Real World Examples A few people are already sharing

57. Smokey a set of cucumber tests for GOV.UK features

58. Lots of scenarios for individual applications

59. Integrates those tests with the Icinga monitoring stack

60. Acceptance tests for the use of a CDN, again from

    GOV.UK

61. Serverspec tests for verifying an OpenStack setup

62. Conclusions Hopefully a few things to take away

63. Writing tests for infrastructure is easy Gareth Rushgrove

64. Deciding what tests to write is still hard Gareth Rushgrove

65. Retroactively writing tests is harder than writing them at the

    time Gareth Rushgrove

66. Sharing tests will help everyone learn (including you) Gareth Rushgrove

67. Questions? And thanks for listening