Maintaining control by letting go - security and devops

Slide 1

Slide 1 text

Maintaining control by letting go Puppet Labs Gareth Rushgrove Security in a devops world

Slide 2

Slide 2 text

@garethr

Slide 3

Slide 3 text

Gareth Rushgrove

Slide 4

Slide 4 text

Introduction

Slide 5

Slide 5 text

We can build more secure systems, not just react to problems better Gareth Rushgrove

Slide 6

Slide 6 text

Shadow IT Gareth Rushgrove Security VS

Slide 7

Slide 7 text

Self-service Gareth Rushgrove Security VS

Slide 8

Slide 8 text

VS Rapid software development Gareth Rushgrove Security

Slide 9

Slide 9 text

VS Rapid software deployment Gareth Rushgrove Security

Slide 10

Slide 10 text

VS New tools and technologies Gareth Rushgrove Security

Slide 11

Slide 11 text

Security as a solitary gatekeeper for all changes to production doesn’t scale Gareth Rushgrove

Slide 12

Slide 12 text

Automation can help scale security expertise Gareth Rushgrove

Slide 13

Slide 13 text

Automation is also the perfect middle-ground to build better relationships between security, development and operations Gareth Rushgrove

Slide 14

Slide 14 text

This talk

Slide 15

Slide 15 text

The importance of shared tools to collaboration Gareth Rushgrove

Slide 16

Slide 16 text

Examples of applying infrastructure as code, and other software practices to security problems Gareth Rushgrove

Slide 17

Slide 17 text

What the security community can learn from the successes of Devops Gareth Rushgrove

Slide 18

Slide 18 text

WARNING This talk will feature code but is not about software development Gareth Rushgrove

Slide 19

Slide 19 text

Sharing tools and practices Bringing developers, operations and security together

Slide 20

Slide 20 text

Why share? - Common language between teams - Builds trust amongst team members - Avoid duplication of effort and data - Limits the number of systems Gareth Rushgrove

Slide 21

Slide 21 text

Example: Infrastructure as Code Gareth Rushgrove

Slide 22

Slide 22 text

Gareth Rushgrove The architecture docs Your mental model Reality The developers mental model = = = =

Slide 23

Slide 23 text

Gareth Rushgrove The architecture docs Your mental model Reality The developers mental model = = = =

Slide 24

Slide 24 text

Gareth Rushgrove

Slide 25

Slide 25 text

Gareth Rushgrove Puppet code Reality =

Slide 26

Slide 26 text

An unambiguous description of your infrastructure, stored in version control Gareth Rushgrove

Slide 27

Slide 27 text

Puppet is a domain speciﬁc language, designed to describe your infrastructure conﬁguration

Slide 28

Slide 28 text

Gareth Rushgrove From ﬁles up to higher level constructs like applications running in IIS

Slide 29

Slide 29 text

And from host level resources to remote resources like DNS records or AWS Autoscaling Groups

Slide 30

Slide 30 text

Machines check they match the model every half an hour, and automatically remediate and report any differences Gareth Rushgrove

Slide 31

Slide 31 text

Gareth Rushgrove You can see what is changing across your infrastructure, all from the Puppet Enterprise Console

Slide 32

Slide 32 text

Shared resource between operations and development Gareth Rushgrove

Slide 33

Slide 33 text

Shared resource between operations, development, security and anyone with a stake in the infrastructure Gareth Rushgrove

Slide 34

Slide 34 text

“The constant discussion between software and infrastructure teams really helps us proactively ﬁnd issues before deploying to production.”

Slide 35

Slide 35 text

All of the conﬁguration information ends up in PuppetDB. How might that be of interest to a security team? Gareth Rushgrove

Slide 36

Slide 36 text

(The framework for this is about 20 lines of code and could be written by any good developer in an hour) Gareth Rushgrove

Slide 37

Slide 37 text

(expect installed-on-all-clients? "auditd") A one line test to check auditd is installed everywhere

Slide 38

Slide 38 text

(expect running-on-all-clients? "selinux") Because people seem to like to disable SELinux

Slide 39

Slide 39 text

(expect (every? ubuntu? (facts "operatingsystem"))) A one line test to check all machines are using the permitted Operating System

Slide 40

Slide 40 text

(expect latest-on-all-clients? "openssh") A one line test to check we’re running the latest version of Open SSH

Slide 41

Slide 41 text

Hook the tests into your operations teams monitoring system and security can get alerts when tests fail Gareth Rushgrove

Slide 42

Slide 42 text

Combine existing operations data with developers willingness to write code and the security teams knowledge of potential threats Gareth Rushgrove

Slide 43

Slide 43 text

Embrace pipelines Security testing in continuous delivery pipelines

Slide 44

Slide 44 text

Gareth Rushgrove

Slide 45

Slide 45 text

“We went from all-hands-on- deck, war-room sort of deployments to non-events” Gareth Rushgrove Jez Miller, Heartland Payment Systems, Puppet Enterprise Customer

Slide 46

Slide 46 text

Gareth Rushgrove Regular releases reduce risk

Slide 47

Slide 47 text

As applicable to changes to web applications as infrastructure, end user devices, network hardware and more Gareth Rushgrove

Slide 48

Slide 48 text

Gareth Rushgrove Login to a computer Make change

Slide 49

Slide 49 text

Gareth Rushgrove Write the code Check syntax Check style Unit tests Acceptance tests Deploy

Slide 50

Slide 50 text

Gareth Rushgrove Write the code Check syntax Check style Unit tests Acceptance tests Deploy Deploy to staging Preﬂight checks

Slide 51

Slide 51 text

Gareth Rushgrove Write the code Check syntax Check style Unit tests Acceptance tests Deploy Security tests

Slide 52

Slide 52 text

Automate where possible, so manual effort is focused on high risk work Gareth Rushgrove

Slide 53

Slide 53 text

Gareth Rushgrove Acceptance tests Deploy Security tests Manual review by security Automated security tests Automated security tests Is this a high risk change?

Slide 54

Slide 54 text

What is a high risk change? - Change to ﬁrewall policy? - Change to authentication code? - Change to SSH conﬁguration? - New user added? Gareth Rushgrove

Slide 55

Slide 55 text

The important thing is the discussion with colleagues in development and operations Gareth Rushgrove

Slide 56

Slide 56 text

Turn security practices into standard processes Gareth Rushgrove

Slide 57

Slide 57 text

Turn standard processes into automated steps Gareth Rushgrove

Slide 58

Slide 58 text

Unit tests for security concerns Applying development practices to security

Slide 59

Slide 59 text

Once you know about the pipeline, you can add steps related to security Gareth Rushgrove

Slide 60

Slide 60 text

Ensure security policy is implemented in practice Gareth Rushgrove

Slide 61

Slide 61 text

Catch security problems before they hit production Gareth Rushgrove

Slide 62

Slide 62 text

Replace occasionally accessed spreadsheets with constantly running code Gareth Rushgrove

Slide 63

Slide 63 text

Useful in different contexts: - Production monitoring system - Tests in development - Smoke tests for deployment Gareth Rushgrove

Slide 64

Slide 64 text

Test your network Because everyone likes an open port Gareth Rushgrove

Slide 65

Slide 65 text

Gareth Rushgrove

Slide 66

Slide 66 text

Discovered open port 22/tcp on 45.56.74.113 Completed Connect Scan at 07:09, 3.31s elapsed (12 total ports) Nmap scan report for www.puppetlabs.com (45.56.74.113) Host is up (0.082s latency). rDNS record for 45.56.74.113: li924-113.members.linode.com PORT STATE SERVICE 20/tcp filtered ftp-data 21/tcp filtered ftp 22/tcp open ssh 23/tcp filtered telnet 25/tcp filtered smtp 80/tcp open http 110/tcp filtered pop3 443/tcp open https 512/tcp filtered exec 522/tcp filtered ulp 1080/tcp filtered socks 8080/tcp open http-proxy Standard nmap output requires manual analysis

Slide 67

Slide 67 text

Gareth Rushgrove Security tests Run network security tests

Slide 68

Slide 68 text

it 'has only a limited number of open ports' do expect(@open_ports.count).to eq(3) end it 'exposes a web server' do expect(@open_ports).to include('80/tcp') expect(@open_ports).to include('443/tcp') end it 'exposes an SSH server' do expect(@open_ports).to include('22/tcp') end it 'rejects email traffic' do expect(@closed_ports).to include('25/tcp') end Using a unit testing framework we can make explicit assertions

Slide 69

Slide 69 text

www.puppetlabs.com from 192.168.1.10 has only a limited number of open ports (FAILED - 3) exposes a web server exposes an SSH server rejects accept email traffic (FAILED - 4) Anyone can run the tests and understand what is expected and what is currently broken

Slide 70

Slide 70 text

Test your supply chain Checking packages for vulnerabilities Gareth Rushgrove

Slide 71

Slide 71 text

Old versions of third libraries might have known vulnerabilities Gareth Rushgrove

Slide 72

Slide 72 text

Gareth Rushgrove rubysec/bundler-audit

Slide 73

Slide 73 text

Gareth Rushgrove Security tests Run network security tests Check for vulnerable packages

Slide 74

Slide 74 text

source "https://rubygems.org" gem "bundler-audit" gem "rake" gem "pry" group :vulnerable do gem "actionpack", "=3.2.10" end An example Gemﬁle, a list of ruby packages used in an application or environment, with a vulnerable package

Slide 75

Slide 75 text

Name: actionpack Version: 3.2.10 Advisory: OSVDB-103440 Name: actionpack Version: 3.2.10 Advisory: OSVDB-89026 Criticality: High URL: http://osvdb.org/show/osvdb/89026 Description: Ruby on Rails contains a flaw in params_parser.rb of the Action Pack. The issue is triggered when a type casting error occurs during the parsing of parameters. This may allow a remote attacker to potentially execute arbitrary code. Solution: upgrade to ~> 2.3.15, ~> 3.0.19, ~> 3.1.10, >= 3.2.11 An example report from bunder-audit

Slide 76

Slide 76 text

Test your web applications Checking for XSS, SQLi and more Gareth Rushgrove

Slide 77

Slide 77 text

Gareth Rushgrove continuumsecurity/bdd-security

Slide 78

Slide 78 text

An example test for checking for SQL injection vulnerabilities Scenario: The application should not contain SQL injection vulnerabilities Meta: @id scan_sql_injection @cwe-89 Given a scanner with all policies disabled And the SQL-Injection policy is enabled And the attack strength is set to High And the alert threshold is set to Low When the scanner is run And the XML report is written to the file sql_injection.xml Then no Medium or higher risk vulnerabilities should be present

Slide 79

Slide 79 text

An example test for checking for XSS. Lots more build-in Scenario: The application should not contain XSS vulnerabilities Meta: @id scan_sql_injection @cwe-89 Given a scanner with all policies disabled And the Cross-Site-Scripting policy is enabled And the attack strength is set to High When the scanner is run Then no Medium or higher risk vulnerabilities should be present

Slide 80

Slide 80 text

Test what you deploy, not just what you write Gareth Rushgrove

Slide 81

Slide 81 text

Gareth Rushgrove Security tests Run network security tests Check for vulnerable packages Run web application security tests

Slide 82

Slide 82 text

Gareth Rushgrove Security tests Run network security tests Check for vulnerable packages Run web application security tests Static analysis Metrics analysis SSL conﬁg

Slide 83

Slide 83 text

These are just quick examples. The point is automated testing and security go together incredibly well Gareth Rushgrove

Slide 84

Slide 84 text

And embracing the pipeline means you can add small steps easily Gareth Rushgrove

Slide 85

Slide 85 text

Conclusions

Slide 86

Slide 86 text

Focus efforts on the mechanism for change, over individual changes Gareth Rushgrove

Slide 87

Slide 87 text

Focus on steps in the pipeline. No single tool will ﬁx all problems Gareth Rushgrove

Slide 88

Slide 88 text

Favour tools (open source, in-house or commercial) with APIs that can be integrated into a pipeline Gareth Rushgrove

Slide 89

Slide 89 text

Developers have never been more interested in security, use that Gareth Rushgrove

Slide 90

Slide 90 text

Adding steps into a continuous delivery pipeline is the perfect opportunity for a conversation Gareth Rushgrove