Slide 1

Slide 1 text

Everything as Code Puppet Labs Gareth Rushgrove Examples, practices and issues with all that software

Slide 2

Slide 2 text

Gareth Rushgrove @garethr

Slide 3

Slide 3 text

Gareth Rushgrove

Slide 4

Slide 4 text

Context Whether you’re in product development or enterprise IT

Slide 5

Slide 5 text

Shadow IT Gareth Rushgrove Control VS

Slide 6

Slide 6 text

Self-service Gareth Rushgrove Control VS

Slide 7

Slide 7 text

VS Rapid software development Gareth Rushgrove Control

Slide 8

Slide 8 text

VS Continuous deployment Gareth Rushgrove Control

Slide 9

Slide 9 text

VS New tools and technologies Gareth Rushgrove Control

Slide 10

Slide 10 text

If Amazon release to production every 11.6 seconds, how often does the Change Approval Board meet? Gareth Rushgrove http://assets.en.oreilly.com/1/event/60/Velocity%20Culture%20Presentation.pdf

Slide 11

Slide 11 text

The gap between the leading edge and every-one-else is growing Gareth Rushgrove

Slide 12

Slide 12 text

30x Gareth Rushgrove More frequent deployments Faster lead times than their peers 200x 2015 State of DevOps Report

Slide 13

Slide 13 text

60x Gareth Rushgrove Change success rate Faster mean time to recover 168x 2015 State of DevOps Report

Slide 14

Slide 14 text

“We went from all-hands-on- deck, war-room sort of deployments to non-events” Gareth Rushgrove Jez Miller, Heartland Payment Systems

Slide 15

Slide 15 text

Siloed teams Long cycle times Poor visibility Manual processes Gareth Rushgrove

Slide 16

Slide 16 text

Siloed teams Long cycle times Poor visibility Manual processes Gareth Rushgrove Cross-functional teams Short cycle times Fast feedback Automated processes

Slide 17

Slide 17 text

Change control And why code has advantages for managing change

Slide 18

Slide 18 text

Change management, configuration management, supplier management, capacity management, request fulfilment, problem management, access management. Gareth Rushgrove

Slide 19

Slide 19 text

Avoiding spreadsheets as the source of truth Gareth Rushgrove

Slide 20

Slide 20 text

Going faster, without losing confidence or increasing risk Gareth Rushgrove

Slide 21

Slide 21 text

Gareth Rushgrove Then Move fast and break things

Slide 22

Slide 22 text

Move fast with stable infra Gareth Rushgrove Now

Slide 23

Slide 23 text

Gareth Rushgrove Regular releases reduce risk

Slide 24

Slide 24 text

Applying software development practices to change management Gareth Rushgrove

Slide 25

Slide 25 text

Source code can be checked into version control, so clear understanding of who changed what and when Gareth Rushgrove

Slide 26

Slide 26 text

Strong controls around a central code repository Gareth Rushgrove

Slide 27

Slide 27 text

Single artefacts representing change. Can be signed along with accompanying metadata Gareth Rushgrove

Slide 28

Slide 28 text

Changes can be made through a pipeline, everything is visible and nothing is adhoc Gareth Rushgrove

Slide 29

Slide 29 text

Gareth Rushgrove Login to a computer Make change

Slide 30

Slide 30 text

Gareth Rushgrove Write the code Check syntax Check style Unit tests Acceptance tests Deploy

Slide 31

Slide 31 text

Gareth Rushgrove Write the code Check syntax Check style Unit tests Acceptance tests Deploy Deploy to staging Preflight checks

Slide 32

Slide 32 text

Gareth Rushgrove Write the code Check syntax Check style Unit tests Acceptance tests Deploy Security tests

Slide 33

Slide 33 text

Automate where possible, so manual effort is focused on high risk work Gareth Rushgrove

Slide 34

Slide 34 text

Gareth Rushgrove Acceptance tests Deploy Security tests Manual review by security Automated security tests Automated security tests Is this a high risk change?

Slide 35

Slide 35 text

What is a high risk change? - Change to firewall policy? - Change to authentication code? - Out of hours change? - New user added? Gareth Rushgrove

Slide 36

Slide 36 text

The important thing is the discussion with colleagues in development, operations, etc. Gareth Rushgrove

Slide 37

Slide 37 text

Where is code eating the world Examples of code in use today

Slide 38

Slide 38 text

Infrastructure as code

Slide 39

Slide 39 text

Gareth Rushgrove The architecture docs Your mental model Reality The developers mental model = = = =

Slide 40

Slide 40 text

Gareth Rushgrove The architecture docs Your mental model Reality The developers mental model = = = =

Slide 41

Slide 41 text

If you don’t know the state of your system how can you trust a given change will work? Gareth Rushgrove

Slide 42

Slide 42 text

Gareth Rushgrove

Slide 43

Slide 43 text

Gareth Rushgrove

Slide 44

Slide 44 text

Gareth Rushgrove

Slide 45

Slide 45 text

No content

Slide 46

Slide 46 text

Gareth Rushgrove

Slide 47

Slide 47 text

- Cut down provisioning time - Single way of changing all systems - Detect and remediate config drift Gareth Rushgrove

Slide 48

Slide 48 text

Acceptance testing

Slide 49

Slide 49 text

From manual runbooks to executable specifications Gareth Rushgrove

Slide 50

Slide 50 text

describe 'azure_vm_classic when creating a machine' do include_context 'with certificate copied to system under test' include_context 'with a known name and storage account name' include_context 'with known network' it_behaves_like 'an idempotent resource' include_context 'destroy left-over created resources after use' it 'should have the correct size' do expect(@machine.role_size).to eq(@config[:optional][:size]) end it 'should have the correct deployment name' do expect(@machine.deployment_name).to eq(@config[:optional][:deployment]) end it 'should have the correct cloud service name' do expect(@machine.cloud_service_name).to eq(@config[:optional] [:cloud_service]) end Reusing unit testing frameworks to automate manual checks

Slide 51

Slide 51 text

Higher-level tools exist with more human language interfaces Scenario: The application should not contain SQL injection vulnerabilities Meta: @id scan_sql_injection @cwe-89 Given a scanner with all policies disabled And the SQL-Injection policy is enabled And the attack strength is set to High And the alert threshold is set to Low When the scanner is run And the XML report is written to the file sql_injection.xml Then no Medium or higher risk vulnerabilities should be present

Slide 52

Slide 52 text

Gareth Rushgrove GOV.UK CDN Acceptance tests

Slide 53

Slide 53 text

- Reduce cost and time per release - Make testing repeatable - Make running tests accessible Gareth Rushgrove

Slide 54

Slide 54 text

Confirming network state

Slide 55

Slide 55 text

Discovered open port 22/tcp on 45.56.74.113 Completed Connect Scan at 07:09, 3.31s elapsed (12 total ports) Nmap scan report for www.puppetlabs.com (45.56.74.113) Host is up (0.082s latency). rDNS record for 45.56.74.113: li924-113.members.linode.com PORT STATE SERVICE 20/tcp filtered ftp-data 21/tcp filtered ftp 22/tcp open ssh 23/tcp filtered telnet 25/tcp filtered smtp 80/tcp open http 110/tcp filtered pop3 443/tcp open https 512/tcp filtered exec 522/tcp filtered ulp 1080/tcp filtered socks 8080/tcp open http-proxy Standard nmap output requires manual analysis

Slide 56

Slide 56 text

it 'has only a limited number of open ports' do expect(@open_ports.count).to eq(3) end it 'exposes a web server' do expect(@open_ports).to include('80/tcp') expect(@open_ports).to include('443/tcp') end it 'exposes an SSH server' do expect(@open_ports).to include('22/tcp') end it 'rejects email traffic' do expect(@closed_ports).to include('25/tcp') end Using a unit testing framework we can make explicit assertions

Slide 57

Slide 57 text

www.puppetlabs.com from 192.168.1.10 has only a limited number of open ports (FAILED - 3) exposes a web server exposes an SSH server rejects accept email traffic (FAILED - 4) Anyone can run the tests and understand what is expected and what is currently broken

Slide 58

Slide 58 text

Code allows for more than just using tools Gareth Rushgrove

Slide 59

Slide 59 text

Answering common questions

Slide 60

Slide 60 text

(expect running-on-all-clients? "selinux") Because people seem to like to disable SELinux

Slide 61

Slide 61 text

(expect (every? redhat? (facts "operatingsystem"))) A one line test to check all machines are using the permitted Operating System

Slide 62

Slide 62 text

(expect latest-on-all-clients? "openssh") A one line test to check we’re running the latest version of Open SSH

Slide 63

Slide 63 text

SELECT uid, name FROM listening_ports l, processes p WHERE l.pid=p.pid; Different tools and interfaces exist. OSquery for example uses SQL to query infrastructure state

Slide 64

Slide 64 text

Creating your own domain specific language Gareth Rushgrove

Slide 65

Slide 65 text

Should architects write code? A relevant tangent

Slide 66

Slide 66 text

Gareth Rushgrove IEEE Why architects should code

Slide 67

Slide 67 text

Developers have more difficulty implementing the architectural choices (e.g. patterns/tactics) than functional features. Gareth Rushgrove

Slide 68

Slide 68 text

Non-architecture savvy developers introduce more defects into architecturally- significant code snippets than architecture-savvy developers. Gareth Rushgrove

Slide 69

Slide 69 text

The answers isn’t to architect harder Gareth Rushgrove

Slide 70

Slide 70 text

More powerpoint will not fix the problem Gareth Rushgrove

Slide 71

Slide 71 text

When software architects write code, the number of defects in the tactical fragments of the systems will be reduced. Gareth Rushgrove

Slide 72

Slide 72 text

The doesn’t mean you should have written more code. Software and architecture are changing Gareth Rushgrove

Slide 73

Slide 73 text

Sharp edges Problems to avoid when introducing code everywhere

Slide 74

Slide 74 text

Speed kills too

Slide 75

Slide 75 text

How to lose $177,222 a second for 45 minutes Gareth Rushgrove

Slide 76

Slide 76 text

Gareth Rushgrove Knight Capital Americas LLC

Slide 77

Slide 77 text

Failure cases are hard. Controls need to operate at the speed of software. Gareth Rushgrove

Slide 78

Slide 78 text

Exacerbating risk

Slide 79

Slide 79 text

The empowered developer problem Gareth Rushgrove

Slide 80

Slide 80 text

The empowered developer systems administrator problem Gareth Rushgrove

Slide 81

Slide 81 text

The importance of teams over individuals Gareth Rushgrove

Slide 82

Slide 82 text

The importance of friction Gareth Rushgrove

Slide 83

Slide 83 text

As people outside the control loop architects are well positioned to introduce the right level of friction Gareth Rushgrove

Slide 84

Slide 84 text

Hiring and skills

Slide 85

Slide 85 text

Skilled developers are expensive Gareth Rushgrove

Slide 86

Slide 86 text

Skilled developers are in demand Gareth Rushgrove

Slide 87

Slide 87 text

Choosing technology to attract developers might be a valid strategy, but has a cost Gareth Rushgrove

Slide 88

Slide 88 text

Gareth Rushgrove Choose boring technology

Slide 89

Slide 89 text

Realise that architecture has a direct influence on hiring and retention Gareth Rushgrove

Slide 90

Slide 90 text

Conclusions Things to ponder later

Slide 91

Slide 91 text

Gareth Rushgrove Software is eating the world

Slide 92

Slide 92 text

Gareth Rushgrove Developers as the new kingmakers

Slide 93

Slide 93 text

Architect as influencer of what code gets written Gareth Rushgrove

Slide 94

Slide 94 text

Architect as custodian of code safety Gareth Rushgrove

Slide 95

Slide 95 text

Architect as cultural change agent Gareth Rushgrove

Slide 96

Slide 96 text

Questions? And thanks for listening