Everything as Code
Puppet Labs
Gareth Rushgrove
Examples, practices and issues
with all that software
Slide 2
Slide 2 text
Gareth Rushgrove
@garethr
Slide 3
Slide 3 text
Gareth Rushgrove
Slide 4
Slide 4 text
Context
Whether you’re in product
development or enterprise IT
Slide 5
Slide 5 text
Shadow IT
Gareth Rushgrove
Control
VS
Slide 6
Slide 6 text
Self-service
Gareth Rushgrove
Control
VS
Slide 7
Slide 7 text
VS
Rapid software
development
Gareth Rushgrove
Control
Slide 8
Slide 8 text
VS
Continuous
deployment
Gareth Rushgrove
Control
Slide 9
Slide 9 text
VS
New tools and
technologies
Gareth Rushgrove
Control
Slide 10
Slide 10 text
If Amazon release to production
every 11.6 seconds, how often
does the Change Approval
Board meet?
Gareth Rushgrove
http://assets.en.oreilly.com/1/event/60/Velocity%20Culture%20Presentation.pdf
Slide 11
Slide 11 text
The gap between the leading
edge and every-one-else
is growing
Gareth Rushgrove
Slide 12
Slide 12 text
30x
Gareth Rushgrove
More frequent
deployments
Faster lead times
than their peers
200x
2015 State of DevOps Report
Slide 13
Slide 13 text
60x
Gareth Rushgrove
Change
success rate
Faster mean
time to recover
168x
2015 State of DevOps Report
Slide 14
Slide 14 text
“We went from all-hands-on-
deck, war-room sort of
deployments to non-events”
Gareth Rushgrove
Jez Miller, Heartland Payment Systems
Slide 15
Slide 15 text
Siloed teams
Long cycle times
Poor visibility
Manual processes
Gareth Rushgrove
Slide 16
Slide 16 text
Siloed teams
Long cycle times
Poor visibility
Manual processes
Gareth Rushgrove
Cross-functional teams
Short cycle times
Fast feedback
Automated processes
Slide 17
Slide 17 text
Change control
And why code has advantages
for managing change
Avoiding spreadsheets as the
source of truth
Gareth Rushgrove
Slide 20
Slide 20 text
Going faster, without losing
confidence or increasing risk
Gareth Rushgrove
Slide 21
Slide 21 text
Gareth Rushgrove
Then
Move fast and break things
Slide 22
Slide 22 text
Move fast with stable infra
Gareth Rushgrove
Now
Slide 23
Slide 23 text
Gareth Rushgrove
Regular releases reduce risk
Slide 24
Slide 24 text
Applying software development
practices to change management
Gareth Rushgrove
Slide 25
Slide 25 text
Source code can be checked into
version control, so clear
understanding of who changed
what and when
Gareth Rushgrove
Slide 26
Slide 26 text
Strong controls around a central
code repository
Gareth Rushgrove
Slide 27
Slide 27 text
Single artefacts representing
change. Can be signed along
with accompanying metadata
Gareth Rushgrove
Slide 28
Slide 28 text
Changes can be made through a
pipeline, everything is visible and
nothing is adhoc
Gareth Rushgrove
Slide 29
Slide 29 text
Gareth Rushgrove
Login to a
computer
Make
change
Slide 30
Slide 30 text
Gareth Rushgrove
Write the
code
Check
syntax
Check
style
Unit tests
Acceptance
tests
Deploy
Slide 31
Slide 31 text
Gareth Rushgrove
Write the
code
Check
syntax
Check
style
Unit tests
Acceptance
tests
Deploy Deploy to
staging
Preflight
checks
Slide 32
Slide 32 text
Gareth Rushgrove
Write the
code
Check
syntax
Check
style
Unit tests
Acceptance
tests
Deploy
Security
tests
Slide 33
Slide 33 text
Automate where possible, so
manual effort is focused on
high risk work
Gareth Rushgrove
Slide 34
Slide 34 text
Gareth Rushgrove
Acceptance
tests
Deploy
Security
tests
Manual
review by
security
Automated
security
tests
Automated
security
tests
Is this a high risk change?
Slide 35
Slide 35 text
What is a high risk change?
- Change to firewall policy?
- Change to authentication code?
- Out of hours change?
- New user added?
Gareth Rushgrove
Slide 36
Slide 36 text
The important thing is the
discussion with colleagues in
development, operations, etc.
Gareth Rushgrove
Slide 37
Slide 37 text
Where is code
eating the world
Examples of code in use today
Slide 38
Slide 38 text
Infrastructure
as code
Slide 39
Slide 39 text
Gareth Rushgrove
The
architecture
docs
Your mental
model
Reality
The
developers
mental
model
= = = =
Slide 40
Slide 40 text
Gareth Rushgrove
The
architecture
docs
Your mental
model
Reality
The
developers
mental
model
= = = =
Slide 41
Slide 41 text
If you don’t know the state of your
system how can you trust a given
change will work?
Gareth Rushgrove
Slide 42
Slide 42 text
Gareth Rushgrove
Slide 43
Slide 43 text
Gareth Rushgrove
Slide 44
Slide 44 text
Gareth Rushgrove
Slide 45
Slide 45 text
No content
Slide 46
Slide 46 text
Gareth Rushgrove
Slide 47
Slide 47 text
- Cut down provisioning time
- Single way of changing all systems
- Detect and remediate config drift
Gareth Rushgrove
Slide 48
Slide 48 text
Acceptance testing
Slide 49
Slide 49 text
From manual runbooks to
executable specifications
Gareth Rushgrove
Slide 50
Slide 50 text
describe 'azure_vm_classic when creating a machine' do
include_context 'with certificate copied to system under test'
include_context 'with a known name and storage account name'
include_context 'with known network'
it_behaves_like 'an idempotent resource'
include_context 'destroy left-over created resources after use'
it 'should have the correct size' do
expect(@machine.role_size).to eq(@config[:optional][:size])
end
it 'should have the correct deployment name' do
expect(@machine.deployment_name).to eq(@config[:optional][:deployment])
end
it 'should have the correct cloud service name' do
expect(@machine.cloud_service_name).to eq(@config[:optional]
[:cloud_service])
end
Reusing unit testing frameworks to
automate manual checks
Slide 51
Slide 51 text
Higher-level tools exist with more
human language interfaces
Scenario: The application should not contain SQL injection vulnerabilities
Meta: @id scan_sql_injection @cwe-89
Given a scanner with all policies disabled
And the SQL-Injection policy is enabled
And the attack strength is set to High
And the alert threshold is set to Low
When the scanner is run
And the XML report is written to the file sql_injection.xml
Then no Medium or higher risk vulnerabilities should be present
Slide 52
Slide 52 text
Gareth Rushgrove
GOV.UK CDN Acceptance tests
Slide 53
Slide 53 text
- Reduce cost and time per release
- Make testing repeatable
- Make running tests accessible
Gareth Rushgrove
Slide 54
Slide 54 text
Confirming
network state
Slide 55
Slide 55 text
Discovered open port 22/tcp on 45.56.74.113
Completed Connect Scan at 07:09, 3.31s elapsed (12 total ports)
Nmap scan report for www.puppetlabs.com (45.56.74.113)
Host is up (0.082s latency).
rDNS record for 45.56.74.113: li924-113.members.linode.com
PORT STATE SERVICE
20/tcp filtered ftp-data
21/tcp filtered ftp
22/tcp open ssh
23/tcp filtered telnet
25/tcp filtered smtp
80/tcp open http
110/tcp filtered pop3
443/tcp open https
512/tcp filtered exec
522/tcp filtered ulp
1080/tcp filtered socks
8080/tcp open http-proxy
Standard nmap output requires manual analysis
Slide 56
Slide 56 text
it 'has only a limited number of open ports' do
expect(@open_ports.count).to eq(3)
end
it 'exposes a web server' do
expect(@open_ports).to include('80/tcp')
expect(@open_ports).to include('443/tcp')
end
it 'exposes an SSH server' do
expect(@open_ports).to include('22/tcp')
end
it 'rejects email traffic' do
expect(@closed_ports).to include('25/tcp')
end
Using a unit testing framework we can make
explicit assertions
Slide 57
Slide 57 text
www.puppetlabs.com from 192.168.1.10
has only a limited number of open ports (FAILED - 3)
exposes a web server
exposes an SSH server
rejects accept email traffic (FAILED - 4)
Anyone can run the tests and understand what is expected
and what is currently broken
Slide 58
Slide 58 text
Code allows for more than
just using tools
Gareth Rushgrove
Slide 59
Slide 59 text
Answering common
questions
Slide 60
Slide 60 text
(expect running-on-all-clients? "selinux")
Because people seem to like to disable SELinux
Slide 61
Slide 61 text
(expect (every? redhat? (facts "operatingsystem")))
A one line test to check all machines are using the
permitted Operating System
Slide 62
Slide 62 text
(expect latest-on-all-clients? "openssh")
A one line test to check we’re running the latest
version of Open SSH
Slide 63
Slide 63 text
SELECT uid, name
FROM listening_ports l, processes p
WHERE l.pid=p.pid;
Different tools and interfaces exist. OSquery for example
uses SQL to query infrastructure state
Slide 64
Slide 64 text
Creating your own domain
specific language
Gareth Rushgrove
Slide 65
Slide 65 text
Should architects
write code?
A relevant tangent
Slide 66
Slide 66 text
Gareth Rushgrove
IEEE Why architects should code
Slide 67
Slide 67 text
Developers have more difficulty
implementing the architectural
choices (e.g. patterns/tactics)
than functional features.
Gareth Rushgrove
Slide 68
Slide 68 text
Non-architecture savvy
developers introduce more
defects into architecturally-
significant code snippets than
architecture-savvy developers.
Gareth Rushgrove
Slide 69
Slide 69 text
The answers isn’t to
architect harder
Gareth Rushgrove
Slide 70
Slide 70 text
More powerpoint will not
fix the problem
Gareth Rushgrove
Slide 71
Slide 71 text
When software architects write
code, the number of defects in
the tactical fragments of the
systems will be reduced.
Gareth Rushgrove
Slide 72
Slide 72 text
The doesn’t mean you should
have written more code. Software
and architecture are changing
Gareth Rushgrove
Slide 73
Slide 73 text
Sharp edges
Problems to avoid when
introducing code everywhere
Slide 74
Slide 74 text
Speed kills too
Slide 75
Slide 75 text
How to lose $177,222 a second
for 45 minutes
Gareth Rushgrove
Slide 76
Slide 76 text
Gareth Rushgrove
Knight Capital Americas LLC
Slide 77
Slide 77 text
Failure cases are hard. Controls
need to operate at the speed
of software.
Gareth Rushgrove
Slide 78
Slide 78 text
Exacerbating risk
Slide 79
Slide 79 text
The empowered developer
problem
Gareth Rushgrove
Slide 80
Slide 80 text
The empowered developer
systems administrator problem
Gareth Rushgrove
Slide 81
Slide 81 text
The importance of teams
over individuals
Gareth Rushgrove
Slide 82
Slide 82 text
The importance of friction
Gareth Rushgrove
Slide 83
Slide 83 text
As people outside the control loop
architects are well positioned to
introduce the right level of friction
Gareth Rushgrove
Slide 84
Slide 84 text
Hiring and skills
Slide 85
Slide 85 text
Skilled developers are expensive
Gareth Rushgrove
Slide 86
Slide 86 text
Skilled developers are in demand
Gareth Rushgrove
Slide 87
Slide 87 text
Choosing technology to
attract developers might be
a valid strategy, but has a cost
Gareth Rushgrove
Slide 88
Slide 88 text
Gareth Rushgrove
Choose boring technology
Slide 89
Slide 89 text
Realise that architecture has
a direct influence on hiring
and retention
Gareth Rushgrove
Slide 90
Slide 90 text
Conclusions
Things to ponder later
Slide 91
Slide 91 text
Gareth Rushgrove
Software is eating the world
Slide 92
Slide 92 text
Gareth Rushgrove
Developers as the new kingmakers
Slide 93
Slide 93 text
Architect as influencer of what
code gets written
Gareth Rushgrove
Slide 94
Slide 94 text
Architect as custodian of
code safety
Gareth Rushgrove
Slide 95
Slide 95 text
Architect as cultural
change agent
Gareth Rushgrove