Continuous Integration for Infrastructure

Slide 1

Slide 1 text

Continuous Integration for Infrastructure Craft Conference, Budapest, 2014 Gareth Rushgrove

Slide 2

Slide 2 text

Who (Who is this person?)

Slide 3

Slide 3 text

@garethr

Slide 4

Slide 4 text

UK Government Digital Service

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

What is Continuous Integration (And what has that to do with infrastructure)

Slide 8

Slide 8 text

A common software development practice Gareth Rushgrove

Slide 9

Slide 9 text

Test every commit Gareth Rushgrove

Slide 10

Slide 10 text

Get fast feedback on changes Gareth Rushgrove

Slide 11

Slide 11 text

Discover bugs early, when they are easier to ﬁx Gareth Rushgrove

Slide 12

Slide 12 text

Avoid working in isolation, encourage good team work Gareth Rushgrove

Slide 13

Slide 13 text

Lots of good tools to start with Gareth Rushgrove

Slide 14

Slide 14 text

Gareth Rushgrove jenkins-ci.org

Slide 15

Slide 15 text

Gareth Rushgrove jetbrains.com/teamcity

Slide 16

Slide 16 text

Gareth Rushgrove travis-ci.com

Slide 17

Slide 17 text

Gareth Rushgrove wercker.com

Slide 18

Slide 18 text

And lots more Gareth Rushgrove

Slide 19

Slide 19 text

Infrastructure as code means your infrastructure is software Gareth Rushgrove

Slide 20

Slide 20 text

So let’s apply traditional software engineering practices to infrastructure Gareth Rushgrove

Slide 21

Slide 21 text

Conﬁguration management! Acceptance testing! Verifying machine images! Load testing! Provisioning Gareth Rushgrove

Slide 22

Slide 22 text

Conﬁguration management! Acceptance testing! Verifying machine images ! Load testing! Provisioning Gareth Rushgrove

Slide 23

Slide 23 text

Conﬁguration management! Acceptance testing! Verifying machine images ! Load testing! Provisioning Gareth Rushgrove

Slide 24

Slide 24 text

Conﬁguration management! Acceptance testing! Verifying machine images! Load testing! Provisioning Gareth Rushgrove

Slide 25

Slide 25 text

Conﬁguration management! Acceptance testing! Verifying machine images! Load testing! Provisioning Gareth Rushgrove

Slide 26

Slide 26 text

Infrastructure testing as code (A quick wrecker example)

Slide 27

Slide 27 text

Deﬁne your continuous integration setup in code too Gareth Rushgrove

Slide 28

Slide 28 text

- install-packages: packages: ruby1.9.1-dev - script: name: install bundler code: sudo gem install bundler - bundle-install - script: name: print ruby version code: ruby —version wercker.yaml Gareth Rushgrove

Slide 29

Slide 29 text

Slide 30

Slide 30 text

View builds in a web browser Gareth Rushgrove

Slide 31

Slide 31 text

Gareth Rushgrove

Slide 32

Slide 32 text

Slide 33

Slide 33 text

Gareth Rushgrove

Slide 34

Slide 34 text

You can do similar things with Jenkins Job Builder and with Travis Gareth Rushgrove

Slide 35

Slide 35 text

Testing conﬁg management code (The easy bit)

Slide 36

Slide 36 text

Gareth Rushgrove

Slide 37

Slide 37 text

Both Chef and Puppet have good testing support Gareth Rushgrove

Slide 38

Slide 38 text

Write unit tests for modules or cookbooks Gareth Rushgrove

Slide 39

Slide 39 text

rspec-puppet or chefspec Gareth Rushgrove

Slide 40

Slide 40 text

class sample { file { '/tmp/sample': ensure => present, } } sample.pp Gareth Rushgrove

Slide 41

Slide 41 text

describe "sample" do it { should create_file('/tmp/sample') } end sample_spec.rb Gareth Rushgrove

Slide 42

Slide 42 text

sample should contain File[/tmp/sample] ! Finished in 0.3881 seconds 1 example, 0 failures Gareth Rushgrove

Slide 43

Slide 43 text

Linting, code coverage, syntax checking Gareth Rushgrove

Slide 44

Slide 44 text

Lots of good material available on this subject Gareth Rushgrove

Slide 45

Slide 45 text

Gareth Rushgrove

Slide 46

Slide 46 text

Gareth Rushgrove

Slide 47

Slide 47 text

Acceptance testing infrastructure code (Testing with real machines)

Slide 48

Slide 48 text

You don’t need to be using conﬁg management tools to test your code Gareth Rushgrove

Slide 49

Slide 49 text

Run code against short lived virtual machines and make assertions against the resulting state Gareth Rushgrove

Slide 50

Slide 50 text

Gareth Rushgrove Test Kitchen

Slide 51

Slide 51 text

First the code we want to test Gareth Rushgrove

Slide 52

Slide 52 text

#!/bin/bash apt-get install ntp -y scripts/bootstrap.sh Gareth Rushgrove

Slide 53

Slide 53 text

Then a simple test kitchen conﬁguration Gareth Rushgrove

Slide 54

Slide 54 text

--- driver: name: digitalocean ! provisioner: name: shell ! platforms: - name: ubuntu-12.04 ! suites: - name: default provisioner: script: scripts/bootstrap.sh .kitchen.yml Gareth Rushgrove

Slide 55

Slide 55 text

You can provision machines anywhere with different drivers Gareth Rushgrove

Slide 56

Slide 56 text

kitchen driver discover Gareth Rushgrove

Slide 57

Slide 57 text

Gareth Rushgrove

Slide 58

Slide 58 text

And have different platforms running in parallel Gareth Rushgrove

Slide 59

Slide 59 text

platforms: - name: ubuntu-14.04 - name: ubuntu-13.10 - name: centos-6.5 - name: centos-6.4 - name: ubuntu-12.04 - name: centos-6.4 - name: centos-6.4 - name: archlinux-2013.05 .kitchen.yml Gareth Rushgrove

Slide 60

Slide 60 text

And then write tests against the provisioned machines Gareth Rushgrove

Slide 61

Slide 61 text

Gareth Rushgrove serverspec.org

Slide 62

Slide 62 text

RSpec tests for your servers conﬁgured ! by Puppet, Chef or anything else Gareth Rushgrove

Slide 63

Slide 63 text

describe package('ntp') do it { should be_installed } end serverspec/sample_spec.rb Gareth Rushgrove

Slide 64

Slide 64 text

describe service('ntp') do it { should be_enabled } it { should be_running } end serverspec/sample_spec.rb Gareth Rushgrove

Slide 65

Slide 65 text

kitchen test -c Gareth Rushgrove

Slide 66

Slide 66 text

Package "ntp" should be installed ! Service "ntp" should be enabled should be running ! Finished in 0.11191 seconds 3 examples, 0 failures Finished verifying (0m7.39s). -----> Destroying ... Digital Ocean instance <1493500> destroyed. Finished destroying (0m4.47s). Finished testing (3m59.40s). -----> Kitchen is finished. (3m59.95s) Gareth Rushgrove

Slide 67

Slide 67 text

Kitchen is finished. (3m59.95s) Gareth Rushgrove

Slide 68

Slide 68 text

Gareth Rushgrove garethr/do-test-kitchen

Slide 69

Slide 69 text

Gareth Rushgrove

Slide 70

Slide 70 text

Verifying machine images (AMIs and other artefacts)

Slide 71

Slide 71 text

Your virtual machine images need a pipeline too Gareth Rushgrove

Slide 72

Slide 72 text

We’ll start with an example using AMIs on Gareth Rushgrove

Slide 73

Slide 73 text

Gareth Rushgrove packer.io

Slide 74

Slide 74 text

Packer is a tool for creating identical machine images for multiple platforms from a single source conﬁguration Gareth Rushgrove

Slide 75

Slide 75 text

Conﬁguration is lots of JSON Gareth Rushgrove

Slide 76

Slide 76 text

{ "variables": { "aws_access_key": "{{env ÀWS_ACCESS_KEY_ID`}}", "aws_secret_key": "{{env ÀWS_SECRET_ACCESS_KEY`}}" }, "builders": [{ "type": "amazon-ebs", "access_key": "{{user àws_access_key`}}", "secret_key": "{{user àws_secret_key`}}", "region": "eu-west-1", "source_ami": "ami-480bea3f", "instance_type": "t1.micro", "ssh_username": "ubuntu", "ami_name": "packer-serverspec-example-{{timestamp}}" }], "provisioners": [{ "type": "shell", "script": "scripts/puppet.sh" }, { "type": "file", template.json Gareth Rushgrove

Slide 77

Slide 77 text

First build an AMI Gareth Rushgrove

Slide 78

Slide 78 text

Then install some packages and related conﬁguration Gareth Rushgrove

Slide 79

Slide 79 text

! "provisioners": [{ "type": "shell", "script": "scripts/puppet.sh" }, { "type": "puppet-masterless", "manifest_file": "manifests/site.pp", "hiera_config_path": "hiera.yaml", "module_paths": ["modules"] } template.json Gareth Rushgrove

Slide 80

Slide 80 text

First we’ll install puppet Gareth Rushgrove

Slide 81

Slide 81 text

Slide 82

Slide 82 text

#!/bin/bash ! # Install language locale as without can # interfere with package installation sudo apt-get install language-pack-en -y ! # Install Ruby sudo apt-get update sudo apt-get install ruby1.9.1 -y ! # Install puppet/facter sudo gem install puppet facter --no-ri --no-rdoc scripts/puppet.sh Gareth Rushgrove

Slide 83

Slide 83 text

Then run puppet to conﬁgure the machine Gareth Rushgrove

Slide 84

Slide 84 text

Slide 85

Slide 85 text

Packer supports a range of provisioners Gareth Rushgrove

Slide 86

Slide 86 text

Shell, File upload, Puppet, Chef, Salt, Ansible Gareth Rushgrove

Slide 87

Slide 87 text

How to test our built-but- not-saved image? Gareth Rushgrove

Slide 88

Slide 88 text

Run serverspec tests from within the new image Gareth Rushgrove

Slide 89

Slide 89 text

Only if the tests pass do we save the AMI Gareth Rushgrove

Slide 90

Slide 90 text

Gareth Rushgrove garethr/packer-serverspec-example

Slide 91

Slide 91 text

Gareth Rushgrove

Slide 92

Slide 92 text

Linux containers are another good artefact example Gareth Rushgrove

Slide 93

Slide 93 text

Testing! based on changes to the Dockerﬁle works in the same way Gareth Rushgrove

Slide 94

Slide 94 text

Gareth Rushgrove garethr/docker-spec-example

Slide 95

Slide 95 text

Gareth Rushgrove

Slide 96

Slide 96 text

Continuous load testing (The power of aggregate assertions)

Slide 97

Slide 97 text

Load testing is often! ad hoc Gareth Rushgrove

Slide 98

Slide 98 text

Load testing is more than just an application concern Gareth Rushgrove

Slide 99

Slide 99 text

Lots of open source tools Gareth Rushgrove

Slide 100

Slide 100 text

JMeter, Funkload, Tsung, Grinder, HTTPerf, Siege, Gatling, Locust, Vegeta, Gor … Gareth Rushgrove

Slide 101

Slide 101 text

Nothing is perfect for our purposes Gareth Rushgrove

Slide 102

Slide 102 text

Gareth Rushgrove We’ll use Gatling as an example

Slide 103

Slide 103 text

class ExampleSimulation extends Simulation { val httpConf = http.baseURL("http://www.example.com") ! val sample = scenario("Simple example").exec( http("Homepage").get("/").check(status.is(200))) ! setUp(sample.inject(ramp(3 users) over (10 seconds))) .protocols(httpConf) } ExampleSimulation.scala Gareth Rushgrove

Slide 104

Slide 104 text

Aggregate assertions are the killer feature of Gatling Gareth Rushgrove

Slide 105

Slide 105 text

.assertions( global.responseTime.mean.lessThan(750), ) ExampleSimulation.scala Gareth Rushgrove

Slide 106

Slide 106 text

.assertions( global.successfulRequests.percent.is(100), global.responseTime.max.lessThan(1000), global.responseTime.mean.lessThan(750), global.responseTime.percentile1.lessThan(800), global.requestsPerSec.greaterThan(10) ) ExampleSimulation.scala Gareth Rushgrove

Slide 107

Slide 107 text

Continuous load testing supplements rather than replaces ad hoc load testing Gareth Rushgrove

Slide 108

Slide 108 text

Gareth Rushgrove garethr/gatling-demo

Slide 109

Slide 109 text

Gareth Rushgrove

Slide 110

Slide 110 text

Entire infrastructures as code (Testing out provisioning)

Slide 111

Slide 111 text

So far we’ve tested units of infrastructure Gareth Rushgrove

Slide 112

Slide 112 text

This next example tests an entire system Gareth Rushgrove

Slide 113

Slide 113 text

As an example we’ll use Ansible Gareth Rushgrove

Slide 114

Slide 114 text

Gareth Rushgrove ansible.com

Slide 115

Slide 115 text

Speciﬁcally we’ll use the DigitalOcean module Gareth Rushgrove

Slide 116

Slide 116 text

First we deﬁne our system in code Gareth Rushgrove

Slide 117

Slide 117 text

--- droplets: - name: production.web.1 - name: production.web.2 - name: production.app.1 size: 64 - name: production.app.2 size: 64 - name: production.db.1 size: 69 vars/droplets.yml Gareth Rushgrove

Slide 118

Slide 118 text

For testing we’ll use a combination of serverspec and bats Gareth Rushgrove

Slide 119

Slide 119 text

Gareth Rushgrove sstephenson/bats

Slide 120

Slide 120 text

bats is great for testing combinations of shell scripts Gareth Rushgrove

Slide 121

Slide 121 text

#!/usr/bin/env bats ! @test "addition using bc" { result="$(echo 2+2 | bc)" [ "$result" -eq 4 ] } ! @test "addition using dc" { result="$(echo 2 2+p | dc)" [ "$result" -eq 4 ] } example.bats Gareth Rushgrove

Slide 122

Slide 122 text

Run out provisioning code inside a bats test Gareth Rushgrove

Slide 123

Slide 123 text

@test "no droplets exist" { run tugboat droplets [ "$status" -eq 0 ] [ "${lines[0]}" = "You don't appear to have any droplets." ] } ! @test "provisioner runs successfully" { run ansible-playbook -i hosts provision_digital_ocean.yml [ "$status" -eq 0 ] } acceptance.bats Gareth Rushgrove

Slide 124

Slide 124 text

Conﬁrm we have started some machines Gareth Rushgrove

Slide 125

Slide 125 text

@test “we can count the correct number of machines" { run bash -c "ansible -i hosts all --list-hosts | wc - l | tr -d ' '" [ "$status" -eq 0 ] [ "$output" -eq 5 ] } acceptance.bats Gareth Rushgrove

Slide 126

Slide 126 text

Serverspec has a mode of operation where it runs against remote hosts Gareth Rushgrove

Slide 127

Slide 127 text

@test "per host serverspec tests pass" { run rake spec [ "$status" -eq 0 ] } acceptance.bats Gareth Rushgrove

Slide 128

Slide 128 text

bats acceptance.bats Gareth Rushgrove

Slide 129

Slide 129 text

✓ no droplets exist ✓ provisioner runs successfully ✓ we can count the correct number of hosts ✓ we can list the new hosts ✓ all host serverspec tests pass ✓ clean up all droplets ✓ droplets have been cleaned up ! 7 tests, 0 failures Gareth Rushgrove