Clouds in Government - Perils of Portability

Slide 1

Slide 1 text

http://www.ﬂickr.com/photos/wallyg/299908721/ Clouds in Government Perils of Portability QCon 6th February 2013 gareth rushgrove | morethanseven.net

Slide 2

Slide 2 text

Slide 3

Slide 3 text

Gareth Rushgrove @garethr gareth rushgrove | morethanseven.net

Slide 4

Slide 4 text

Curate devopsweekly.com gareth rushgrove | morethanseven.net

Slide 5

Slide 5 text

Blog at morethanseven.net gareth rushgrove | morethanseven.net

Slide 6

Slide 6 text

Work at UK Government Digital Service Text gareth rushgrove | morethanseven.net

Slide 7

Slide 7 text

http://www.ﬂickr.com/photos/benterrett/6852348725/ I am a Civil Servant gareth rushgrove | morethanseven.net

Slide 8

Slide 8 text

http://www.ﬂickr.com/photos/iancarroll/5027441664 Perils Clouds and portability

Slide 9

Slide 9 text

The 2nd deﬁnition gareth rushgrove | morethanseven.net per·il /ˈperəl/ Noun 1. Serious and immediate danger. 2. The dangers or diﬃculties that arise from a particular situation or activity.

Slide 10

Slide 10 text

Peril 1 Caring about Image formats http://www.ﬂickr.com/photos/uk_parliament/2700327415

Slide 11

Slide 11 text

AMI, VMDK, OVF, VHD, VDI, etc. gareth rushgrove | morethanseven.net

Slide 12

Slide 12 text

http://www.ﬂickr.com/photos/uk_parliament/2700311119/ But I have many machines gareth rushgrove | morethanseven.net

Slide 13

Slide 13 text

http://www.ﬂickr.com/photos/uk_parliament/2700327415 And my infrastructure is more than just machines gareth rushgrove | morethanseven.net

Slide 14

Slide 14 text

Peril 2 API proliferation http://www.ﬂickr.com/photos/uk_parliament/2700327415

Slide 15

Slide 15 text

Amazon EC2 gareth rushgrove | morethanseven.net

Slide 16

Slide 16 text

Big API (Just EC2) gareth rushgrove | morethanseven.net 160+ actions

Slide 17

Slide 17 text

Lots more APIs gareth rushgrove | morethanseven.net

Slide 18

Slide 18 text

API compatibility and de facto standards gareth rushgrove | morethanseven.net http://www.ﬂickr.com/photos/uk_parliament/2700357007/

Slide 19

Slide 19 text

Greenqcloud is EC2 compatible gareth rushgrove | morethanseven.net greenqloud.com

Slide 20

Slide 20 text

Eucalyptus gareth rushgrove | morethanseven.net www.eucalyptus.com

Slide 21

Slide 21 text

gareth rushgrove | morethanseven.net EUCALYPtUS Funny story

Slide 22

Slide 22 text

Eucalyptus is an acronym gareth rushgrove | morethanseven.net Elastic Utility Computing Architecture for Linking Your Programs to Useful Systems

Slide 23

Slide 23 text

Ta da gareth rushgrove | morethanseven.net Elastic Utility Computing Architecture for Linking Your Programs to Useful Systems

Slide 24

Slide 24 text

It’s not all about the APIs gareth rushgrove | morethanseven.net http://www.ﬂickr.com/photos/uk_parliament/2757120644

Slide 25

Slide 25 text

Peril 3 Cloud primitives http://www.ﬂickr.com/photos/uk_parliament/2700327415

Slide 26

Slide 26 text

AWS - All the acronyms! gareth rushgrove | morethanseven.net - Instance - Images - Elastic Compute Cloud (EC2) - Elastic IP (EIP) - Elastic Network Interfaces (EIN) - Elastic Block Store (EBS) - Simple Storage Service (S3) - Elastic Load Balancers (ELB)

Slide 27

Slide 27 text

OpenStack gareth rushgrove | morethanseven.net www.openstack.org

Slide 28

Slide 28 text

OpenStack gareth rushgrove | morethanseven.net - Compute - Storage - Networking - Instance - Security group - Object store - Block store

Slide 29

Slide 29 text

CloudStack gareth rushgrove | morethanseven.net incubator.apache.org/cloudstack/

Slide 30

Slide 30 text

CloudStack gareth rushgrove | morethanseven.net - Network - VPC - Virtual machine - VPN - Load balancer - Router - Project - Network - ISO - Volume - Template - Security group - User - Snapshot - Firewall - Account - NAT - VM group - Resource tag - Address - Zone - Disk oﬀering - Hypervisor - Guest OS

Slide 31

Slide 31 text

Abstractions to the rescue? gareth rushgrove | morethanseven.net http://www.ﬂickr.com/photos/uk_parliament/2701192648/

Slide 32

Slide 32 text

Fog (Ruby) gareth rushgrove | morethanseven.net fog.io

Slide 33

Slide 33 text

Fog primitives gareth rushgrove | morethanseven.net - Compute - Storage - CDN - DNS

Slide 34

Slide 34 text

libcloud (Python) gareth rushgrove | morethanseven.net libcloud.apache.org

Slide 35

Slide 35 text

libcloud primitives gareth rushgrove | morethanseven.net - Compute - Storage - Load balancers - DNS

Slide 36

Slide 36 text

jclouds (Java) gareth rushgrove | morethanseven.net www.jclouds.org

Slide 37

Slide 37 text

jclouds primitives gareth rushgrove | morethanseven.net - Computeservice - Blob store

Slide 38

Slide 38 text

Naming things is hard gareth rushgrove | morethanseven.net There are only two hard things in Computer Science: cache invalidation and naming things. Phil Karlton “ ”

Slide 39

Slide 39 text

Peril 4 Slippery slope of Platform as a Service http://www.ﬂickr.com/photos/uk_parliament/2700327415

Slide 40

Slide 40 text

Deﬁnitions gareth rushgrove | morethanseven.net ...does not manage or control the underlying cloud infrastructure including network, servers, operating systems, or storage... ...does not manage or control the underlying cloud infrastructure but has control over operating systems, storage, and deployed applications; and possibly limited control of select networking components... PaaS IaaS

Slide 41

Slide 41 text

Platform as a Service gareth rushgrove | morethanseven.net

Slide 42

Slide 42 text

Not PaaS gareth rushgrove | morethanseven.net

Slide 43

Slide 43 text

Heroku gareth rushgrove | morethanseven.net

Slide 44

Slide 44 text

Heroku gareth rushgrove | morethanseven.net

Slide 45

Slide 45 text

Amazon Elastic Beanstalk gareth rushgrove | morethanseven.net

Slide 46

Slide 46 text

Amazon Elastic Beanstalk gareth rushgrove | morethanseven.net

Slide 47

Slide 47 text

Amazon EC2 gareth rushgrove | morethanseven.net

Slide 48

Slide 48 text

Amazon EC2 gareth rushgrove | morethanseven.net

Slide 49

Slide 49 text

vCloud Director gareth rushgrove | morethanseven.net

Slide 50

Slide 50 text

vCloud Director gareth rushgrove | morethanseven.net

Slide 51

Slide 51 text

Amazon DynamoDB gareth rushgrove | morethanseven.net

Slide 52

Slide 52 text

Amazon DynamoDB gareth rushgrove | morethanseven.net

Slide 53

Slide 53 text

Amazon ElastiCache gareth rushgrove | morethanseven.net

Slide 54

Slide 54 text

Amazon ElastiCache gareth rushgrove | morethanseven.net

Slide 55

Slide 55 text

Peril 5 Vendor lock-in

Slide 56

Slide 56 text

Capability lock-in gareth rushgrove | morethanseven.net

Slide 57

Slide 57 text

Capacity lock-in gareth rushgrove | morethanseven.net

Slide 58

Slide 58 text

Ecosystem lock-in gareth rushgrove | morethanseven.net http://www.ﬂickr.com/photos/uk_parliament/2700549757/

Slide 59

Slide 59 text

http://www.ﬂickr.com/photos/iancarroll/5027441664 Interlude The story of GOV.UK

Slide 60

Slide 60 text

gareth rushgrove | morethanseven.net Government is Big 464,000 55,000 UK Civil Service Google 19,995 BBC x8 x23

Slide 61

Slide 61 text

Martha Lane-Fox Report - October 2010 gareth rushgrove | morethanseven.net

Slide 62

Slide 62 text

Alpha - June 2011 gareth rushgrove | morethanseven.net

Slide 63

Slide 63 text

Me - September 2011 gareth rushgrove | morethanseven.net

Slide 64

Slide 64 text

GDS Government Digital Service - December 2011 gareth rushgrove | morethanseven.net

Slide 65

Slide 65 text

Beta - January 2012 gareth rushgrove | morethanseven.net

Slide 66

Slide 66 text

Design Principles - April 2012 gareth rushgrove | morethanseven.net

Slide 67

Slide 67 text

gareth rushgrove | morethanseven.net Why Infrastructure as a Service? digital.cabinetoﬃce.gov.uk/2012/09/25/why-iaas/

Slide 68

Slide 68 text

gareth rushgrove | morethanseven.net G-Cloud Procurement Framework gcloud.civilservice.gov.uk

Slide 69

Slide 69 text

gareth rushgrove | morethanseven.net EC2 to VMWare http://www.ﬂickr.com/photos/uk_parliament/2701203048/

Slide 70

Slide 70 text

GOV.UK - October 2012 gareth rushgrove | morethanseven.net

Slide 71

Slide 71 text

Government Digital Strategy - November 2012 gareth rushgrove | morethanseven.net publications.cabinetoﬃce.gov.uk/digital/

Slide 72

Slide 72 text

13 of 24 Departments - So far gareth rushgrove | morethanseven.net

Slide 73

Slide 73 text

http://www.ﬂickr.com/photos/iancarroll/5027441664 Solutions? What can we do

Slide 74

Slide 74 text

Solution 1 Infrastructure as code http://www.ﬂickr.com/photos/uk_parliament/2700327415

Slide 75

Slide 75 text

gareth rushgrove | morethanseven.net Conﬁguration Management

Slide 76

Slide 76 text

gareth rushgrove | morethanseven.net Chef opscode.com

Slide 77

Slide 77 text

gareth rushgrove | morethanseven.net Chef code example cookbook_file "#{home_dir}/.ssh/authorized_keys" do source "authorized_keys" mode "0600" owner username group username end group "sysadmin" do members ["garethr"] end

Slide 78

Slide 78 text

gareth rushgrove | morethanseven.net CFEngine cfengine.com

Slide 79

Slide 79 text

gareth rushgrove | morethanseven.net CFEngine code example bundle agent test { packages: redhat:: "wget" package_policy => "addupdate", package_method => yum, package_select => ">=", package_version => "1.11.4-2.el5_4.1", package_architectures => { "x86_64" }; }

Slide 80

Slide 80 text

gareth rushgrove | morethanseven.net Puppet puppetlabs.com

Slide 81

Slide 81 text

package { 'apache2': ensure => latest, } service { 'apache2': ensure => running, provider => upstart, require => Package['apache2'] } gareth rushgrove | morethanseven.net Resources

Slide 82

Slide 82 text

class govuk::apps::calendars( $port = 3011 ) { govuk::app { 'calendars': app_type => 'rack', port => $port, health_check_path => ‘/bank-holidays’, } } gareth rushgrove | morethanseven.net Applications

Slide 83

Slide 83 text

class govuk::node::s_frontend inherits govuk::n include govuk::node::s_ruby_app_server include govuk::apps::businesssupportfinder include govuk::apps::calendars include govuk::apps::canary_frontend include govuk::apps::datainsight_frontend include govuk::apps::designprinciples include govuk::apps::feedback include govuk::apps::frontend include govuk::apps::licencefinder include govuk::apps::smartanswers include govuk::apps::static gareth rushgrove | morethanseven.net Node types

Slide 84

Slide 84 text

Slide 85

Slide 85 text

Slide 86

Slide 86 text

gareth rushgrove | morethanseven.net More on Infrastructure as Code speakerdeck.com/garethr

Slide 87

Slide 87 text

Solution 2 API abstractions http://www.ﬂickr.com/photos/uk_parliament/2700327415

Slide 88

Slide 88 text

gareth rushgrove | morethanseven.net libcloud

Slide 89

Slide 89 text

gareth rushgrove | morethanseven.net libcloud OpenStack example from libcloud.compute.types import Provider from libcloud.compute.providers import get_driver OpenStack = get_driver(Provider.OPENSTACK) driver = OpenStack('username', 'password', ex_force_auth_url='https://nova-api.trystack.org:5 ex_force_auth_version='2.0_password') nodes = driver.list_nodes() images = driver.list_images()

Slide 90

Slide 90 text

gareth rushgrove | morethanseven.net libcloud VCloud example from libcloud.compute.types import Provider from libcloud.compute.providers import get_driver vcloud = get_driver(Provider.VCLOUD) driver = vcloud('username', 'password', host='vcloud.local', api_version='1.5') nodes = driver.list_nodes() images = driver.list_images()

Slide 91

Slide 91 text

gareth rushgrove | morethanseven.net But abstractions leak images = driver.list_images() sizes = driver.list_sizes() size = [s for s in sizes if s.ram == 512][0] image = [i for i in images if i.name == 'natty-amd64'][0] node = driver.create_node(name='test node', image=image, size=size)

Slide 92

Slide 92 text

Slide 93

Slide 93 text

gareth rushgrove | morethanseven.net But abstractions leak take two vcloud = get_driver(Provider.VCLOUD) driver = vcloud('username', 'password', host='vcloud.local', api_version='1.5') node = driver.create_node(name='test node 4', image=image, ex_vm_network='your vm net name', ex_network='your org net name', ex_vm_fence='bridged', ex_vm_ipmode='DHCP')

Slide 94

Slide 94 text

gareth rushgrove | morethanseven.net More capabilities, more leaks vcloud = get_driver(Provider.VCLOUD) driver = vcloud('username', 'password', host='vcloud.local', api_version='1.5') node = driver.create_node(name='test node 4', image=image, ex_vm_network='your vm net name', ex_network='your org net name', ex_vm_fence='bridged', ex_vm_ipmode='DHCP')

Slide 95

Slide 95 text

gareth rushgrove | morethanseven.net Fog

Slide 96

Slide 96 text

gareth rushgrove | morethanseven.net jclouds

Slide 97

Slide 97 text

Solution 3 Conﬁg managent plus APIs http://www.ﬂickr.com/photos/uk_parliament/2700327415

Slide 98

Slide 98 text

gareth rushgrove | morethanseven.net Pallet github.com/pallet/pallet

Slide 99

Slide 99 text

gareth rushgrove | morethanseven.net Pallet code example (use 'pallet.crate.java) (defnode webserver {} :configure (phase (java :openjdk))) (converge {webserver 10} :compute service)

Slide 100

Slide 100 text

gareth rushgrove | morethanseven.net Ironfan github.com/infochimps-labs/ironfan

Slide 101

Slide 101 text

gareth rushgrove | morethanseven.net Ironfan example Ironfan.cluster 'web_demo' do cloud(:ec2) do flavor 't1.micro' end role :base_role facet :dbnode do instances 2 role :mysql_server end end

Slide 102

Slide 102 text

gareth rushgrove | morethanseven.net puppet-iaas github.com/garethr/garethr-iaas

Slide 103

Slide 103 text

gareth rushgrove | morethanseven.net Cloud instances as resources server { 'web-server': ensure => present, count => 5, provider => brightbox, image => 'img-q6gc8', # ubuntu 12.04 }

Slide 104

Slide 104 text

gareth rushgrove | morethanseven.net Switch the provider server { 'web-server': ensure => present, count => 5, provider => rackspace, image => 'img-q6gc8', # ubuntu 12.04 }

Slide 105

Slide 105 text

gareth rushgrove | morethanseven.net Leaky interface server { 'web-server': ensure => present, count => 5, provider => rackspace, image => '5cebb13a-f783-4f8c-8058 c4182c7 flavor => 2, # 512 MB }

Slide 106

Slide 106 text

gareth rushgrove | morethanseven.net Vagrant 1.1 vagrantup.com

Slide 107

Slide 107 text

gareth rushgrove | morethanseven.net Deﬁne our instance Vagrant::Config.run do |config| config.vm.box = "precise64" config.vm.forward_port 5555, 5555 config.vm.forward_port 5556, 5556 config.vm.forward_port 4567, 4567 config.vm.provision :puppet do |puppet| puppet.manifests_path = "manifests" puppet.module_path = "modules" puppet.manifest_file = "site.pp" end end

Slide 108

Slide 108 text

Slide 109

Slide 109 text

gareth rushgrove | morethanseven.net Choose your own provider $ vagrant up --provider=virtualbox

Slide 110

Slide 110 text

gareth rushgrove | morethanseven.net Switch your provider $ vagrant up --provider=ec2

Slide 111

Slide 111 text

Solution 4 Software deﬁned networks http://www.ﬂickr.com/photos/uk_parliament/2700327415

Slide 112

Slide 112 text

gareth rushgrove | morethanseven.net Ruby DSL require 'rubygems' require 'nat' nat do snat :interface => "Client Data", :original => { :ip => "10.0.0.0/xx" }, :translated => { :ip => "xx.xx.xx.xx" }, :desc => "Outbound internet traffic" dnat :interface => "Client Data", :original => { :ip => "xx.xx.xx.xx", :port => 22 }, :translated => { :ip => "10.0.0.xx", :port => 22 }, :desc => "jumpbox-1 SSH" dnat :interface => "Client Data", :original => { :ip => "xx.xx.xx.xx", :port => 80 },, :translated => { :ip => "10.0.0.xx", :port => 80 }, :desc => "jenkins, logging, monitoring HTTP"

Slide 113

Slide 113 text

require 'rubygems' require 'firewall' firewall do # internal rules rule "ssh access to jumpbox1" do source :ip => "Any" destination :ip => "xx.xx.xx.xx", :port => 22 end rule "http to backend applications" do source :ip => "Any" destination :ip => "xx.xx.xx.xx", :port => 80 end rule "https to backend applications" do gareth rushgrove | morethanseven.net Including Firewall and Loadbalancer

Slide 114

Slide 114 text

http://www.ﬂickr.com/photos/iancarroll/5027441664 Conclusions if all you remember is

Slide 115

Slide 115 text

http://www.ﬂickr.com/photos/kevharb/5314268567 gareth rushgrove | morethanseven.net Solve the problem for the complex case

Slide 116

Slide 116 text

gareth rushgrove | morethanseven.net Focus on capabilities over APIs http://www.ﬂickr.com/photos/sprengben/5136170057

Slide 117

Slide 117 text

The End

Slide 118

Slide 118 text

gareth rushgrove | morethanseven.net Thanks for the photos

Slide 119

Slide 119 text

Questions? gareth rushgrove | morethanseven.net http://ﬂickr.com/photos/psd/102332391/

Slide 120

Slide 120 text

QCon session code gareth rushgrove | morethanseven.net 4172