Clouds in Government - Perils of Portability

Transcript

1. http://www.flickr.com/photos/wallyg/299908721/ Clouds in Government Perils of Portability QCon 6th February

   2013 gareth rushgrove | morethanseven.net

2. Me

3. Gareth Rushgrove @garethr gareth rushgrove | morethanseven.net

4. Curate devopsweekly.com gareth rushgrove | morethanseven.net

5. Blog at morethanseven.net gareth rushgrove | morethanseven.net

6. Work at UK Government Digital Service Text gareth rushgrove |

   morethanseven.net

7. http://www.flickr.com/photos/benterrett/6852348725/ I am a Civil Servant gareth rushgrove | morethanseven.net

8. http://www.flickr.com/photos/iancarroll/5027441664 Perils Clouds and portability

9. The 2nd definition gareth rushgrove | morethanseven.net per·il /ˈperəl/ Noun

   1. Serious and immediate danger. 2. The dangers or difficulties that arise from a particular situation or activity.

10. Peril 1 Caring about Image formats http://www.flickr.com/photos/uk_parliament/2700327415

11. AMI, VMDK, OVF, VHD, VDI, etc. gareth rushgrove | morethanseven.net

12. http://www.flickr.com/photos/uk_parliament/2700311119/ But I have many machines gareth rushgrove | morethanseven.net

13. http://www.flickr.com/photos/uk_parliament/2700327415 And my infrastructure is more than just machines gareth

    rushgrove | morethanseven.net

14. Peril 2 API proliferation http://www.flickr.com/photos/uk_parliament/2700327415

15. Amazon EC2 gareth rushgrove | morethanseven.net

16. Big API (Just EC2) gareth rushgrove | morethanseven.net 160+ actions

17. Lots more APIs gareth rushgrove | morethanseven.net

18. API compatibility and de facto standards gareth rushgrove | morethanseven.net

    http://www.flickr.com/photos/uk_parliament/2700357007/

19. Greenqcloud is EC2 compatible gareth rushgrove | morethanseven.net greenqloud.com

20. Eucalyptus gareth rushgrove | morethanseven.net www.eucalyptus.com

21. gareth rushgrove | morethanseven.net EUCALYPtUS Funny story

22. Eucalyptus is an acronym gareth rushgrove | morethanseven.net Elastic Utility

    Computing Architecture for Linking Your Programs to Useful Systems

23. Ta da gareth rushgrove | morethanseven.net Elastic Utility Computing Architecture

    for Linking Your Programs to Useful Systems

24. It’s not all about the APIs gareth rushgrove | morethanseven.net

    http://www.flickr.com/photos/uk_parliament/2757120644

25. Peril 3 Cloud primitives http://www.flickr.com/photos/uk_parliament/2700327415

26. AWS - All the acronyms! gareth rushgrove | morethanseven.net -

    Instance - Images - Elastic Compute Cloud (EC2) - Elastic IP (EIP) - Elastic Network Interfaces (EIN) - Elastic Block Store (EBS) - Simple Storage Service (S3) - Elastic Load Balancers (ELB)

27. OpenStack gareth rushgrove | morethanseven.net www.openstack.org

28. OpenStack gareth rushgrove | morethanseven.net - Compute - Storage -

    Networking - Instance - Security group - Object store - Block store

29. CloudStack gareth rushgrove | morethanseven.net incubator.apache.org/cloudstack/

30. CloudStack gareth rushgrove | morethanseven.net - Network - VPC -

    Virtual machine - VPN - Load balancer - Router - Project - Network - ISO - Volume - Template - Security group - User - Snapshot - Firewall - Account - NAT - VM group - Resource tag - Address - Zone - Disk offering - Hypervisor - Guest OS

31. Abstractions to the rescue? gareth rushgrove | morethanseven.net http://www.flickr.com/photos/uk_parliament/2701192648/

32. Fog (Ruby) gareth rushgrove | morethanseven.net fog.io

33. Fog primitives gareth rushgrove | morethanseven.net - Compute - Storage

    - CDN - DNS

34. libcloud (Python) gareth rushgrove | morethanseven.net libcloud.apache.org

35. libcloud primitives gareth rushgrove | morethanseven.net - Compute - Storage

    - Load balancers - DNS

36. jclouds (Java) gareth rushgrove | morethanseven.net www.jclouds.org

37. jclouds primitives gareth rushgrove | morethanseven.net - Computeservice - Blob

    store

38. Naming things is hard gareth rushgrove | morethanseven.net There are

    only two hard things in Computer Science: cache invalidation and naming things. Phil Karlton “ ”

39. Peril 4 Slippery slope of Platform as a Service http://www.flickr.com/photos/uk_parliament/2700327415

40. Definitions gareth rushgrove | morethanseven.net ...does not manage or control

    the underlying cloud infrastructure including network, servers, operating systems, or storage... ...does not manage or control the underlying cloud infrastructure but has control over operating systems, storage, and deployed applications; and possibly limited control of select networking components... PaaS IaaS

41. Platform as a Service gareth rushgrove | morethanseven.net

42. Not PaaS gareth rushgrove | morethanseven.net

43. Heroku gareth rushgrove | morethanseven.net

44. Heroku gareth rushgrove | morethanseven.net

45. Amazon Elastic Beanstalk gareth rushgrove | morethanseven.net

46. Amazon Elastic Beanstalk gareth rushgrove | morethanseven.net

47. Amazon EC2 gareth rushgrove | morethanseven.net

48. Amazon EC2 gareth rushgrove | morethanseven.net

49. vCloud Director gareth rushgrove | morethanseven.net

50. vCloud Director gareth rushgrove | morethanseven.net

51. Amazon DynamoDB gareth rushgrove | morethanseven.net

52. Amazon DynamoDB gareth rushgrove | morethanseven.net

53. Amazon ElastiCache gareth rushgrove | morethanseven.net

54. Amazon ElastiCache gareth rushgrove | morethanseven.net

55. Peril 5 Vendor lock-in

56. Capability lock-in gareth rushgrove | morethanseven.net

57. Capacity lock-in gareth rushgrove | morethanseven.net

58. Ecosystem lock-in gareth rushgrove | morethanseven.net http://www.flickr.com/photos/uk_parliament/2700549757/

59. http://www.flickr.com/photos/iancarroll/5027441664 Interlude The story of GOV.UK

60. gareth rushgrove | morethanseven.net Government is Big 464,000 55,000 UK

    Civil Service Google 19,995 BBC x8 x23

61. Martha Lane-Fox Report - October 2010 gareth rushgrove | morethanseven.net

62. Alpha - June 2011 gareth rushgrove | morethanseven.net

63. Me - September 2011 gareth rushgrove | morethanseven.net

64. GDS Government Digital Service - December 2011 gareth rushgrove |

    morethanseven.net

65. Beta - January 2012 gareth rushgrove | morethanseven.net

66. Design Principles - April 2012 gareth rushgrove | morethanseven.net

67. gareth rushgrove | morethanseven.net Why Infrastructure as a Service? digital.cabinetoffice.gov.uk/2012/09/25/why-iaas/

68. gareth rushgrove | morethanseven.net G-Cloud Procurement Framework gcloud.civilservice.gov.uk

69. gareth rushgrove | morethanseven.net EC2 to VMWare http://www.flickr.com/photos/uk_parliament/2701203048/

70. GOV.UK - October 2012 gareth rushgrove | morethanseven.net

71. Government Digital Strategy - November 2012 gareth rushgrove | morethanseven.net

    publications.cabinetoffice.gov.uk/digital/

72. 13 of 24 Departments - So far gareth rushgrove |

    morethanseven.net

73. http://www.flickr.com/photos/iancarroll/5027441664 Solutions? What can we do

74. Solution 1 Infrastructure as code http://www.flickr.com/photos/uk_parliament/2700327415

75. gareth rushgrove | morethanseven.net Configuration Management

76. gareth rushgrove | morethanseven.net Chef opscode.com

77. gareth rushgrove | morethanseven.net Chef code example cookbook_file "#{home_dir}/.ssh/authorized_keys" do

    source "authorized_keys" mode "0600" owner username group username end group "sysadmin" do members ["garethr"] end

78. gareth rushgrove | morethanseven.net CFEngine cfengine.com

79. gareth rushgrove | morethanseven.net CFEngine code example bundle agent test

    { packages: redhat:: "wget" package_policy => "addupdate", package_method => yum, package_select => ">=", package_version => "1.11.4-2.el5_4.1", package_architectures => { "x86_64" }; }

80. gareth rushgrove | morethanseven.net Puppet puppetlabs.com

81. package { 'apache2': ensure => latest, } service { 'apache2':

    ensure => running, provider => upstart, require => Package['apache2'] } gareth rushgrove | morethanseven.net Resources

82. class govuk::apps::calendars( $port = 3011 ) { govuk::app { 'calendars':

    app_type => 'rack', port => $port, health_check_path => ‘/bank-holidays’, } } gareth rushgrove | morethanseven.net Applications

83. class govuk::node::s_frontend inherits govuk::n include govuk::node::s_ruby_app_server include govuk::apps::businesssupportfinder include govuk::apps::calendars

    include govuk::apps::canary_frontend include govuk::apps::datainsight_frontend include govuk::apps::designprinciples include govuk::apps::feedback include govuk::apps::frontend include govuk::apps::licencefinder include govuk::apps::smartanswers include govuk::apps::static gareth rushgrove | morethanseven.net Node types

84. class govuk::node::s_frontend inherits govuk::n include govuk::node::s_ruby_app_server include govuk::apps::businesssupportfinder include govuk::apps::calendars

    include govuk::apps::canary_frontend include govuk::apps::datainsight_frontend include govuk::apps::designprinciples include govuk::apps::feedback include govuk::apps::frontend include govuk::apps::licencefinder include govuk::apps::smartanswers include govuk::apps::static gareth rushgrove | morethanseven.net Include software on nodes

85. class govuk::node::s_frontend inherits govuk::n include govuk::node::s_ruby_app_server include govuk::apps::businesssupportfinder include govuk::apps::calendars

    include govuk::apps::canary_frontend include govuk::apps::datainsight_frontend include govuk::apps::designprinciples include govuk::apps::feedback include govuk::apps::frontend include govuk::apps::licencefinder include govuk::apps::smartanswers include govuk::apps::static gareth rushgrove | morethanseven.net Include out applications on nodes

86. gareth rushgrove | morethanseven.net More on Infrastructure as Code speakerdeck.com/garethr

87. Solution 2 API abstractions http://www.flickr.com/photos/uk_parliament/2700327415

88. gareth rushgrove | morethanseven.net libcloud

89. gareth rushgrove | morethanseven.net libcloud OpenStack example from libcloud.compute.types import

    Provider from libcloud.compute.providers import get_driver OpenStack = get_driver(Provider.OPENSTACK) driver = OpenStack('username', 'password', ex_force_auth_url='https://nova-api.trystack.org:5 ex_force_auth_version='2.0_password') nodes = driver.list_nodes() images = driver.list_images()

90. gareth rushgrove | morethanseven.net libcloud VCloud example from libcloud.compute.types import

    Provider from libcloud.compute.providers import get_driver vcloud = get_driver(Provider.VCLOUD) driver = vcloud('username', 'password', host='vcloud.local', api_version='1.5') nodes = driver.list_nodes() images = driver.list_images()

91. gareth rushgrove | morethanseven.net But abstractions leak images = driver.list_images()

    sizes = driver.list_sizes() size = [s for s in sizes if s.ram == 512][0] image = [i for i in images if i.name == 'natty-amd64'][0] node = driver.create_node(name='test node', image=image, size=size)

92. gareth rushgrove | morethanseven.net But abstractions leak images = driver.list_images()

    sizes = driver.list_sizes() size = [s for s in sizes if s.ram == 512][0] image = [i for i in images if i.name == 'natty-amd64'][0] node = driver.create_node(name='test node', image=image, size=size)

93. gareth rushgrove | morethanseven.net But abstractions leak take two vcloud

    = get_driver(Provider.VCLOUD) driver = vcloud('username', 'password', host='vcloud.local', api_version='1.5') node = driver.create_node(name='test node 4', image=image, ex_vm_network='your vm net name', ex_network='your org net name', ex_vm_fence='bridged', ex_vm_ipmode='DHCP')

94. gareth rushgrove | morethanseven.net More capabilities, more leaks vcloud =

    get_driver(Provider.VCLOUD) driver = vcloud('username', 'password', host='vcloud.local', api_version='1.5') node = driver.create_node(name='test node 4', image=image, ex_vm_network='your vm net name', ex_network='your org net name', ex_vm_fence='bridged', ex_vm_ipmode='DHCP')

95. gareth rushgrove | morethanseven.net Fog

96. gareth rushgrove | morethanseven.net jclouds

97. Solution 3 Config managent plus APIs http://www.flickr.com/photos/uk_parliament/2700327415

98. gareth rushgrove | morethanseven.net Pallet github.com/pallet/pallet

99. gareth rushgrove | morethanseven.net Pallet code example (use 'pallet.crate.java) (defnode

    webserver {} :configure (phase (java :openjdk))) (converge {webserver 10} :compute service)

100. gareth rushgrove | morethanseven.net Ironfan github.com/infochimps-labs/ironfan

101. gareth rushgrove | morethanseven.net Ironfan example Ironfan.cluster 'web_demo' do cloud(:ec2)

     do flavor 't1.micro' end role :base_role facet :dbnode do instances 2 role :mysql_server end end

102. gareth rushgrove | morethanseven.net puppet-iaas github.com/garethr/garethr-iaas

103. gareth rushgrove | morethanseven.net Cloud instances as resources server {

     'web-server': ensure => present, count => 5, provider => brightbox, image => 'img-q6gc8', # ubuntu 12.04 }

104. gareth rushgrove | morethanseven.net Switch the provider server { 'web-server':

     ensure => present, count => 5, provider => rackspace, image => 'img-q6gc8', # ubuntu 12.04 }

105. gareth rushgrove | morethanseven.net Leaky interface server { 'web-server': ensure

     => present, count => 5, provider => rackspace, image => '5cebb13a-f783-4f8c-8058 c4182c7 flavor => 2, # 512 MB }

106. gareth rushgrove | morethanseven.net Vagrant 1.1 vagrantup.com

107. gareth rushgrove | morethanseven.net Define our instance Vagrant::Config.run do |config|

     config.vm.box = "precise64" config.vm.forward_port 5555, 5555 config.vm.forward_port 5556, 5556 config.vm.forward_port 4567, 4567 config.vm.provision :puppet do |puppet| puppet.manifests_path = "manifests" puppet.module_path = "modules" puppet.manifest_file = "site.pp" end end

108. gareth rushgrove | morethanseven.net Configure different providers Vagrant.configure("2") do |config|

     config.vm.box = "precise64" config.vm.provider :vmware_fusion do |v| v.vmx["memsize"] = "1024" end config.vm.provider :aws do |aws| aws.instance_type = "m1.small" end end

109. gareth rushgrove | morethanseven.net Choose your own provider $ vagrant

     up --provider=virtualbox

110. gareth rushgrove | morethanseven.net Switch your provider $ vagrant up

     --provider=ec2

111. Solution 4 Software defined networks http://www.flickr.com/photos/uk_parliament/2700327415

112. gareth rushgrove | morethanseven.net Ruby DSL require 'rubygems' require 'nat'

     nat do snat :interface => "Client Data", :original => { :ip => "10.0.0.0/xx" }, :translated => { :ip => "xx.xx.xx.xx" }, :desc => "Outbound internet traffic" dnat :interface => "Client Data", :original => { :ip => "xx.xx.xx.xx", :port => 22 }, :translated => { :ip => "10.0.0.xx", :port => 22 }, :desc => "jumpbox-1 SSH" dnat :interface => "Client Data", :original => { :ip => "xx.xx.xx.xx", :port => 80 },, :translated => { :ip => "10.0.0.xx", :port => 80 }, :desc => "jenkins, logging, monitoring HTTP"

113. require 'rubygems' require 'firewall' firewall do # internal rules rule

     "ssh access to jumpbox1" do source :ip => "Any" destination :ip => "xx.xx.xx.xx", :port => 22 end rule "http to backend applications" do source :ip => "Any" destination :ip => "xx.xx.xx.xx", :port => 80 end rule "https to backend applications" do gareth rushgrove | morethanseven.net Including Firewall and Loadbalancer

114. http://www.flickr.com/photos/iancarroll/5027441664 Conclusions if all you remember is

115. http://www.flickr.com/photos/kevharb/5314268567 gareth rushgrove | morethanseven.net Solve the problem for the

     complex case

116. gareth rushgrove | morethanseven.net Focus on capabilities over APIs http://www.flickr.com/photos/sprengben/5136170057

117. The End

118. gareth rushgrove | morethanseven.net Thanks for the photos

119. Questions? gareth rushgrove | morethanseven.net http://flickr.com/photos/psd/102332391/

120. QCon session code gareth rushgrove | morethanseven.net 4172