Infrastructure as Code

An introductory talk about using configuration management tools and API to manage web infrastructure. Given at CloudEast 28th June 2012

Transcript

1. www.flickr.com/photos/mugley/5013931959/ Infrastructure as Code Cloud East 28th June 2012 gareth rushgrove | morethanseven.net
2. Me
3. Gareth Rushgrove gareth rushgrove | morethanseven.net
4. Blog at morethanseven.net gareth rushgrove | morethanseven.net
5. Curate devopsweekly.com gareth rushgrove | morethanseven.net
6. Work at UK Government Digital Service Text gareth rushgrove | morethanseven.net
7. Serious Government Business gareth rushgrove | morethanseven.net
8. http://www.flickr.com/photos/iancarroll/5027441664 Definitions (What we mean by...)
9. gareth rushgrove | morethanseven.net Infrastructure in·fra·struc·ture /ˈinfrəˌstrəkCHər/ An underlying base or foundation especially for an organisation or system.
10. gareth rushgrove | morethanseven.net Infrastructure
11. gareth rushgrove | morethanseven.net Code code /kōd/ A system of symbols and rules used to represent instructions to a computer; a computer program.
12. gareth rushgrove | morethanseven.net Code
13. http://www.flickr.com/photos/iancarroll/5027441664 Not as code (What lots of people do now)
14. gareth rushgrove | morethanseven.net Manual www.flickr.com/photos/swisscan/4860653795
15. gareth rushgrove | morethanseven.net Often error prone www.flickr.com/photos/almondbutterscotch/6160016599
16. gareth rushgrove | morethanseven.net Slow www.flickr.com/photos/swisscan/3250054769
17. gareth rushgrove | morethanseven.net Time consuming www.flickr.com/photos/swisscan/1545202070
18. gareth rushgrove | morethanseven.net Process heavy www.flickr.com/photos/postsumptio/5994581987
19. http://www.flickr.com/photos/iancarroll/5027441664 Some Code (I know, lets write bash scripts)
20. gareth rushgrove | morethanseven.net SSH for loops #!/bin/bash NODES="webserver.example.com database.example.com" for n in $NODES do ssh $n uptime done
21. gareth rushgrove | morethanseven.net Tests, what tests? www.flickr.com/photos/swisscan/2918682767
22. gareth rushgrove | morethanseven.net (Yes, I know about shUnit) #!/bin/sh testEquality() { assertEquals 1 1 } . ../src/shell/shunit2
23. gareth rushgrove | morethanseven.net Unique snow flake problem www.flickr.com/photos/swisscan/2264972703
24. www.flickr.com/photos/swisscan/2308034084 gareth rushgrove | morethanseven.net Ignoring software engineering practices
25. http://www.flickr.com/photos/iancarroll/5027441664 Configuration Management (and cloud APIs)
26. gareth rushgrove | morethanseven.net Not new - 1993 CFEngine - 2003 Puppet - 2006 Amazon EC2 - 2009 Chef
27. gareth rushgrove | morethanseven.net Examples as code www.flickr.com/photos/thomashawk/130601225
28. gareth rushgrove | morethanseven.net CFEngine cfengine.com
29. gareth rushgrove | morethanseven.net CFEngine code example bundle agent test { packages: redhat:: "wget" package_policy => "addupdate", package_method => yum, package_select => ">=", package_version => "1.11.4-2.el5_4.1", package_architectures => { "x86_64" }; }
30. gareth rushgrove | morethanseven.net Puppet puppetlabs.com
31. gareth rushgrove | morethanseven.net Puppet code example package { 'web-facter': ensure => latest, provider => gem, } service { 'web-facter': ensure => running, provider => upstart, require => Package['web-facter'] }
32. gareth rushgrove | morethanseven.net Chef opscode.com
33. gareth rushgrove | morethanseven.net Chef code example cookbook_file "#{home_dir}/.ssh/authorized_keys" do source "authorized_keys" mode "0600" owner username group username end group "sysadmin" do members ["garethr"] end
34. gareth rushgrove | morethanseven.net Pallet palletops.com
35. gareth rushgrove | morethanseven.net Pallet code example (use 'pallet.crate.java) (defnode webserver {} :configure (phase (java :openjdk))) (converge {webserver 10} :compute service)
36. gareth rushgrove | morethanseven.net Development tools www.flickr.com/photos/swisscan/2286781443
37. gareth rushgrove | morethanseven.net Rspec-puppet require_relative '../../spec_helper' describe 'development', :type => :class do let(:facts) { { :govuk_class => "development" } } it { should create_package("nginx") } it { should_not raise_error(Puppet::ParseError) } end
38. gareth rushgrove | morethanseven.net Rspec-puppet results govuk should include Class[puppet] should include Class[cron] should not raise Puppet::ParseError puppet should contain File[/etc/puppet/puppet.conf] should schedule regular puppet updates Finished in 3.42 seconds 12 examples, 0 failures
39. gareth rushgrove | morethanseven.net Puppet lint Evaluating manifests/classes/development.pp 14:double_quoted_strings:WARNING:double quoted string containing no variables 37:arrow_alignment:WARNING:=> on line isn't properly aligned for resource
40. gareth rushgrove | morethanseven.net Geppetto Puppet IDE
41. gareth rushgrove | morethanseven.net Foodcritic for Chef
42. gareth rushgrove | morethanseven.net Metrics
43. gareth rushgrove | morethanseven.net Chef examples thanks @portertech
44. gareth rushgrove | morethanseven.net Multiple nodes www.flickr.com/photos/wecand/4862594210
45. gareth rushgrove | morethanseven.net Puppet master
46. gareth rushgrove | morethanseven.net Puppet cloud provisioner puppet node_aws create \ --image ami-2d4aa444 \ --type m1.small \ --keypair puppetlabs.admin
47. gareth rushgrove | morethanseven.net Chef Server
48. knife ec2 server create \ -r "role[webserver]" \ -I ami-2d4aa444 \ --flavor m1.small gareth rushgrove | morethanseven.net Chef Knife EC2
49. { "Description" : "Create an EC2 instance running Amazon Linux 32" "Parameters" : { "KeyPair" : { "Description" : "The EC2 Key Pair to allow SSH access", "Type" : "String" } }, "Resources" : { "Ec2Instance" : { "Type" : "AWS::EC2::Instance", "Properties" : { "KeyName" : { "Ref" : "KeyPair" }, "ImageId" : "ami-75g0061f" } } }, "Outputs" : { "InstanceId" : { "Description" : "The InstanceId of the created EC2 instance", "Value" : { "Ref" : "Ec2Instance" } } }, "AWSTemplateFormatVersion" : "2010-09-09" } gareth rushgrove | morethanseven.net Amazon CloudFormation
50. http://www.flickr.com/photos/iancarroll/5027441664 Case Study (the day job)
51. GOV.UK gareth rushgrove | morethanseven.net
52. gareth rushgrove | morethanseven.net Define infrastructure components early www.flickr.com/photos/swisscan/2151073152
53. www.flickr.com/photos/swisscan/2292829724 gareth rushgrove | morethanseven.net Define infrastructure implementation later
54. www.flickr.com/photos/swisscan/651760224 gareth rushgrove | morethanseven.net Part of development activities
55. gareth rushgrove | morethanseven.net High level primitives apache2::vhost::passenger { "app1.$::govuk_platform.internal":; "app2.$::govuk_platform.internal":; }
56. gareth rushgrove | morethanseven.net Under the hood - Create Apache Virtual host - Setup Ruby web application server - Reload Apache if needed - Setup monitoring checks in Nagios - Send log files to Ganglia and Greylog
57. gareth rushgrove | morethanseven.net Bootstrap new machines 1. Bring up new instance using Fog 2. Install Ruby and Puppet 3. Let instance know what type of node it is 4. Register new instance with puppet master 5. Let Puppet install the rest of required software 6. Collected puppet resources add monitoring
58. gareth rushgrove | morethanseven.net Fog fog.io
59. { "role": "client", "platform": "preview", "class": "frontend", "security_groups": ["preview-frontend", "preview"], "key": "preview.pem", "flavor": "m1.large", "image": "ami-5c417128", "master": "puppet.example.com", "debug": true, "log_level": 0 } gareth rushgrove | morethanseven.net Describe machine types
60. { "role": "client", "platform": "preview", "class": "frontend", "security_groups": ["preview-frontend", "preview"], "key": "preview.pem", "flavor": "m1.large", "image": "ami-5c417128", "master": "puppet.example.com", "debug": true, "log_level": 0 } gareth rushgrove | morethanseven.net Used by Puppet
61. bundle exec bin/provision --file=frontend.json -n 5 gareth rushgrove | morethanseven.net Summon 5 new web servers
62. www.flickr.com/photos/swisscan/2110710885 gareth rushgrove | morethanseven.net Local development environment
63. http://www.flickr.com/photos/iancarroll/5027441664 Takeaway (if all you remember is)
64. gareth rushgrove | morethanseven.net Infrastructure can be described in code - Domain specific - Readable - Reviewable - Testable - Reusable
65. gareth rushgrove | morethanseven.net It’s easy, just start simple - Just manage users - Just manage cron jobs - Just manage the web server - Just manage your database configuration
66. http://www.flickr.com/photos/lkanies/5996581482 gareth rushgrove | morethanseven.net Engage the community
67. gareth rushgrove | morethanseven.net Talk to people (on IRC) - #puppet - #chef - #cfengine - #infratalk all on irc.freenode.net
68. The End
69. http://www.flickr.com/photos/benterrett/6852348725/ One more thing, we’re hiring gareth rushgrove | morethanseven.net
70. Questions? gareth rushgrove | morethanseven.net http://flickr.com/photos/psd/102332391/