Introduction to Infrastructure as Code & Automation / Introduction to Chef

Your customers expect you to continuously deliver delightful experiences. This means that you'll need to continuously deliver application and infrastructure updates. Hand-crafted servers lovingly built and maintained by a system administrator are a thing of the past. Golden images are fine for initial provisioning but will quickly fail as your configuration requirements change over time.

It's time for you to fully automate the provisioning and management of your infrastructure components. Welcome to the world of infrastructure as code! In this new world, you'll be able to programmatically provision and configure the components of your infrastructure.

Disposable infrastructure whose provisioning, configuration, and on-going maintenance is fully automated allow you to change the way you build and deliver applications. Move your applications and infrastructure towards continuous delivery.

In this talk, we'll explore the ideas behind "infrastructure as code" and, specifically, look at how Chef allows you to fully automate your infrastructure. If you're brave enough, we'll even let you get your hands on some Chef and experience the delight of using Chef to build and deploy some infrastructure components.

Transcript

1. Introduction to Infrastructure as Code & Automation / Introduction to Chef 1 Nathen Harvey @nathenharvey nharvey@getchef.com github.com/nathenharvey
2. Nathen Harvey • Director, Community and #learnchef • Co-host of the Food Fight Show Podcast • Meetup Organizer • DevOpsDC • @nathenharvey • nharvey@getchef.com
3. Demo
4. Who are you? • Were you in Isaac’s talk? • Were you in my talk yesterday? 4
5. Who are you? • System administrator? 5
6. Who are you? • System administrator? • Developer? 6
7. Who are you? • System administrator? • Developer? • Ruby Developer? 7
8. Who are you? • System administrator? • Developer? • Ruby Developer? • DevOp? 8
9. Who are you? • Experience with Configuration Management? • Experience with Chef? 9
10. Who are you? • Favorite text editor? 10
11. Who are you? • Which version control system do you use? •cp foo foo.bak 11
12. Who are you? • Which version control system do you use? •cp foo foo.bak •cp foo{,.`date +%Y%m%d%H%M`} 12
13. Who are you? • Which version control system do you use? •cp foo foo.bak •cp foo{,.`date +%Y%m%d%H%M`-`$USER`} 13
14. http://www.flickr.com/photos/michaelheiss/3090102907/ Complexity 14
15. Managing Complexity 15 • SSH, make with the typey typey
16. Managing Complexity 16 • SSH, make with the typey typey • Keep notes in ~/server.txt
17. Managing Complexity 17 • SSH, make with the typey typey • Keep notes in ~/server.txt • Move notes to the wiki
18. Managing Complexity 18 • SSH, make with the typey typey • Keep notes in ~/server.txt • Move notes to the wiki • Custom scripts (setup.sh)
19. Managing Complexity 19 • SSH, make with the typey typey • Keep notes in ~/server.txt • Move notes to the wiki • Custom scripts (setup.sh) • Golden Images
20. Golden Images are not the answer • Gold is heavy • Hard to transport • Hard to mold • Easy to lose configuration detail http://www.flickr.com/photos/garysoup/2977173063/
21. Typical Infrastructure 21 Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite
22. New Compliance Mandate! 22 Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite • Move SSH off port 22 • Lets put it on 2022
23. 6 Golden Image Updates 23 Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite • edit /etc/ssh/sshd_config 1 2 3 4 5 6
24. 12 Instance Replacements 24 Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite • Delete, launch 1 2 3 4 5 6 7 8 9 10 11 12 • Repeat • Typically manually
25. Done in Maintenance Windows 25 • Don’t break anything! • Bob just got fired =( 5 Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite 1 2 4 5 6 7 8 9 10 11 12 3
26. Different IP Addresses? 26 Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite • Invalid configs!
27. Managing Complexity 27 • SSH, make with the typey typey • Keep notes in ~/server.txt • Move notes to the wiki • Custom scripts (setup.sh) • Golden Images • Policy-driven configuration management
28. Policies 28 • Declarations about the state of thing in a system • applied repeatedly and repair the system when needed • often change
29. Repeatable Operations • Idempotent • can be applied an infinite number of times and yield the same result every time • Convergent • test state and repair if needed 29
30. Autonomous agent Policy: The box should be closed
31. Autonomous agent Policy: The box should be closed
32. Policy Evolves 31
33. Following Policy • A control loop keeps the system stable and allows for change when policy is updated 32
34. Policy Evolves 33
35. Policy Evolves 33 That's great and all, but tell me about Chef!
36. What is Chef? • Open source framework for managing complexity • Infrastructure as code • a domain-specific language (DSL) for describing convergent operations • A community of professionals • A company 34
37. How does Chef work? • Ensure desired state by continually testing and repairing individual resources in the system • Compose policies using a series of abstractions 35
38. Desired Configuration Node Chef Server chef-client What policy should I follow? 36
39. Desired Configuration Node Chef Server chef-client What policy should I follow? "recipe[ntp::client]" "recipe[users]" "role[webserver]" 37
40. Desired Configuration Chef Server chef-client What policy should I follow? "recipe[ntp::client]" "recipe[users]" "role[webserver]" 38
42. Recipes package "apache2" 40 resource one
43. Recipes package "apache2" template "/etc/apache2/apache2.conf" do source "apache2.conf.erb" owner "root" group "root" mode "0644" variables(:allow_override => "All") notifies :reload, "service[apache2]" end 41 resource one resource two
44. Recipes package "apache2" template "/etc/apache2/apache2.conf" do source "apache2.conf.erb" owner "root" group "root" mode "0644" variables(:allow_override => "All") notifies :reload, "service[apache2]" end service "apache2" do action [:enable,:start] supports :reload => true end 42 resource one resource two resource three
45. Recipes 43 resource one
46. Recipes 44 resource one resource two
47. Recipes 45 resource one resource two resource three
48. Built-in Resources • package • template • service • cron • directory • mount • user • group • registry_key • remote_directory • route • ...and many more! 46
51. Managing Complexity • Organizations • Environments • Roles • Nodes • Recipes • Cookbooks • Search • Data 48
52. Their Infrastructure Organizations My Infrastructure Your Infrastructure 49
53. Environments Development Staging Production 50
54. Roles Load Balancers Application Servers DB Cache Database 51
55. Nodes 52
57. Search • Search for nodes with Roles • Find Topology Data • IP addresses • Hostnames • FQDNs http://www.flickr.com/photos/kathycsus/2686772625 54
58. Search for Nodes pool_members = search("node","role:webserver") template "/etc/haproxy/haproxy.cfg" do source "haproxy-app_lb.cfg.erb" owner "root" group "root" mode 0644 variables :pool_members => pool_members.uniq notifies :restart, "service[haproxy]" end 55
59. Search for Nodes pool_members = search("node","role:webserver") template "/etc/haproxy/haproxy.cfg" do source "haproxy-app_lb.cfg.erb" owner "root" group "root" mode 0644 variables :pool_members => pool_members.uniq notifies :restart, "service[haproxy]" end 56
60. Pass results into Templates # Set up application listeners here. listen application 0.0.0.0:80 balance roundrobin <% @pool_members.each do |member| -%> server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check <% end -%> <% if node["haproxy"]["enable_admin"] -%> listen admin 0.0.0.0:22002 mode http stats uri / <% end -%> 57
61. Pass results into Templates # Set up application listeners here. listen application 0.0.0.0:80 balance roundrobin <% @pool_members.each do |member| -%> server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check <% end -%> <% if node["haproxy"]["enable_admin"] -%> listen admin 0.0.0.0:22002 mode http stats uri / <% end -%> 58
62. # Set up application listeners here. listen application 0.0.0.0:80 balance roundrobin <% @pool_members.each do |member| -%> server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check <% end -%> <% if node["haproxy"]["enable_admin"] -%> listen admin 0.0.0.0:22002 mode http stats uri / <% end -%> Pass results into Templates 59
63. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite So when this... 60
64. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite ...becomes this 61
65. Memcache Postgres Slaves Postgres Master Nagios Graphite Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite ...this can happen automatically 62
66. Nagios Graphite Nagios Graphite Memcache Postgres Slaves • Load balancer config • Nagios host ping • Nagios host ssh • Nagios host HTTP • Nagios host app health • Graphite CPU • Graphite Memory • Graphite Disk • Graphite SNMP • Memcache firewall • Postgres firewall • Postgres authZ config • 12+ resource changes for 1 node addition Count the Resources Jboss App 63
67. Build Anything • Simple internal applications • Complex external applications • Workstations • Hadoop clusters • IaaS infrastructure • PaaS infrastructure • SaaS applications • Storage systems • You name it 64 http://www.flickr.com/photos/hyku/245010680/
68. And Manage it Simply • Automatically reconfigure everything • Linux, Windows, Unixes, BSDs • Load balancers • Metrics collection systems • Monitoring systems • Cloud migrations become trivial • 65 http://www.flickr.com/photos/helico/404640681/
69. 66 Knife
70. Community • Apache 2.0 Licensed • 1500+ Individual Contributors • 200+ Corporate Contributors • 1500+ Cookbooks 67
71. Time for More? • Ask me anything! • @nathenharvey 68