Introduction to Infrastructure as Code & Automation / Introduction to Chef

Your customers expect you to continuously deliver delightful experiences. This means that you'll need to continuously deliver application and infrastructure updates. Hand-crafted servers lovingly built and maintained by a system administrator are a thing of the past. Golden images are fine for initial provisioning but will quickly fail as your configuration requirements change over time.

It's time for you to fully automate the provisioning and management of your infrastructure components. Welcome to the world of infrastructure as code! In this new world, you'll be able to programmatically provision and configure the components of your infrastructure.

Disposable infrastructure whose provisioning, configuration, and on-going maintenance is fully automated allow you to change the way you build and deliver applications. Move your applications and infrastructure towards continuous delivery.

In this talk, we'll explore the ideas behind "infrastructure as code" and, specifically, look at how Chef allows you to fully automate your infrastructure. If you're brave enough, we'll even let you get your hands on some Chef and experience the delight of using Chef to build and deploy some infrastructure components.

Transcript

1. Introduction to Infrastructure as
   Code & Automation /
   Introduction to Chef
   1
   Nathen Harvey
   @nathenharvey
   nharvey@getchef.com
   github.com/nathenharvey
   

   View Slide

2. Nathen Harvey
   • Director, Community and #learnchef
   • Co-host of the Food Fight Show Podcast
   • Meetup Organizer
   • DevOpsDC
   • @nathenharvey
   • nharvey@getchef.com
   

   View Slide

3. Demo
   

   View Slide

4. Who are you?
   • Were you in Isaac’s talk?
   • Were you in my talk yesterday?
   4
   

   View Slide

5. Who are you?
   • System administrator?
   5
   

   View Slide

6. Who are you?
   • System administrator?
   • Developer?
   6
   

   View Slide

7. Who are you?
   • System administrator?
   • Developer?
   • Ruby Developer?
   7
   

   View Slide

8. Who are you?
   • System administrator?
   • Developer?
   • Ruby Developer?
   • DevOp?
   8
   

   View Slide

9. Who are you?
   • Experience with Configuration Management?
   • Experience with Chef?
   9
   

   View Slide

10. Who are you?
    • Favorite text editor?
    10
    

    View Slide

11. Who are you?
    • Which version control system do you use?
    •cp foo foo.bak
    11
    

    View Slide

12. Who are you?
    • Which version control system do you use?
    •cp foo foo.bak
    •cp foo{,.`date +%Y%m%d%H%M`}
    12
    

    View Slide

13. Who are you?
    • Which version control system do you use?
    •cp foo foo.bak
    •cp foo{,.`date +%Y%m%d%H%M`-`$USER`}
    13
    

    View Slide

14. http://www.flickr.com/photos/michaelheiss/3090102907/
    Complexity
    14
    

    View Slide

15. Managing Complexity
    15
    • SSH, make with the typey typey
    

    View Slide

16. Managing Complexity
    16
    • SSH, make with the typey typey
    • Keep notes in ~/server.txt
    

    View Slide

17. Managing Complexity
    17
    • SSH, make with the typey typey
    • Keep notes in ~/server.txt
    • Move notes to the wiki
    

    View Slide

18. Managing Complexity
    18
    • SSH, make with the typey typey
    • Keep notes in ~/server.txt
    • Move notes to the wiki
    • Custom scripts (setup.sh)
    

    View Slide

19. Managing Complexity
    19
    • SSH, make with the typey typey
    • Keep notes in ~/server.txt
    • Move notes to the wiki
    • Custom scripts (setup.sh)
    • Golden Images
    

    View Slide

20. Golden Images are not the answer
    • Gold is heavy
    • Hard to transport
    • Hard to mold
    • Easy to lose
    configuration detail
    http://www.flickr.com/photos/garysoup/2977173063/
    

    View Slide

21. Typical Infrastructure
    21
    Jboss App
    Memcache
    Postgres Slaves
    Postgres Master
    Nagios
    Graphite
    

    View Slide

22. New Compliance Mandate!
    22
    Jboss App
    Memcache
    Postgres Slaves
    Postgres Master
    Nagios
    Graphite
    • Move SSH off port 22
    • Lets put it on 2022
    

    View Slide

23. 6 Golden Image Updates
    23
    Jboss App
    Memcache
    Postgres Slaves
    Postgres Master
    Nagios
    Graphite
    • edit /etc/ssh/sshd_config
    1 2
    3
    4
    5
    6
    

    View Slide

24. 12 Instance Replacements
    24
    Jboss App
    Memcache
    Postgres Slaves
    Postgres Master
    Nagios
    Graphite
    • Delete, launch
    1 2
    3 4 5 6 7
    8 9
    10 11
    12
    • Repeat
    • Typically manually
    

    View Slide

25. Done in Maintenance Windows
    25
    • Don’t break anything!
    • Bob just got fired =(
    5
    Jboss App
    Memcache
    Postgres Slaves
    Postgres Master
    Nagios
    Graphite 1 2
    4 5 6 7
    8 9
    10 11
    12
    3
    

    View Slide

26. Different IP Addresses?
    26
    Jboss App
    Memcache
    Postgres Slaves
    Postgres Master
    Nagios
    Graphite
    • Invalid configs!
    

    View Slide

27. Managing Complexity
    27
    • SSH, make with the typey typey
    • Keep notes in ~/server.txt
    • Move notes to the wiki
    • Custom scripts (setup.sh)
    • Golden Images
    • Policy-driven configuration management
    

    View Slide

28. Policies
    28
    • Declarations about the state of thing in a system
    • applied repeatedly and repair the system when
    needed
    • often change
    

    View Slide

29. Repeatable Operations
    • Idempotent
    • can be applied an infinite number of times and
    yield the same result every time
    • Convergent
    • test state and repair if needed
    29
    

    View Slide

30. Autonomous agent
    Policy: The box should be closed
    

    View Slide

31. Autonomous agent
    Policy: The box should be closed
    

    View Slide

32. Policy Evolves
    31
    

    View Slide

33. Following Policy
    • A control loop keeps the system stable and allows
    for change when policy is updated
    32
    

    View Slide

34. Policy Evolves
    33
    

    View Slide

35. Policy Evolves
    33
    That's great and all,
    but tell me about
    Chef!
    

    View Slide

36. What is Chef?
    • Open source framework for managing complexity
    • Infrastructure as code
    • a domain-specific language (DSL) for describing
    convergent operations
    • A community of professionals
    • A company
    34
    

    View Slide

37. How does Chef work?
    • Ensure desired state by continually testing and
    repairing individual resources in the system
    • Compose policies using a series of abstractions
    35
    

    View Slide

38. Desired Configuration
    Node
    Chef
    Server
    chef-client
    What policy should I follow?
    36
    

    View Slide

39. Desired Configuration
    Node
    Chef
    Server
    chef-client
    What policy should I follow?
    "recipe[ntp::client]"
    "recipe[users]"
    "role[webserver]"
    37
    

    View Slide

40. Desired Configuration
    Chef
    Server
    chef-client
    What policy should I follow?
    "recipe[ntp::client]"
    "recipe[users]"
    "role[webserver]"
    38
    

    View Slide

41. View Slide

42. Recipes
    package "apache2"
    40
    resource one
    

    View Slide

43. Recipes
    package "apache2"
    template "/etc/apache2/apache2.conf" do
    source "apache2.conf.erb"
    owner "root"
    group "root"
    mode "0644"
    variables(:allow_override => "All")
    notifies :reload, "service[apache2]"
    end
    41
    resource one
    resource two
    

    View Slide

44. Recipes
    package "apache2"
    template "/etc/apache2/apache2.conf" do
    source "apache2.conf.erb"
    owner "root"
    group "root"
    mode "0644"
    variables(:allow_override => "All")
    notifies :reload, "service[apache2]"
    end
    service "apache2" do
    action [:enable,:start]
    supports :reload => true
    end
    42
    resource one
    resource two
    resource three
    

    View Slide

45. Recipes
    43
    resource one
    

    View Slide

46. Recipes
    44
    resource one
    resource two
    

    View Slide

47. Recipes
    45
    resource one
    resource two
    resource three
    

    View Slide

48. Built-in Resources
    • package
    • template
    • service
    • cron
    • directory
    • mount
    • user
    • group
    • registry_key
    • remote_directory
    • route
    • ...and many more!
    46
    

    View Slide

49. View Slide

50. View Slide

51. Managing Complexity
    • Organizations
    • Environments
    • Roles
    • Nodes
    • Recipes
    • Cookbooks
    • Search
    • Data
    48
    

    View Slide

52. Their Infrastructure
    Organizations
    My Infrastructure Your Infrastructure
    49
    

    View Slide

53. Environments
    Development Staging Production
    50
    

    View Slide

54. Roles
    Load Balancers
    Application Servers
    DB Cache
    Database
    51
    

    View Slide

55. Nodes
    52
    

    View Slide

56. View Slide

57. Search
    • Search for nodes with Roles
    • Find Topology Data
    • IP addresses
    • Hostnames
    • FQDNs
    http://www.flickr.com/photos/kathycsus/2686772625
    54
    

    View Slide

58. Search for Nodes
    pool_members = search("node","role:webserver")
    template "/etc/haproxy/haproxy.cfg" do
    source "haproxy-app_lb.cfg.erb"
    owner "root"
    group "root"
    mode 0644
    variables :pool_members => pool_members.uniq
    notifies :restart, "service[haproxy]"
    end
    55
    

    View Slide

59. Search for Nodes
    pool_members = search("node","role:webserver")
    template "/etc/haproxy/haproxy.cfg" do
    source "haproxy-app_lb.cfg.erb"
    owner "root"
    group "root"
    mode 0644
    variables :pool_members => pool_members.uniq
    notifies :restart, "service[haproxy]"
    end
    56
    

    View Slide

60. Pass results into Templates
    # Set up application listeners here.
    listen application 0.0.0.0:80
    balance roundrobin
    
    server :>
    weight 1 maxconn 1 check
    
    
    listen admin 0.0.0.0:22002
    mode http
    stats uri /
    
    57
    

    View Slide

61. Pass results into Templates
    # Set up application listeners here.
    listen application 0.0.0.0:80
    balance roundrobin
    
    server :>
    weight 1 maxconn 1 check
    
    
    listen admin 0.0.0.0:22002
    mode http
    stats uri /
    
    58
    

    View Slide

62. # Set up application listeners here.
    listen application 0.0.0.0:80
    balance roundrobin
    
    server :>
    weight 1 maxconn 1 check
    
    
    listen admin 0.0.0.0:22002
    mode http
    stats uri /
    
    Pass results into Templates
    59
    

    View Slide

63. Jboss App
    Memcache
    Postgres Slaves
    Postgres Master
    Nagios
    Graphite
    So when this...
    60
    

    View Slide

64. Jboss App
    Memcache
    Postgres Slaves
    Postgres Master
    Nagios
    Graphite
    ...becomes this
    61
    

    View Slide

65. Memcache
    Postgres Slaves
    Postgres Master
    Nagios
    Graphite
    Jboss App
    Memcache
    Postgres Slaves
    Postgres Master
    Nagios
    Graphite
    ...this can happen automatically
    62
    

    View Slide

66. Nagios
    Graphite Nagios
    Graphite
    Memcache
    Postgres Slaves
    • Load balancer config
    • Nagios host ping
    • Nagios host ssh
    • Nagios host HTTP
    • Nagios host app health
    • Graphite CPU
    • Graphite Memory
    • Graphite Disk
    • Graphite SNMP
    • Memcache firewall
    • Postgres firewall
    • Postgres authZ config
    • 12+ resource changes for 1 node addition
    Count the Resources
    Jboss App
    63
    

    View Slide

67. Build Anything
    • Simple internal applications
    • Complex external
    applications
    • Workstations
    • Hadoop clusters
    • IaaS infrastructure
    • PaaS infrastructure
    • SaaS applications
    • Storage systems
    • You name it
    64
    http://www.flickr.com/photos/hyku/245010680/
    

    View Slide

68. And Manage it Simply
    • Automatically reconfigure
    everything
    • Linux, Windows, Unixes,
    BSDs
    • Load balancers
    • Metrics collection systems
    • Monitoring systems
    • Cloud migrations become
    trivial
    •
    65
    http://www.flickr.com/photos/helico/404640681/
    

    View Slide

69. 66
    Knife
    

    View Slide

70. Community
    • Apache 2.0 Licensed
    • 1500+ Individual Contributors
    • 200+ Corporate Contributors
    • 1500+ Cookbooks
    67
    

    View Slide

71. Time for More?
    • Ask me anything!
    • @nathenharvey
    68
    

    View Slide