Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Introduction to Infrastructure as Code & Automation / Introduction to Chef

Introduction to Infrastructure as Code & Automation / Introduction to Chef

Your customers expect you to continuously deliver delightful experiences. This means that you'll need to continuously deliver application and infrastructure updates. Hand-crafted servers lovingly built and maintained by a system administrator are a thing of the past. Golden images are fine for initial provisioning but will quickly fail as your configuration requirements change over time.

It's time for you to fully automate the provisioning and management of your infrastructure components. Welcome to the world of infrastructure as code! In this new world, you'll be able to programmatically provision and configure the components of your infrastructure.

Disposable infrastructure whose provisioning, configuration, and on-going maintenance is fully automated allow you to change the way you build and deliver applications. Move your applications and infrastructure towards continuous delivery.

In this talk, we'll explore the ideas behind "infrastructure as code" and, specifically, look at how Chef allows you to fully automate your infrastructure. If you're brave enough, we'll even let you get your hands on some Chef and experience the delight of using Chef to build and deploy some infrastructure components.

Nathen Harvey

April 03, 2014
Tweet

More Decks by Nathen Harvey

Other Decks in Technology

Transcript

  1. Introduction to Infrastructure as
    Code & Automation /
    Introduction to Chef
    1
    Nathen Harvey
    @nathenharvey
    [email protected]
    github.com/nathenharvey

    View Slide

  2. Nathen Harvey
    • Director, Community and #learnchef
    • Co-host of the Food Fight Show Podcast
    • Meetup Organizer
    • DevOpsDC
    • @nathenharvey
    [email protected]

    View Slide

  3. Demo

    View Slide

  4. Who are you?
    • Were you in Isaac’s talk?
    • Were you in my talk yesterday?
    4

    View Slide

  5. Who are you?
    • System administrator?
    5

    View Slide

  6. Who are you?
    • System administrator?
    • Developer?
    6

    View Slide

  7. Who are you?
    • System administrator?
    • Developer?
    • Ruby Developer?
    7

    View Slide

  8. Who are you?
    • System administrator?
    • Developer?
    • Ruby Developer?
    • DevOp?
    8

    View Slide

  9. Who are you?
    • Experience with Configuration Management?
    • Experience with Chef?
    9

    View Slide

  10. Who are you?
    • Favorite text editor?
    10

    View Slide

  11. Who are you?
    • Which version control system do you use?
    •cp foo foo.bak
    11

    View Slide

  12. Who are you?
    • Which version control system do you use?
    •cp foo foo.bak
    •cp foo{,.`date +%Y%m%d%H%M`}
    12

    View Slide

  13. Who are you?
    • Which version control system do you use?
    •cp foo foo.bak
    •cp foo{,.`date +%Y%m%d%H%M`-`$USER`}
    13

    View Slide

  14. http://www.flickr.com/photos/michaelheiss/3090102907/
    Complexity
    14

    View Slide

  15. Managing Complexity
    15
    • SSH, make with the typey typey

    View Slide

  16. Managing Complexity
    16
    • SSH, make with the typey typey
    • Keep notes in ~/server.txt

    View Slide

  17. Managing Complexity
    17
    • SSH, make with the typey typey
    • Keep notes in ~/server.txt
    • Move notes to the wiki

    View Slide

  18. Managing Complexity
    18
    • SSH, make with the typey typey
    • Keep notes in ~/server.txt
    • Move notes to the wiki
    • Custom scripts (setup.sh)

    View Slide

  19. Managing Complexity
    19
    • SSH, make with the typey typey
    • Keep notes in ~/server.txt
    • Move notes to the wiki
    • Custom scripts (setup.sh)
    • Golden Images

    View Slide

  20. Golden Images are not the answer
    • Gold is heavy
    • Hard to transport
    • Hard to mold
    • Easy to lose
    configuration detail
    http://www.flickr.com/photos/garysoup/2977173063/

    View Slide

  21. Typical Infrastructure
    21
    Jboss App
    Memcache
    Postgres Slaves
    Postgres Master
    Nagios
    Graphite

    View Slide

  22. New Compliance Mandate!
    22
    Jboss App
    Memcache
    Postgres Slaves
    Postgres Master
    Nagios
    Graphite
    • Move SSH off port 22
    • Lets put it on 2022

    View Slide

  23. 6 Golden Image Updates
    23
    Jboss App
    Memcache
    Postgres Slaves
    Postgres Master
    Nagios
    Graphite
    • edit /etc/ssh/sshd_config
    1 2
    3
    4
    5
    6

    View Slide

  24. 12 Instance Replacements
    24
    Jboss App
    Memcache
    Postgres Slaves
    Postgres Master
    Nagios
    Graphite
    • Delete, launch
    1 2
    3 4 5 6 7
    8 9
    10 11
    12
    • Repeat
    • Typically manually

    View Slide

  25. Done in Maintenance Windows
    25
    • Don’t break anything!
    • Bob just got fired =(
    5
    Jboss App
    Memcache
    Postgres Slaves
    Postgres Master
    Nagios
    Graphite 1 2
    4 5 6 7
    8 9
    10 11
    12
    3

    View Slide

  26. Different IP Addresses?
    26
    Jboss App
    Memcache
    Postgres Slaves
    Postgres Master
    Nagios
    Graphite
    • Invalid configs!

    View Slide

  27. Managing Complexity
    27
    • SSH, make with the typey typey
    • Keep notes in ~/server.txt
    • Move notes to the wiki
    • Custom scripts (setup.sh)
    • Golden Images
    • Policy-driven configuration management

    View Slide

  28. Policies
    28
    • Declarations about the state of thing in a system
    • applied repeatedly and repair the system when
    needed
    • often change

    View Slide

  29. Repeatable Operations
    • Idempotent
    • can be applied an infinite number of times and
    yield the same result every time
    • Convergent
    • test state and repair if needed
    29

    View Slide

  30. Autonomous agent
    Policy: The box should be closed

    View Slide

  31. Autonomous agent
    Policy: The box should be closed

    View Slide

  32. Policy Evolves
    31

    View Slide

  33. Following Policy
    • A control loop keeps the system stable and allows
    for change when policy is updated
    32

    View Slide

  34. Policy Evolves
    33

    View Slide

  35. Policy Evolves
    33
    That's great and all,
    but tell me about
    Chef!

    View Slide

  36. What is Chef?
    • Open source framework for managing complexity
    • Infrastructure as code
    • a domain-specific language (DSL) for describing
    convergent operations
    • A community of professionals
    • A company
    34

    View Slide

  37. How does Chef work?
    • Ensure desired state by continually testing and
    repairing individual resources in the system
    • Compose policies using a series of abstractions
    35

    View Slide

  38. Desired Configuration
    Node
    Chef
    Server
    chef-client
    What policy should I follow?
    36

    View Slide

  39. Desired Configuration
    Node
    Chef
    Server
    chef-client
    What policy should I follow?
    "recipe[ntp::client]"
    "recipe[users]"
    "role[webserver]"
    37

    View Slide

  40. Desired Configuration
    Chef
    Server
    chef-client
    What policy should I follow?
    "recipe[ntp::client]"
    "recipe[users]"
    "role[webserver]"
    38

    View Slide

  41. View Slide

  42. Recipes
    package "apache2"
    40
    resource one

    View Slide

  43. Recipes
    package "apache2"
    template "/etc/apache2/apache2.conf" do
    source "apache2.conf.erb"
    owner "root"
    group "root"
    mode "0644"
    variables(:allow_override => "All")
    notifies :reload, "service[apache2]"
    end
    41
    resource one
    resource two

    View Slide

  44. Recipes
    package "apache2"
    template "/etc/apache2/apache2.conf" do
    source "apache2.conf.erb"
    owner "root"
    group "root"
    mode "0644"
    variables(:allow_override => "All")
    notifies :reload, "service[apache2]"
    end
    service "apache2" do
    action [:enable,:start]
    supports :reload => true
    end
    42
    resource one
    resource two
    resource three

    View Slide

  45. Recipes
    43
    resource one

    View Slide

  46. Recipes
    44
    resource one
    resource two

    View Slide

  47. Recipes
    45
    resource one
    resource two
    resource three

    View Slide

  48. Built-in Resources
    • package
    • template
    • service
    • cron
    • directory
    • mount
    • user
    • group
    • registry_key
    • remote_directory
    • route
    • ...and many more!
    46

    View Slide

  49. View Slide

  50. View Slide

  51. Managing Complexity
    • Organizations
    • Environments
    • Roles
    • Nodes
    • Recipes
    • Cookbooks
    • Search
    • Data
    48

    View Slide

  52. Their Infrastructure
    Organizations
    My Infrastructure Your Infrastructure
    49

    View Slide

  53. Environments
    Development Staging Production
    50

    View Slide

  54. Roles
    Load Balancers
    Application Servers
    DB Cache
    Database
    51

    View Slide

  55. Nodes
    52

    View Slide

  56. View Slide

  57. Search
    • Search for nodes with Roles
    • Find Topology Data
    • IP addresses
    • Hostnames
    • FQDNs
    http://www.flickr.com/photos/kathycsus/2686772625
    54

    View Slide

  58. Search for Nodes
    pool_members = search("node","role:webserver")
    template "/etc/haproxy/haproxy.cfg" do
    source "haproxy-app_lb.cfg.erb"
    owner "root"
    group "root"
    mode 0644
    variables :pool_members => pool_members.uniq
    notifies :restart, "service[haproxy]"
    end
    55

    View Slide

  59. Search for Nodes
    pool_members = search("node","role:webserver")
    template "/etc/haproxy/haproxy.cfg" do
    source "haproxy-app_lb.cfg.erb"
    owner "root"
    group "root"
    mode 0644
    variables :pool_members => pool_members.uniq
    notifies :restart, "service[haproxy]"
    end
    56

    View Slide

  60. Pass results into Templates
    # Set up application listeners here.
    listen application 0.0.0.0:80
    balance roundrobin

    server :>
    weight 1 maxconn 1 check


    listen admin 0.0.0.0:22002
    mode http
    stats uri /

    57

    View Slide

  61. Pass results into Templates
    # Set up application listeners here.
    listen application 0.0.0.0:80
    balance roundrobin

    server :>
    weight 1 maxconn 1 check


    listen admin 0.0.0.0:22002
    mode http
    stats uri /

    58

    View Slide

  62. # Set up application listeners here.
    listen application 0.0.0.0:80
    balance roundrobin

    server :>
    weight 1 maxconn 1 check


    listen admin 0.0.0.0:22002
    mode http
    stats uri /

    Pass results into Templates
    59

    View Slide

  63. Jboss App
    Memcache
    Postgres Slaves
    Postgres Master
    Nagios
    Graphite
    So when this...
    60

    View Slide

  64. Jboss App
    Memcache
    Postgres Slaves
    Postgres Master
    Nagios
    Graphite
    ...becomes this
    61

    View Slide

  65. Memcache
    Postgres Slaves
    Postgres Master
    Nagios
    Graphite
    Jboss App
    Memcache
    Postgres Slaves
    Postgres Master
    Nagios
    Graphite
    ...this can happen automatically
    62

    View Slide

  66. Nagios
    Graphite Nagios
    Graphite
    Memcache
    Postgres Slaves
    • Load balancer config
    • Nagios host ping
    • Nagios host ssh
    • Nagios host HTTP
    • Nagios host app health
    • Graphite CPU
    • Graphite Memory
    • Graphite Disk
    • Graphite SNMP
    • Memcache firewall
    • Postgres firewall
    • Postgres authZ config
    • 12+ resource changes for 1 node addition
    Count the Resources
    Jboss App
    63

    View Slide

  67. Build Anything
    • Simple internal applications
    • Complex external
    applications
    • Workstations
    • Hadoop clusters
    • IaaS infrastructure
    • PaaS infrastructure
    • SaaS applications
    • Storage systems
    • You name it
    64
    http://www.flickr.com/photos/hyku/245010680/

    View Slide

  68. And Manage it Simply
    • Automatically reconfigure
    everything
    • Linux, Windows, Unixes,
    BSDs
    • Load balancers
    • Metrics collection systems
    • Monitoring systems
    • Cloud migrations become
    trivial

    65
    http://www.flickr.com/photos/helico/404640681/

    View Slide

  69. 66
    Knife

    View Slide

  70. Community
    • Apache 2.0 Licensed
    • 1500+ Individual Contributors
    • 200+ Corporate Contributors
    • 1500+ Cookbooks
    67

    View Slide

  71. Time for More?
    • Ask me anything!
    • @nathenharvey
    68

    View Slide