Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Service Discovery and Configuration Management

Service Discovery and Configuration Management

A talk given at Loadays, all about the need for two speeds of configuration - modelled and emergent. Includes an example of using Puppet along with Consul.

Gareth Rushgrove

April 11, 2015
Tweet

More Decks by Gareth Rushgrove

Other Decks in Technology

Transcript

  1. Service Discovery and
    Configuration Management
    Puppet Labs
    Gareth Rushgrove
    Two speeds of configuration

    View full-size slide

  2. Gareth Rushgrove

    View full-size slide

  3. Gareth Rushgrove

    View full-size slide

  4. A quick introduction to
    Service Discovery
    Gareth Rushgrove

    View full-size slide

  5. Why it’s useful to have two
    speeds of configuration
    Gareth Rushgrove

    View full-size slide

  6. An example of using Consul
    alongside Puppet
    Gareth Rushgrove

    View full-size slide

  7. Some brief thoughts about
    change control and autonomous
    systems
    Gareth Rushgrove

    View full-size slide

  8. What is Service
    Discovery?

    View full-size slide

  9. Gareth Rushgrove

    View full-size slide

  10. Service discovery is a key
    component of most distributed
    systems and service oriented
    architectures.
    Gareth Rushgrove
    http://jasonwilder.com/blog/2014/02/04/service-discovery-in-the-cloud/

    View full-size slide

  11. The problem seems simple at
    first: How do clients determine
    the IP and port for a service that
    exist on multiple hosts?
    Gareth Rushgrove
    http://jasonwilder.com/blog/2014/02/04/service-discovery-in-the-cloud/

    View full-size slide

  12. Etcd, Consul, Zookeeper
    Gareth Rushgrove

    View full-size slide

  13. Hadoop uses Zookeeper
    Socketplane uses Consul
    CoreOS uses etcd
    Cloud Foundry uses Consul
    Cloud Foundry also uses etcd
    Kubernetes uses etcd
    Gareth Rushgrove

    View full-size slide

  14. Gareth Rushgrove
    Thanks to etcd for the graphics

    View full-size slide

  15. Gareth Rushgrove
    Thanks to etcd for the graphics

    View full-size slide

  16. Gareth Rushgrove
    Thanks to etcd for the graphics

    View full-size slide

  17. Gareth Rushgrove
    Jepsen

    View full-size slide

  18. Every organisation wants to
    move quicker
    Gareth Rushgrove

    View full-size slide

  19. Provisioning times are
    decreasing rapidly
    Gareth Rushgrove

    View full-size slide

  20. The number of applications, and
    instances of those applications, is
    increasing rapidly
    Gareth Rushgrove

    View full-size slide

  21. The number of systems
    administrators is probably going
    to stay the same. Sorry.
    Gareth Rushgrove

    View full-size slide

  22. Technology moving towards a
    cluster centric view rather than a
    host centric view
    Gareth Rushgrove

    View full-size slide

  23. Hard-coding everything
    doesn’t scale
    Gareth Rushgrove

    View full-size slide

  24. Some relevant technology you
    may have heard of
    Gareth Rushgrove

    View full-size slide

  25. Gareth Rushgrove

    View full-size slide

  26. Gareth Rushgrove

    View full-size slide

  27. Gareth Rushgrove

    View full-size slide

  28. Gareth Rushgrove

    View full-size slide

  29. Two speeds of
    configuration

    View full-size slide

  30. Configuration management tools
    tend to focus on upfront
    modelling, managing
    configuration we know and want
    to control over time
    Gareth Rushgrove

    View full-size slide

  31. For example the version of a
    package or application, user
    permissions, security settings,
    system constrains
    Gareth Rushgrove

    View full-size slide

  32. Service discovery tools handle
    emergent configuration which is
    a property of the running system
    Gareth Rushgrove

    View full-size slide

  33. For example load balancer and
    proxy backends, network and
    firewall routes as applications
    come and go
    Gareth Rushgrove

    View full-size slide

  34. Wait, but I want to define some of
    my network configuration upfront
    Gareth Rushgrove

    View full-size slide

  35. Hence two speeds, model
    driven and emergent
    Gareth Rushgrove

    View full-size slide

  36. How to mix
    configuration
    management and
    service discovery

    View full-size slide

  37. We’ll use Consul for this
    example
    Gareth Rushgrove

    View full-size slide

  38. We’ll also use Puppet
    Gareth Rushgrove

    View full-size slide

  39. First lets install Consul
    everywhere
    Gareth Rushgrove

    View full-size slide

  40. Gareth Rushgrove

    View full-size slide

  41. That’s it
    Gareth Rushgrove

    View full-size slide

  42. Gareth Rushgrove
    KyleAnderson/consul

    View full-size slide

  43. We can configure servers
    and clients using Hiera
    Gareth Rushgrove

    View full-size slide

  44. Gareth Rushgrove

    View full-size slide

  45. Gareth Rushgrove

    View full-size slide

  46. Consul nodes need to
    register with a server
    Gareth Rushgrove

    View full-size slide

  47. Gareth Rushgrove

    View full-size slide

  48. Populating the Consul
    service registry
    Gareth Rushgrove

    View full-size slide

  49. Gareth Rushgrove

    View full-size slide

  50. Gareth Rushgrove
    Consul dashboard

    View full-size slide

  51. Create your own types
    Gareth Rushgrove

    View full-size slide

  52. Gareth Rushgrove

    View full-size slide

  53. Gareth Rushgrove

    View full-size slide

  54. Using information from
    Consul in Puppet
    Gareth Rushgrove

    View full-size slide

  55. Gareth Rushgrove

    View full-size slide

  56. Gareth Rushgrove

    View full-size slide

  57. Gareth Rushgrove

    View full-size slide

  58. Gareth Rushgrove
    lynxman/hiera_consul

    View full-size slide

  59. Trigger Puppet when
    something pertinent
    changes
    Gareth Rushgrove

    View full-size slide

  60. Gareth Rushgrove

    View full-size slide

  61. Gareth Rushgrove
    Demo time

    View full-size slide

  62. Same example possible
    with a bit of hacking
    for etcd
    Gareth Rushgrove

    View full-size slide

  63. Gareth Rushgrove
    garethr/hiera-etcd

    View full-size slide

  64. Gareth Rushgrove
    KyleAnderson/etcd

    View full-size slide

  65. Advantages
    Changes happen quickly,
    but without losing the
    ability to address
    configuration drift
    Gareth Rushgrove

    View full-size slide

  66. Advantages
    Reduce complexity of
    watchers, all changes lead
    to a Puppet run
    Gareth Rushgrove

    View full-size slide

  67. Advantages
    All changes to
    configuration, however
    triggered, are tracked in
    one place
    Gareth Rushgrove

    View full-size slide

  68. Advantages
    Define all watchers
    in code
    Gareth Rushgrove

    View full-size slide

  69. Advantages
    Reduce duplication, same
    code runs for new and
    existing instances
    Gareth Rushgrove

    View full-size slide

  70. Bonus distributed
    configuration
    Gareth Rushgrove

    View full-size slide

  71. Replace lots of identical
    local files on disk with
    distributed key/values
    Gareth Rushgrove

    View full-size slide

  72. Gareth Rushgrove

    View full-size slide

  73. Gareth Rushgrove

    View full-size slide

  74. Gareth Rushgrove
    garethr/key_value_config

    View full-size slide

  75. Change control and autonomous
    systems
    Gareth Rushgrove

    View full-size slide

  76. The importance of building trust
    in automation
    Gareth Rushgrove

    View full-size slide

  77. Two speeds of configuration
    Gareth Rushgrove

    View full-size slide

  78. Questions?
    And thanks for listening

    View full-size slide