Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Service Discovery and Configuration Management

Service Discovery and Configuration Management

A talk given at Loadays, all about the need for two speeds of configuration - modelled and emergent. Includes an example of using Puppet along with Consul.

Gareth Rushgrove

April 11, 2015
Tweet

More Decks by Gareth Rushgrove

Other Decks in Technology

Transcript

  1. Service Discovery and
    Configuration Management
    Puppet Labs
    Gareth Rushgrove
    Two speeds of configuration

    View Slide

  2. @garethr

    View Slide

  3. Gareth Rushgrove

    View Slide

  4. Gareth Rushgrove

    View Slide

  5. This talk

    View Slide

  6. A quick introduction to
    Service Discovery
    Gareth Rushgrove

    View Slide

  7. Why it’s useful to have two
    speeds of configuration
    Gareth Rushgrove

    View Slide

  8. An example of using Consul
    alongside Puppet
    Gareth Rushgrove

    View Slide

  9. Some brief thoughts about
    change control and autonomous
    systems
    Gareth Rushgrove

    View Slide

  10. What is Service
    Discovery?

    View Slide

  11. Gareth Rushgrove

    View Slide

  12. Service discovery is a key
    component of most distributed
    systems and service oriented
    architectures.
    Gareth Rushgrove
    http://jasonwilder.com/blog/2014/02/04/service-discovery-in-the-cloud/

    View Slide

  13. The problem seems simple at
    first: How do clients determine
    the IP and port for a service that
    exist on multiple hosts?
    Gareth Rushgrove
    http://jasonwilder.com/blog/2014/02/04/service-discovery-in-the-cloud/

    View Slide

  14. Etcd, Consul, Zookeeper
    Gareth Rushgrove

    View Slide

  15. Hadoop uses Zookeeper
    Socketplane uses Consul
    CoreOS uses etcd
    Cloud Foundry uses Consul
    Cloud Foundry also uses etcd
    Kubernetes uses etcd
    Gareth Rushgrove

    View Slide

  16. Gareth Rushgrove
    Thanks to etcd for the graphics

    View Slide

  17. Gareth Rushgrove
    Thanks to etcd for the graphics

    View Slide

  18. Gareth Rushgrove
    Thanks to etcd for the graphics

    View Slide

  19. Gareth Rushgrove
    Jepsen

    View Slide

  20. Why now?

    View Slide

  21. Every organisation wants to
    move quicker
    Gareth Rushgrove

    View Slide

  22. Provisioning times are
    decreasing rapidly
    Gareth Rushgrove

    View Slide

  23. The number of applications, and
    instances of those applications, is
    increasing rapidly
    Gareth Rushgrove

    View Slide

  24. The number of systems
    administrators is probably going
    to stay the same. Sorry.
    Gareth Rushgrove

    View Slide

  25. Technology moving towards a
    cluster centric view rather than a
    host centric view
    Gareth Rushgrove

    View Slide

  26. Hard-coding everything
    doesn’t scale
    Gareth Rushgrove

    View Slide

  27. Some relevant technology you
    may have heard of
    Gareth Rushgrove

    View Slide

  28. Gareth Rushgrove

    View Slide

  29. Gareth Rushgrove

    View Slide

  30. Gareth Rushgrove

    View Slide

  31. Gareth Rushgrove

    View Slide

  32. Two speeds of
    configuration

    View Slide

  33. Configuration management tools
    tend to focus on upfront
    modelling, managing
    configuration we know and want
    to control over time
    Gareth Rushgrove

    View Slide

  34. For example the version of a
    package or application, user
    permissions, security settings,
    system constrains
    Gareth Rushgrove

    View Slide

  35. Service discovery tools handle
    emergent configuration which is
    a property of the running system
    Gareth Rushgrove

    View Slide

  36. For example load balancer and
    proxy backends, network and
    firewall routes as applications
    come and go
    Gareth Rushgrove

    View Slide

  37. Wait, but I want to define some of
    my network configuration upfront
    Gareth Rushgrove

    View Slide

  38. Hence two speeds, model
    driven and emergent
    Gareth Rushgrove

    View Slide

  39. How to mix
    configuration
    management and
    service discovery

    View Slide

  40. We’ll use Consul for this
    example
    Gareth Rushgrove

    View Slide

  41. We’ll also use Puppet
    Gareth Rushgrove

    View Slide

  42. First lets install Consul
    everywhere
    Gareth Rushgrove

    View Slide

  43. Gareth Rushgrove

    View Slide

  44. That’s it
    Gareth Rushgrove

    View Slide

  45. Gareth Rushgrove
    KyleAnderson/consul

    View Slide

  46. We can configure servers
    and clients using Hiera
    Gareth Rushgrove

    View Slide

  47. Gareth Rushgrove

    View Slide

  48. Gareth Rushgrove

    View Slide

  49. Consul nodes need to
    register with a server
    Gareth Rushgrove

    View Slide

  50. Gareth Rushgrove

    View Slide

  51. Populating the Consul
    service registry
    Gareth Rushgrove

    View Slide

  52. Gareth Rushgrove

    View Slide

  53. Gareth Rushgrove
    Consul dashboard

    View Slide

  54. Create your own types
    Gareth Rushgrove

    View Slide

  55. Gareth Rushgrove

    View Slide

  56. Gareth Rushgrove

    View Slide

  57. Using information from
    Consul in Puppet
    Gareth Rushgrove

    View Slide

  58. Gareth Rushgrove

    View Slide

  59. Gareth Rushgrove

    View Slide

  60. Gareth Rushgrove

    View Slide

  61. Gareth Rushgrove
    lynxman/hiera_consul

    View Slide

  62. Trigger Puppet when
    something pertinent
    changes
    Gareth Rushgrove

    View Slide

  63. Gareth Rushgrove

    View Slide

  64. Gareth Rushgrove
    Demo time

    View Slide

  65. Same example possible
    with a bit of hacking
    for etcd
    Gareth Rushgrove

    View Slide

  66. Gareth Rushgrove
    garethr/hiera-etcd

    View Slide

  67. Gareth Rushgrove
    KyleAnderson/etcd

    View Slide

  68. Advantages
    Changes happen quickly,
    but without losing the
    ability to address
    configuration drift
    Gareth Rushgrove

    View Slide

  69. Advantages
    Reduce complexity of
    watchers, all changes lead
    to a Puppet run
    Gareth Rushgrove

    View Slide

  70. Advantages
    All changes to
    configuration, however
    triggered, are tracked in
    one place
    Gareth Rushgrove

    View Slide

  71. Advantages
    Define all watchers
    in code
    Gareth Rushgrove

    View Slide

  72. Advantages
    Reduce duplication, same
    code runs for new and
    existing instances
    Gareth Rushgrove

    View Slide

  73. Bonus distributed
    configuration
    Gareth Rushgrove

    View Slide

  74. Replace lots of identical
    local files on disk with
    distributed key/values
    Gareth Rushgrove

    View Slide

  75. Gareth Rushgrove

    View Slide

  76. Gareth Rushgrove

    View Slide

  77. Gareth Rushgrove
    garethr/key_value_config

    View Slide

  78. Conclusions

    View Slide

  79. Change control and autonomous
    systems
    Gareth Rushgrove

    View Slide

  80. The importance of building trust
    in automation
    Gareth Rushgrove

    View Slide

  81. Two speeds of configuration
    Gareth Rushgrove

    View Slide

  82. Questions?
    And thanks for listening

    View Slide