Service Discovery and Configuration Management

Service Discovery and Configuration Management

A talk given at Loadays, all about the need for two speeds of configuration - modelled and emergent. Includes an example of using Puppet along with Consul.

98234c645fe8c935edc0fec0186d28b8?s=128

Gareth Rushgrove

April 11, 2015
Tweet

Transcript

  1. Service Discovery and Configuration Management Puppet Labs Gareth Rushgrove Two

    speeds of configuration
  2. @garethr

  3. Gareth Rushgrove

  4. Gareth Rushgrove

  5. This talk

  6. A quick introduction to Service Discovery Gareth Rushgrove

  7. Why it’s useful to have two speeds of configuration Gareth

    Rushgrove
  8. An example of using Consul alongside Puppet Gareth Rushgrove

  9. Some brief thoughts about change control and autonomous systems Gareth

    Rushgrove
  10. What is Service Discovery?

  11. Gareth Rushgrove

  12. Service discovery is a key component of most distributed systems

    and service oriented architectures. Gareth Rushgrove http://jasonwilder.com/blog/2014/02/04/service-discovery-in-the-cloud/
  13. The problem seems simple at first: How do clients determine

    the IP and port for a service that exist on multiple hosts? Gareth Rushgrove http://jasonwilder.com/blog/2014/02/04/service-discovery-in-the-cloud/
  14. Etcd, Consul, Zookeeper Gareth Rushgrove

  15. Hadoop uses Zookeeper Socketplane uses Consul CoreOS uses etcd Cloud

    Foundry uses Consul Cloud Foundry also uses etcd Kubernetes uses etcd Gareth Rushgrove
  16. Gareth Rushgrove Thanks to etcd for the graphics

  17. Gareth Rushgrove Thanks to etcd for the graphics

  18. Gareth Rushgrove Thanks to etcd for the graphics

  19. Gareth Rushgrove Jepsen

  20. Why now?

  21. Every organisation wants to move quicker Gareth Rushgrove

  22. Provisioning times are decreasing rapidly Gareth Rushgrove

  23. The number of applications, and instances of those applications, is

    increasing rapidly Gareth Rushgrove
  24. The number of systems administrators is probably going to stay

    the same. Sorry. Gareth Rushgrove
  25. Technology moving towards a cluster centric view rather than a

    host centric view Gareth Rushgrove
  26. Hard-coding everything doesn’t scale Gareth Rushgrove

  27. Some relevant technology you may have heard of Gareth Rushgrove

  28. Gareth Rushgrove

  29. Gareth Rushgrove

  30. Gareth Rushgrove

  31. Gareth Rushgrove

  32. Two speeds of configuration

  33. Configuration management tools tend to focus on upfront modelling, managing

    configuration we know and want to control over time Gareth Rushgrove
  34. For example the version of a package or application, user

    permissions, security settings, system constrains Gareth Rushgrove
  35. Service discovery tools handle emergent configuration which is a property

    of the running system Gareth Rushgrove
  36. For example load balancer and proxy backends, network and firewall

    routes as applications come and go Gareth Rushgrove
  37. Wait, but I want to define some of my network

    configuration upfront Gareth Rushgrove
  38. Hence two speeds, model driven and emergent Gareth Rushgrove

  39. How to mix configuration management and service discovery

  40. We’ll use Consul for this example Gareth Rushgrove

  41. We’ll also use Puppet Gareth Rushgrove

  42. First lets install Consul everywhere Gareth Rushgrove

  43. Gareth Rushgrove

  44. That’s it Gareth Rushgrove

  45. Gareth Rushgrove KyleAnderson/consul

  46. We can configure servers and clients using Hiera Gareth Rushgrove

  47. Gareth Rushgrove

  48. Gareth Rushgrove

  49. Consul nodes need to register with a server Gareth Rushgrove

  50. Gareth Rushgrove

  51. Populating the Consul service registry Gareth Rushgrove

  52. Gareth Rushgrove

  53. Gareth Rushgrove Consul dashboard

  54. Create your own types Gareth Rushgrove

  55. Gareth Rushgrove

  56. Gareth Rushgrove

  57. Using information from Consul in Puppet Gareth Rushgrove

  58. Gareth Rushgrove

  59. Gareth Rushgrove

  60. Gareth Rushgrove

  61. Gareth Rushgrove lynxman/hiera_consul

  62. Trigger Puppet when something pertinent changes Gareth Rushgrove

  63. Gareth Rushgrove

  64. Gareth Rushgrove Demo time

  65. Same example possible with a bit of hacking for etcd

    Gareth Rushgrove
  66. Gareth Rushgrove garethr/hiera-etcd

  67. Gareth Rushgrove KyleAnderson/etcd

  68. Advantages Changes happen quickly, but without losing the ability to

    address configuration drift Gareth Rushgrove
  69. Advantages Reduce complexity of watchers, all changes lead to a

    Puppet run Gareth Rushgrove
  70. Advantages All changes to configuration, however triggered, are tracked in

    one place Gareth Rushgrove
  71. Advantages Define all watchers in code Gareth Rushgrove

  72. Advantages Reduce duplication, same code runs for new and existing

    instances Gareth Rushgrove
  73. Bonus distributed configuration Gareth Rushgrove

  74. Replace lots of identical local files on disk with distributed

    key/values Gareth Rushgrove
  75. Gareth Rushgrove

  76. Gareth Rushgrove

  77. Gareth Rushgrove garethr/key_value_config

  78. Conclusions

  79. Change control and autonomous systems Gareth Rushgrove

  80. The importance of building trust in automation Gareth Rushgrove

  81. Two speeds of configuration Gareth Rushgrove

  82. Questions? And thanks for listening