Service Discovery and Configuration Management

A talk given at Loadays, all about the need for two speeds of configuration - modelled and emergent. Includes an example of using Puppet along with Consul.

Transcript

1. Service Discovery and
   Configuration Management
   Puppet Labs
   Gareth Rushgrove
   Two speeds of configuration
   

   View Slide

2. @garethr
   

   View Slide

3. Gareth Rushgrove
   

   View Slide

4. Gareth Rushgrove
   

   View Slide

5. This talk
   

   View Slide

6. A quick introduction to
   Service Discovery
   Gareth Rushgrove
   

   View Slide

7. Why it’s useful to have two
   speeds of configuration
   Gareth Rushgrove
   

   View Slide

8. An example of using Consul
   alongside Puppet
   Gareth Rushgrove
   

   View Slide

9. Some brief thoughts about
   change control and autonomous
   systems
   Gareth Rushgrove
   

   View Slide

10. What is Service
    Discovery?
    

    View Slide

11. Gareth Rushgrove
    

    View Slide

12. Service discovery is a key
    component of most distributed
    systems and service oriented
    architectures.
    Gareth Rushgrove
    http://jasonwilder.com/blog/2014/02/04/service-discovery-in-the-cloud/
    

    View Slide

13. The problem seems simple at
    first: How do clients determine
    the IP and port for a service that
    exist on multiple hosts?
    Gareth Rushgrove
    http://jasonwilder.com/blog/2014/02/04/service-discovery-in-the-cloud/
    

    View Slide

14. Etcd, Consul, Zookeeper
    Gareth Rushgrove
    

    View Slide

15. Hadoop uses Zookeeper
    Socketplane uses Consul
    CoreOS uses etcd
    Cloud Foundry uses Consul
    Cloud Foundry also uses etcd
    Kubernetes uses etcd
    Gareth Rushgrove
    

    View Slide

16. Gareth Rushgrove
    Thanks to etcd for the graphics
    

    View Slide

17. Gareth Rushgrove
    Thanks to etcd for the graphics
    

    View Slide

18. Gareth Rushgrove
    Thanks to etcd for the graphics
    

    View Slide

19. Gareth Rushgrove
    Jepsen
    

    View Slide

20. Why now?
    

    View Slide

21. Every organisation wants to
    move quicker
    Gareth Rushgrove
    

    View Slide

22. Provisioning times are
    decreasing rapidly
    Gareth Rushgrove
    

    View Slide

23. The number of applications, and
    instances of those applications, is
    increasing rapidly
    Gareth Rushgrove
    

    View Slide

24. The number of systems
    administrators is probably going
    to stay the same. Sorry.
    Gareth Rushgrove
    

    View Slide

25. Technology moving towards a
    cluster centric view rather than a
    host centric view
    Gareth Rushgrove
    

    View Slide

26. Hard-coding everything
    doesn’t scale
    Gareth Rushgrove
    

    View Slide

27. Some relevant technology you
    may have heard of
    Gareth Rushgrove
    

    View Slide

28. Gareth Rushgrove
    

    View Slide

29. Gareth Rushgrove
    

    View Slide

30. Gareth Rushgrove
    

    View Slide

31. Gareth Rushgrove
    

    View Slide

32. Two speeds of
    configuration
    

    View Slide

33. Configuration management tools
    tend to focus on upfront
    modelling, managing
    configuration we know and want
    to control over time
    Gareth Rushgrove
    

    View Slide

34. For example the version of a
    package or application, user
    permissions, security settings,
    system constrains
    Gareth Rushgrove
    

    View Slide

35. Service discovery tools handle
    emergent configuration which is
    a property of the running system
    Gareth Rushgrove
    

    View Slide

36. For example load balancer and
    proxy backends, network and
    firewall routes as applications
    come and go
    Gareth Rushgrove
    

    View Slide

37. Wait, but I want to define some of
    my network configuration upfront
    Gareth Rushgrove
    

    View Slide

38. Hence two speeds, model
    driven and emergent
    Gareth Rushgrove
    

    View Slide

39. How to mix
    configuration
    management and
    service discovery
    

    View Slide

40. We’ll use Consul for this
    example
    Gareth Rushgrove
    

    View Slide

41. We’ll also use Puppet
    Gareth Rushgrove
    

    View Slide

42. First lets install Consul
    everywhere
    Gareth Rushgrove
    

    View Slide

43. Gareth Rushgrove
    

    View Slide

44. That’s it
    Gareth Rushgrove
    

    View Slide

45. Gareth Rushgrove
    KyleAnderson/consul
    

    View Slide

46. We can configure servers
    and clients using Hiera
    Gareth Rushgrove
    

    View Slide

47. Gareth Rushgrove
    

    View Slide

48. Gareth Rushgrove
    

    View Slide

49. Consul nodes need to
    register with a server
    Gareth Rushgrove
    

    View Slide

50. Gareth Rushgrove
    

    View Slide

51. Populating the Consul
    service registry
    Gareth Rushgrove
    

    View Slide

52. Gareth Rushgrove
    

    View Slide

53. Gareth Rushgrove
    Consul dashboard
    

    View Slide

54. Create your own types
    Gareth Rushgrove
    

    View Slide

55. Gareth Rushgrove
    

    View Slide

56. Gareth Rushgrove
    

    View Slide

57. Using information from
    Consul in Puppet
    Gareth Rushgrove
    

    View Slide

58. Gareth Rushgrove
    

    View Slide

59. Gareth Rushgrove
    

    View Slide

60. Gareth Rushgrove
    

    View Slide

61. Gareth Rushgrove
    lynxman/hiera_consul
    

    View Slide

62. Trigger Puppet when
    something pertinent
    changes
    Gareth Rushgrove
    

    View Slide

63. Gareth Rushgrove
    

    View Slide

64. Gareth Rushgrove
    Demo time
    

    View Slide

65. Same example possible
    with a bit of hacking
    for etcd
    Gareth Rushgrove
    

    View Slide

66. Gareth Rushgrove
    garethr/hiera-etcd
    

    View Slide

67. Gareth Rushgrove
    KyleAnderson/etcd
    

    View Slide

68. Advantages
    Changes happen quickly,
    but without losing the
    ability to address
    configuration drift
    Gareth Rushgrove
    

    View Slide

69. Advantages
    Reduce complexity of
    watchers, all changes lead
    to a Puppet run
    Gareth Rushgrove
    

    View Slide

70. Advantages
    All changes to
    configuration, however
    triggered, are tracked in
    one place
    Gareth Rushgrove
    

    View Slide

71. Advantages
    Define all watchers
    in code
    Gareth Rushgrove
    

    View Slide

72. Advantages
    Reduce duplication, same
    code runs for new and
    existing instances
    Gareth Rushgrove
    

    View Slide

73. Bonus distributed
    configuration
    Gareth Rushgrove
    

    View Slide

74. Replace lots of identical
    local files on disk with
    distributed key/values
    Gareth Rushgrove
    

    View Slide

75. Gareth Rushgrove
    

    View Slide

76. Gareth Rushgrove
    

    View Slide

77. Gareth Rushgrove
    garethr/key_value_config
    

    View Slide

78. Conclusions
    

    View Slide

79. Change control and autonomous
    systems
    Gareth Rushgrove
    

    View Slide

80. The importance of building trust
    in automation
    Gareth Rushgrove
    

    View Slide

81. Two speeds of configuration
    Gareth Rushgrove
    

    View Slide

82. Questions?
    And thanks for listening
    

    View Slide